Bruin AI Current Events

Training DeepSeek | The Cost of Progress

Good afternoon Bruin Bots 🤖

Our deepest condolences go out to everyone impacted by the devastating wildfires in Los Angeles. We hope that you and your loved ones are staying safe and healthy during these difficult times.

In this edition, we explore advancements of AI technology with China’s DeepSeek AI. We also discuss the alarming rise in global temperatures and how it contributes to increasingly dry and volatile weather conditions.

Thank you for being a part of the Bruin AI community. Feel free to comment and share your thoughts below.

💸 Economy

🧭 Ethics

-Karen

Does DeepSeek V3 Prove AI Development Has Been Wasteful?

On December 26th of 2024, China's DeepSeek AI released DeepSeek-V3 to the public with detailed technical reports highlighting its abilities. The model was trained on 14.8 trillion tokens and 37 billion active parameters, aiding it in demonstrating advancements over most other AI models such as GPT-4, Claude-3.5 and more.

While the advancements are impressive, we have generally come to expect such advances when novel AI engines are launched. What is truly impressive was the cost efficiency that was put on full display with the launch of this model. According to DeepSeek's technical report, its newest model required only 2.788 million GPU hours, a common measure of training time, while comparable models such as the Llama 3.1 405b took almost 31 million hours to be trained.

DeepSeek made a particular effort to improve upon or utilize a multitude of efficient training methods for this model such as experimenting with partial 8-bit native training and multi-head latent attention, all of which can be read about in the technical report but would exceed the technical depth intended for this post.

In the states, leading AI companies can be seen essentially bragging about the size and scale of their training facilities, such as xAI taking pride in their Memphis Supercluster of GPUs, one of the largest "training cluster[s] in the world." For Chinese companies working with the backdrop of substantial export controls, however, a greater focus on efficient development has come to the forefront.

Nevertheless, the cost efficiency may not be as impressive as meets the eye. While tracking training times and the amount of GPU usage can estimate costs, it seems unhelpful to truly understand the actual costs involved with a model. Many costs come prior to the AI being developed, for example electricity costs or employment costs.

Employment particularly comes to mind when looking at DeepSeek as its technical paper has 139 authors, an astonishingly large technical team that likely rakes in impressively large salaries, with just some individuals earning salaries in the millions. Additionally, with expanded expertise thanks to the knowledge produced by existing AI models such as ChatGPT and especially open-source models such as Llama, teams are able to build models for much cheaper with far less risk because they know what has been shown to work and where risks lie.

While DeepSeek-V3 may not have been as cheap to produce as it would appear on face, what it does demonstrate is that the advancements in AI research are lending themselves well producing cheaper and more efficient development methods in the future. While DeepSeek absolutely was able to achieve an impressive result in both the quality and efficiency of its model, to simply label the current Big Tech leaders in AI as wasteful would be to disregard the high price-tag developments that have allowed cheaper and more efficient AI training methods to be created.​​​​​​​​​​​​​​​​

-Tobin

Rising Global Temperatures

As 2024 has come to a close, the earth has recorded its hottest year in recorded history, beating out 2023, the previous hottest year. Between the supercharged hurricanes that swept through the Gulf of Mexico and the dry conditions that set the stage for the current Los Angeles fires, the impacts of climate change are being felt at an increasingly devastating scale.

Having moved beyond the 1.5-degree goal, recent studies highlight the extent to which institutional constraints have made 1.6 degrees Celsius a possible best-case scenario. In light of these developments, the biggest players in AI have not reduced their commitments to any significant extent. Microsoft, for example, announced its plans to spend $80 billion on data centers to fuel its AI business. In order to power these data centers, however, significant levels of electricity use will be required.

Following the leadership of tech companies in corporate America’s efforts to combat global warming, this paradigm shift seems ironic. In 2020, Microsoft promised to remove its historic carbon emissions from the atmosphere while companies such as Amazon and Google followed on with similar climate pledges. Now, however, these companies seem to be giving up such targets, reporting spikes in their emissions due to the demands of data centers.

While the expansion of AI absolutely risks global climate change goals, upside potential exists. Even before we discuss what AI may provide, the largest companies that are developing AI remain big supporters of renewable power, and many buy carbon credits often. As such, the high demand for energy may be a boon for renewable energy developers as they are given funds to supply and further develop renewable technologies. In fact, as with Microsoft, many companies are looking towards nuclear energy.

Sam Altman of OpenAI has advocated for and invested in nuclear energy including nuclear fusion. By demonstrating such demand, a possible nuclear renaissance may emerge. Additionally, AI can improve energy efficiency and aid in delivering clean energy breakthroughs, benefiting climate goals.

The issue, however, is that these advancements can take decades to materialize, while AI needs power in the present. Even though some experts are confident that AI will be a net positive, overall aiding in reducing emissions more than it hurts, the timing of technological advancements remains critical. Given present issues that have resulted from climate change, we must ask how we can prevent the initial uptick in emissions due to AI from significantly impacting the people we aim to help.

-Tobin

Feel free to elaborate on any of your thoughts through this Anonymous Feedback Form.

All the best,

Tobin Wilson, Editorial Intern

Karen Harrison, Newsletter Manager

The Bruin AI Team 🤖✨

.

.

.

"The only way to discover the limits of the possible is to go beyond them into the impossible." – Arthur C. Clarke