Meta, which develops one of the biggest foundational open-source large language models, Llama, believes it will need significantly more computing power to train models in the future.
Mark Zuckerberg said on Meta’s second-quarter earnings call on Tuesday that to train Llama 4 the company will need 10x more compute than what was needed to train Llama 3. But he still wants Meta to build capacity to train models rather than fall behind its competitors.
“The amount of computing needed to train Llama 4 will likely be almost 10 times more than what we used to train Llama 3, and future models will continue to grow beyond that,” Zuckerberg said.
“It’s hard to predict how this will trend multiple generations out into the future. But at this point, I’d rather risk building capacity before it is needed rather than too late, given the long lead times for spinning up new inference projects.”
Meta released Llama 3 with 80 billion parameters in April. The company last week released an upgraded version of the model, called Llama 3.1 405B, which had 405 billion parameters, making it Meta’s biggest open-source model.
Meta’s CFO, Susan Li, also said the company is thinking about different data center projects and building capacity to train future AI models. She said Meta expects this investment to increase capital expenditures in 2025.
Training large language models can be a costly business. Meta’s capital expenditures rose nearly 33% to $8.5 billion in Q2 2024, from $6.4 billion a year earlier, driven by investments in servers, data centers and network infrastructure.
According to a report from The Information, OpenAI spends $3 billion on training models and an additional $4 billion on renting servers at a discount rate from Microsoft.
“As we scale generative AI training capacity to advance our foundation models, we’ll continue to build our infrastructure in a way that provides us with flexibility in how we use it over time. This will allow us to direct training capacity to gen AI inference or to our core ranking and recommendation work, when we expect that doing so would be more valuable,” Li said during the call.
During the call, Meta also talked about its consumer-facing Meta AI’s usage and said India is the largest market of its chatbot. But Li noted that the company doesn’t expect Gen AI products to contribute to revenue in a significant way.
