What Could Stop AI Scaling?

So far, many of the improvements in large language models’ capabilities have stemmed from a surprisingly simple concept: scaling laws. Essentially, researchers have noticed that the more computing power and data you use to train AI models, the better they perform. 

This realization has pushed labs like OpenAI, Meta Platforms, Anthropic and Google into fierce (and expensive) battles for chips and new sources of training data. But, new research from Epoch AI, a nonprofit AI lab funded by sources including effective altruism-focused foundation Open Philanthropy, questions how much longer model developers can depend on this strategy. 

The answer is, quite a while, most likely. The San Francisco-based lab concluded that the industry can probably keep expanding at the current rate at least until the end of this decade.

The biggest threat to this time frame is access to power, not surprisingly—while a number of other factors, such as a lack of high-quality data due to copyright concerns or supply chain issues for chip manufacturing, also loom on the horizon.