How AI Researchers Are Rising Above Scaling Limits

Now that we know OpenAI isn’t the only artificial intelligence developer seeing a slowdown in improvements using traditional “scaling” methods, it’s worth looking at all the ways companies are trying to make up for that.

There are tried and true methods to make large language models perform better, such as tweaking the parameters—settings that determine how the models “learn” from data and answer questions. And there are loads of other changes developers are making to models after they are initially trained, using a bevy of data and computing power, including asking armies of humans to rate their answers and steer them toward better ones. 

There are also some newer strategies, including one that’s hot off the research presses and has turned some heads this week.

First, let’s look at some common techniques.