Why xAI Spent So Much on Reinforcement Learning

Watching last week’s Grok 4 release, one statistic stood out to us. XAI said it spent 10 times as much computational power on reinforcement learning for Grok 4 as it laid out for Grok 3.

RL is a common AI development technique that rewards a model for accomplishing certain goals and penalizes it for other behaviors. The xAI disclosure provided further evidence of how AI labs are increasingly improving their models in post-training, in which models review curated data (smart answers to tough questions) to refine their responses, rather than in the pretraining process in which models review all types of data and try to draw connections between them.

Let’s take a step back. Historically, researchers have found it easier to use reinforcement learning to improve models for relatively simple tasks like playing a game of Go or chess. In these cases, it’s easy to know when the model has taken the right steps because it will win the game. The researchers can then tell the model to take those sorts of steps more often in the future so it’ll be more likely to win.

But what happens when the task is more complicated than a simple game, such as when a model is asked to design a fusion reactor? In these cases, knowing whether the model’s answer is correct can be extremely difficult.