Where Reinforcement Learning is Going

Ever since researchers began noticing a slowdown in improvements to large language models using traditional training methods, they’ve shifted to reinforcement learning, a technique that rewards a model for accomplishing certain goals and penalizes it for other behaviors, as we covered last week.

That typically means human experts giving feedback on model outputs or showing a model examples of tough questions and answers in fields like software engineering, biology and medicine—essentially, teaching the model what good responses look like.

In many ways, though, researchers tell me that this RL approach is rudimentary: it works best in fields like math where questions only have one right answer, and the process is hard to automate.

In fields where there can be multiple “right” answers, such as asking a model to create a three-dimensional Tetris game, researchers can give feedback while the AI is working on the problem instead of just at the end. That feedback can include comments about the game’s design, controls or how efficient the model’s code is.