Will Chains of Thought Stay Readable for Long?

Chains of thought, the “reasoning” steps that many AI models take before delivering their final answer to a query, offer AI developers a precious window into their models’ decision-making. That’s why Google faced a backlash from its users earlier this month when it followed the lead of OpenAI and Anthropic by deciding to hide its models’ chains of thought, showing a summary instead.

But even if users could read the raw chains of thought, they might not be able to understand them for long. While most chains of thought are legible today, some AI researchers have warned me that as AI companies train increasingly large reasoning models, their chains of thought more frequently deteriorate into gibberish. One pessimistic OpenAI researcher said they give it about a year before the leading models’ chains of thought are illegible.

This isn’t a reflection on the quality of the answer, to be sure, which should remain high. It’s a reflection of how reasoning models are trained: they’re not required to adhere to the strictures of English grammar. Instead, they’re encouraged to use whatever chain of thought produces the correct answers to math and coding problems. It turns out that sometimes the models learn to solve those problems by taking a shortcut through a bit of gibberish.