A New (and Better) Approach to Measuring LLMs’ Intelligence Has Arrived

Finally, there’s good news from the never-ending quest to evaluate how good a large language model is. 

LMSYS, an organization that benchmarks LLMs, released findings last month on how the performance of LLMs changed between normal queries and harder ones that require more reasoning capabilities. LMSYS uses “Elo rankings,” a measure of relative performance—so you’d expect the performance metrics to remain the same in both cases. 

Instead, it found that many models—including Google's Gemini 1.5 Pro, Cohere's Command R+ and Meta Platforms' Llama 3—saw a marked decline in performance when moving from normal to hard prompts. OpenAI’s GPT-4o and Anthropic's Claude 3 Opus were among the few that maintained their performance. Check out the bottom of their blog post here for examples of these “hard” questions, like coding problems. (Gemini Ultra wasn’t included in this analysis because it’s still not widely available to developers through an application programming interface.)

The decline in performance from so many LLMs could be a sign that LLMs still aren’t very good at complex reasoning today, said Ion Stoica, a Berkeley computer science professor who also runs LMSYS, noting that OpenAI and Anthropic might have developed special techniques to give models some of those skills. (Stoica is also a cofounder of AI startup Anyscale and enterprise software firm Databricks).