AI Agenda

Why Anthropic and OpenAI Are Using Cognition’s Coding Test

Art via Getty Images; Mike Sullivan
By
Rocket Drew

As artificial intelligence models improve, the companies developing them are seeking more sophisticated ways to measure how good they are at a wide range of tasks. As a result, they are developing internal benchmarks to test models’ capabilities, rather than relying on publicly available tests.

Cognition, which has created the Devin coding agent, is one of the startups that’s developed internal benchmarks. Its tests for agentic coding capabilities measure an AI system’s skill at solving long programming tasks, which often involves working across multiple files of code. 

According to its main benchmark, Anthropic’s Claude Sonnet 4.5 is the best model for agentic coding, followed by OpenAI’s models, said Russell Kaplan, Cognition’s president. However, the gaps between all the AI companies’ scores have been narrowing over time. 

OpenAI, Anthropic and xAI have asked Cognition to evaluate their models on its primary benchmark—Junior Dev—before they are released to get a sense of their models’ abilities, Kaplan said. 

Junior Dev contains problems that a junior developer might encounter and is focused on testing models’ abilities to modify existing code bases, rather than generating new code from scratch. 

Along with the models’ scores, Cognition also provides feedback on their specific weaknesses, for example if a model is struggling with long-context questions. And Cognition measures how quickly AI agents can solve the coding problems in Junior Dev.

Devin itself incorporates models from other providers, such as Anthropic, along with Cognition's own models. 

AI companies have started to create their own benchmarks because the public ones tend to leak into the data used to train the model, leading models to memorize the answers, which defeats the point of the tests. Plus, internal benchmarks can be more tailored to measure the exact capabilities that the company is targeting. 

For example, compared to SWE-Bench, a popular public benchmark that focuses on the programming language Python, Junior Dev includes problems in a variety of programming languages, Kaplan said. Cognition found that if it designed Devin to succeed just on SWE-Bench, then Devin would not improve on real tasks, he added.

Private benchmarks also offer a competitive edge: the feedback from these tests allows companies to more quickly develop their products. 

Video AI startup Runway has developed internal benchmarks of models’ capabilities to generate videos with realistic physics, co-founder and CTO Anastasis Germanidis said in July. It uses the feedback to improve its video-generating models.

AI companies’ interest in Cognition’s Junior Dev is a sign of the AI industry’s current fixation on agentic coding as a barometer of AI’s progress. As for Senior Dev? It’s on its way, Kaplan said.

Here’s what else is going on…

Deals and Debuts

See The Information’s Generative AI Database for an exclusive list of private companies and their investors.

OpenAI is developing AI that can generate music, as it looks to aggressively expand the types of services it provides beyond its core chatbot, The Information reported.

SoftBank’s board has approved a second installment of $22.5 billion in its $30 billion investment in OpenAI, as long as the startup completes a corporate restructuring that would pave the way for an eventual public offering, The Information reported.

SambaNova Systems, an AI chip startup, is exploring a sale after it struggled to complete a fundraising round, The Information reported.

Crusoe Energy, which builds data centers, held a first close on its $1.4 billion Series E round at a valuation of more than $10 billion led by Valor Equity Partners and Mubadala Capital.

Palantir announced on Thursday that it has struck a partnership with telecommunications company Lumen Technologies, which will use Palantir’s AI software to build capabilities to support enterprise AI services. Lumen is going to spend over $200 million on Palantir’s tech over the course of several years, according to a Bloomberg report.

Valthos, a biodefense startup, raised $30 million in seed funding from OpenAI Startup Fund, Lux Capital and Founders Fund.

Onfire, which uses AI to find sales leads for software vendors, has raised $20 million, including $14 million in a Series A funding round led by Grove Ventures and TLV Partners, with participation from In Venture and LeumiTech77, TechCrunch reported.

Riff, an Oslo, Norway-based vibe coding startup focused on enterprise-grade applications, raised $16 million in Series A funding led by Northzone.

Darwin AI, a New York City-based startup that helps public sector agencies integrate AI safely, raised $15 million in Series A funding led by Insight Partners.

AudioShake, which develops audio separation and processing technology, raised a $14 million seed round led by Shine Capital.

Brico, which uses AI to automate state and federal licensing for financial institutions, raised $13.5 million in Series A funding led by Flourish Ventures.

Wonder Studios, a London-based AI creative studio, raised $12 million in seed funding led by Atomico.

Wolf Games, a Los Angeles-based startup using AI to build video games, raised $9 million in Series A funding led by Main Street Advisors.

VitVio, a Boston-based startup that uses AI to coordinate operating room staff and other hospital operations, raised $8 million in seed funding led by Bek Ventures.

Tempo, a Toronto-based startup that allows designers and engineers to collaborate on building apps, raised $5 million in seed funding from Y Combinator, Golden Ventures, Box Group, Webflow Ventures and others.

Tensormesh, a San Francisco-based inference optimization startup for enterprise AI, raised $4.5 million in seed funding led by Laude Ventures.

OpenAI announced on Thursday that it has acquired Software Applications, Inc., which makes an AI-powered natural language interface for Mac computers called Sky.

Microsoft announced updates to its Copilot AI assistant, including “Copilot Mode” in its Edge browser, an AI assistant that will follow users as they browse the web.

Anjney Midha, an Andreessen Horowitz general partner who helped the firm’s startups gain access to highly sought-after graphics processing units, told the firm’s staff on Friday that he was setting up an outside venture called AMP to “provide compute and capital to frontier AI teams,” The Information reported. 

Recommended Newsletter

Every weekday, The Briefing helps executives get smarter about the latest in tech, media and finance. Subscribe for free now.

Thank you for reading the AI Agenda Newsletter! I’d love your feedback, ideas and tips: [email protected].

If you think someone else might enjoy this newsletter, please pass it forward or they can sign up here.