Anthropic Sounds Alarm on Recursive Self-Improvement

Last week, Anthropic boasted that Claude is now writing 80% of Anthropic’s code, which the company presented as a harbinger of a critical leap in artificial intelligence called recursive self-improvement. That’s when models are so capable that they can create the next generation of AI without human involvement. 

Excitement about recursive self-improvement seems to be in the air in Silicon Valley. Last month OpenAI sponsored a conference on the topic in San Francisco that attracted researchers from companies including Anthropic and Google DeepMind. Meanwhile, Recursive Superintelligence, which aims to use AI to develop a system as smart as “50,000 PhDs,” raised $650 million, and Inherent, which is also pursuing recursive self-improvement, raised $50 million. Earlier this year, Ricursive raised $300 million to create AI tools for designing AI chips.

If taking humans out of the loop during the creation of superhuman AI systems sounds a little Terminator-esque to you, Anthropic agrees. In last week's announcement, it said that instances of models developing their own unintended goals “could compound as the models build their successors, growing more frequent but less understood until we lose control of them.”