Key Points
- Researchers say recursive self-improvement claims lack evidence, calling AI doom forecasts speculative rather than calculated.
- Critics want AI held to safety standards governing airplanes, elevators, food and drugs.
- Focusing on extinction risk, they argue, diverts regulation from harms already occurring.
The latest:
Artificial intelligence cannot yet do the original research needed to build its own successors, according to a paper by roughly two dozen academics at Princeton, Stanford and other institutions cited by The Wall Street Journal. The finding undercuts a central claim in the industry’s doomsday debate, as safety researchers press instead for conventional product regulation covering systems already causing harm.
Details:
- The core claim: The debate centers on recursive self-improvement, the idea that current systems could take over training their next versions. Anthropic Chief Executive Dario Amodei recently proposed a global agreement to slow the release of new AI models, with the stated goal of delaying that moment.
- The skeptics: Melanie Mitchell, a professor at the Santa Fe Institute, said the field is nowhere near artificial general intelligence, and that engineers’ claims of achieving recursive self-improvement do not withstand scrutiny. She acknowledged AI is making impressive strides.
- The hack: Two authors of the Princeton-Stanford paper rejected the idea that the Hugging Face swarm hack by OpenAI agents reflected an intelligence breakthrough. They wrote it succeeded mainly because basic technical guardrails were missing, not because capability had exploded beyond management.
- Industry reaction: Yann LeCun, former chief AI scientist at Meta, posted that the analysis was a welcome dose of sanity in what he called an otherwise insane debate. OpenAI and Anthropic did not respond to several requests for comment from the Journal.
- The other side: Rayan Krishnan, chief executive of Vals AI, said his team projects models will exceed human ability to improve the next generation by August 2027 or sooner. His firm’s RSI Index benchmarks that capability, and no publicly released model is close so far.
- The numbers: Anthropic engineers put the probability of catastrophe over the next decade at 10% or higher. Mitchell said the figure has stayed at 10% for more than a decade because it is a round number, describing the estimates as vibes rather than calculation. Published estimates range from 0% to nearly 100%.
- The alternative: Stuart Russell, a computer-science professor at Berkeley and president of the International Association for Safe and Ethical Artificial Intelligence, wants AI held to the standards governing air travel, buildings, food and drugs, with behavioral red lines around hacking, data theft and bioweapons advice.
- The catch: Russell said such rules would amount to a de facto ban on today’s advanced systems, because the companies building them do not know how to make the models respect those boundaries all of the time.
- Regulatory options: Proposals for an FDA-style agency for AI have been a nonstarter in Congress, with some technology figures arguing it would cede America’s lead to China. Russell said the industry’s mantra that regulation is bad, and its refusal to accept liability, has to change.
- Research abroad: Researchers at Chinese universities and commercial AI labs wrote that autonomous self-improving AI is likely far off, given the breakthroughs required, and that humans will probably stay in the loop supervising the pace.
Background:
Current systems advance largely by absorbing human knowledge, and can sometimes recombine it to exceed human performance through reinforcement learning, as seen in mathematics. Yi Ma, professor of AI at Hong Kong University, wrote that today’s large language models are models of knowledge rather than genuinely intelligent.
Between the lines:
The dispute is less about whether AI is dangerous than about which danger gets regulated. Christopher Canal of EquiStamp argued that a world full of improbable extinction scenarios makes prioritization impossible, and that treating AI as an existential threat implies it could equally cure cancer. Russell’s analogy is blunter: slowing from 60 to 40 miles per hour still ends at the cliff.
What’s next
Watch whether Vals AI’s RSI Index registers movement before its August 2027 projection, whether Amodei’s proposed global slowdown agreement attracts signatories, and whether support in Congress shifts on creating an AI oversight agency.
