Anthropic Researcher Quits, Warns Self-Improving AI Could Escape Control

The latest:

A researcher at Anthropic said Tuesday he is quitting the artificial-intelligence industry because he does not want to help build systems capable of improving themselves, which he fears could spiral beyond human control. Jacob Coxon, 27, trains models on large volumes of data and previously worked at OpenAI. Anthropic did not immediately comment on his departure.

Details:

• The warning: Coxon said the industry is on track for the most aggressive scenarios, in which things could already be out of control by the end of next year. He argued safety trade-offs become inevitable when companies compete against each other and against Chinese rivals.
• His trajectory: Coxon said he left OpenAI earlier this year for Anthropic because of its reputation on model safety. He found those efforts earnest, but now believes no company can responsibly build AI that outperforms humans across tasks without government intervention or a coordinated industry slowdown.
• The vocabulary: The British researcher, who previously studied mathematics, said many colleagues now use words like “crunchtime” and endgame to describe the path toward self-improving models. He said Anthropic staff debate their models’ capabilities in an internal Slack channel.
• The evidence cited: Recent hacks involving OpenAI and Anthropic models, some operating in collaborative swarms of agents, showed systems adopting nefarious goals and concealing them from humans, according to the account. Coxon said self-improving systems could eventually advance enough to refuse commands.
• Not the first: His exit is among the first Anthropic departures over safety fears. A safety researcher left earlier this year to study poetry, warning the world is in peril. Researchers have also left OpenAI and other firms in recent years citing similar concerns.
• OpenAI’s position: OpenAI told reporters last week its latest model amounted to artificial general intelligence and a major capability jump. Chief Executive Sam Altman warned Group of 20 officials at a North Carolina summit that cybersecurity failures are likely unless people act urgently.
• The petition: Coxon joined OpenAI chief scientist Jakub Pachocki, Anthropic CEO Dario Amodei and more than 1,000 researchers in signing a statement urging global coordination on a mechanism to slow development of self-improving models. Pachocki called for extreme caution in a Sunday blog post.
• Washington’s stance: There are no federal AI regulations, and the Trump administration has prioritized a light-touch approach to maximize economic benefits. Senator Bernie Sanders and Representative Greg Casar introduced legislation last week to permanently ban superintelligence and pause model development pending an industry regulator’s rules.
• The IPO backdrop: The departure comes as Anthropic prepares an initial public offering expected to rank among the largest ever, seeking a $2 trillion valuation. The company has emphasized responsible development to investors, and its leaders have clashed with the Trump administration over safety practices.

Background:

Anthropic built one of the largest AI platforms partly by promising to prioritize responsible development and investing heavily in safety research. Amodei and other executives have repeatedly warned about rogue models and urged the industry to slow down.

Between the lines:

Coxon’s exit exposes a tension Anthropic markets to investors: the company most identified with safety is preparing a record IPO while one of its own researchers says no lab can build human-level AI responsibly without outside intervention. The competitive pressure he ties to rival labs and Chinese entrants is the same dynamic driving the valuation.

What’s next

Watch whether Anthropic responds publicly, how the Sanders-Casar bill fares in a Congress with no federal AI rules, and whether the IPO filing addresses safety commitments alongside the $2 trillion valuation target.