The latest

A warning from Anthropic team lead Evan Hubinger that artificial intelligence could kill all humans reached tens of millions of viewers and drew responses from state and federal lawmakers. Hubinger estimated the probability at more than 10 percent within the next decade, writing on X after colleague Jacob Coxon resigned and accused Anthropic of racing ahead despite the dangers. The reaction marked a sharp contrast with 2022, when a similar warning from Hubinger attracted little attention.

Details

• Powerful agents: The renewed attention follows demonstrations in which AI agents surpassed mathematical milestones and hacked company servers, prompting a national security scramble at the White House. Researchers have also begun discussing systems that could act independently on computers and build improved versions of themselves.
• Changed assessments: Nathan Lambert, formerly a senior research scientist at the Allen Institute for AI, said he initially dismissed Anthropic CEO Dario Amodei’s prediction that AI would take over software engineering. Lambert now lets AI write his code and considers safety advocates remarkably farsighted, while cautioning that their more speculative predictions may not materialize.
• Precision questioned: Sara Hooker, CEO and co-founder of AI start-up Adapation and a former Google DeepMind researcher, called extinction predictions unhelpful and questioned the basis for Hubinger’s 10 percent estimate. She said the latest warnings are reaching a public already increasingly uneasy about AI’s growing power.
• Academic shift: Johns Hopkins philosophy professor Seth Lazar previously argued for prioritizing immediate AI harms and doubted systems would become powerful enough to cause civilizational damage independently. More capable reasoning models and functioning computer agents have weakened that confidence. He said rogue agents could wreak havoc within one or two years, although extinction still felt like a leap.
• Political reach: AI safety organizations have backed social-media influencers discussing existential risk and courted members of Congress, including Senator Bernie Sanders. Responding to Coxon’s resignation, Sanders wrote that the people building the technology acknowledge it could threaten humanity’s future.
• Corporate momentum: OpenAI and Anthropic were founded by researchers connected to the AI safety movement, and executives at both companies have voiced concerns about potential harm to humanity. Neither company slowed its expansion, and both filed stock-market listing paperwork. Anthropic is preparing to go public at a valuation exceeding $1 trillion.

Background

The AI safety movement emerged from online forums including LessWrong and intersected with effective altruism and transhumanism. Its central premise is that AI will surpass human capabilities and could extinguish humanity unless its values remain aligned with those of its developers. Wealthy technology donors spent hundreds of millions of dollars over the past decade supporting related university programs, nonprofits, grants and fellowships.

What’s next

The next concrete indicators will be the progress of OpenAI’s and Anthropic’s stock-market filings, lawmakers’ responses to extinction warnings, and new demonstrations of agents performing independent computer and software-development tasks.

Source