‘It could kill us all’: AI researcher quits Anthropic over race to superintelligence

A researcher who spent the past three years training artificial intelligence models at OpenAI and Anthropic has resigned from Anthropic, saying the companies are racing toward self-improving "superintelligence" without adequately addressing the risks.

Jacob Coxon, who specialises in training new AI models using vast amounts of data, announced his resignation in a series of posts on X, saying neither company was "acting responsibly."

"They are racing straight to self-improving superintelligence and gambling with our lives," Coxon said.

Coxon warned that AI systems could soon become capable of hacking systems, transforming entire fields almost overnight and acquiring "real power and resources."

He said many people working on AI privately believed the technology could pose an existential threat to humanity within the decade, despite more measured public statements by executives and senior researchers.

"The people building AI earnestly believe that it could kill us all by the end of the decade," Coxon said.

Evan Hubinger, Alignment Science lead at Anthropic, agreed with Coxon that "AI could kill all humans".

"I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to," he added.

Coxon argued that the development race was particularly risky because companies could feel compelled to move faster out of fear that competitors would not act responsibly.

He said that while some at OpenAI had not fully internalised the "civilizational stakes", researchers at Anthropic understood the risks but felt compelled to keep moving because of competition.

“At Anthropic, the stakes are well-understood, but they are locked in a race to get there first," the AI researcher wrote. “Accepting this race and entering the 'endgame' is a hubristic gamble that should not be launched from a private company's Slack.”

Coxon said he remained optimistic that international and industry coordination could slow the race.

But he warned that preventing a global race could require costly measures, including a temporary ban on improving AI model capabilities.

OpenAI chief scientist's warning

His comments came days after OpenAI chief scientist Jakub Pachocki issued a similarly stark warning about the pace of AI development.

Pachocki said in a blog post published Sunday that he had a "strong expectation" that the current pace of progress could continue into recursive self-improvement, in which AI systems increasingly help develop more capable versions of themselves.

Future AI systems could see capability gains as large as, or larger than, those achieved in recent years, Pachocki said.

"This is a time that calls for extreme caution," he wrote, warning that "no one is prepared for the consequences" of a continued rapid increase in machine intelligence.

Pachocki said OpenAI would continue working on technical approaches to AI alignment and monitoring, as well as defensive systems, and would "unilaterally withhold further scaling as needed."

But he stressed that "broader interventions are required."