AI researcher quits Google DeepMind, warns of ‘terrifying chance’ of AI causing immense harm within 5 years

A former researcher on Google DeepMind's artificial general intelligence (AGI) safety team has left the company to join AI safety organisation METR, saying he believes there is a "terrifying chance" that artificial intelligence systems could cause immense harm within the next five years.

Josh Engels, who left Google DeepMind three weeks ago, said he turned down offers from OpenAI and Anthropic to work at METR, where he plans to study AI misalignment and assess whether existing safeguards are adequate as companies race to develop increasingly capable systems.

"The AI companies are all trying to build superintelligence: systems vastly better than humans at everything," Engels said in a post on X.

He said companies were pursuing recursive self-improvement (RSI), a process in which AI systems help build more capable versions of themselves, potentially creating a feedback loop of rapidly improving systems.

While such systems could help solve major problems, Engels said the industry does not yet know how to make AI systems sufficiently safe before allowing them to engage in recursive self-improvement.

"And unfortunately, current AIs seem to be getting less aligned over time, not more," he said, pointing to recent instances of AI models allegedly colluding with one another, hacking into companies, concealing their activities and socially engineering humans.

Engels said the individual incidents were not necessarily dangerous in themselves, but indicated that current AI systems were not sufficiently aligned to safely initiate recursive self-improvement.

"I now think that there's a terrifying chance that AI systems cause immense harm in the next five years," he said, adding that he did not know the exact probability but believed the risk was high enough to make AI safety "the most important problem in the world".

He called for more time to develop safeguards, arguing that AI development should be paced so that the growth in capabilities does not outstrip the ability to align and evaluate the systems.

Engels said he would work on understanding the sources of misalignment during training, evaluating existing safety measures and assessing whether the industry was on track to solve the alignment problem.

Coxon said the companies were "racing straight to self-improving superintelligence and gambling with our lives".

He warned that increasingly capable AI systems could eventually hack computer systems, transform entire fields almost overnight and acquire "real power and resources".

Coxon also claimed that many people working in AI privately believed the technology could pose an existential threat to humanity within the decade, despite more measured public statements from industry executives and researchers.

"The people building AI earnestly believe that it could kill us all by the end of the decade," Coxon said.

Anthropic's Alignment Science lead Evan Hubinger also said he believed AI could potentially kill all humans, putting his personal estimate of the risk at more than 10% over the next decade.

"I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to," Hubinger said.

The warnings come as leading AI companies publicly acknowledge that rapid advances in AI could lead to new safety challenges.

OpenAI chief scientist Jakub Pachocki recently said he had a "strong expectation" that the current pace of progress could eventually lead to recursive self-improvement.

Future AI systems could achieve capability gains as large as, or greater than, those seen in recent years, Pachocki said, warning that the situation called for "extreme caution".

"No one is prepared for the consequences" of a continued rapid increase in machine intelligence, he said.

Pachocki said OpenAI would continue developing technical approaches to AI alignment and monitoring, along with defensive systems, and would "unilaterally withhold further scaling as needed".

He also said broader interventions would be required as AI capabilities continue to advance.