Anthropic paused next-gen AI model training for over a month over security concerns

Anthropic delayed pre-training a next-generation artificial intelligence model for more than a month to strengthen safeguards against the theft of its model weights and intellectual property, the company disclosed in a regulatory filing.

“We recently chose to delay pre-training a new frontier base model for more than a month to allow time to further enhance our security measures related to preventing model weight exfiltration and protecting our intellectual property,” the AI company said in its preliminary initial public offering (IPO) registration statement filed with the US Securities and Exchange Commission on Monday.

Anthropic warned that safety and security measures could slow model development, delay product releases and weaken its competitive position against rivals that do not adopt similar constraints.

The disclosure comes amid growing concern among AI researchers that competition to develop increasingly capable systems could outpace efforts to ensure adequate safety measures.

Jacob Coxon, a researcher who spent three years conducting pre-training research at Anthropic and OpenAI, resigned from Anthropic in September, saying both companies were failing to adequately address the risks of developing advanced AI.

"They are racing straight to self-improving superintelligence and gambling with our lives," Coxon wrote in a series of posts on X.

Evan Hubinger, Anthropic's alignment science lead, backed Coxon's concerns, saying he believed there was a greater than 10% chance that AI could kill all humans within the next decade.

"I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to," Hubinger wrote.

However, Anthropic's filing does not link the month-long training delay to these existential-risk concerns.

Security risks complicate AI development

Anthropic said it considers safety throughout the development of its models, rather than assessing risks only before their release.

"For models we anticipate will have novel capabilities or present novel safety risks, we plan to evaluate the safety of each stage of training, including pretraining, post-training, and internal deployment," the company said in the filing.

It added that it may "delay or pause a stage when we are not confident in our safety measures for that stage".

The company, however, clarified that there was no guarantee its safeguards would prove adequate, that it would correctly identify which capabilities required restrictions or that the constraints it imposed could not be circumvented.

Anthropic also warned that the potential severity of harm from misaligned or misused AI models increases as their capabilities grow.

Anthropic said precautionary measures, including delaying model releases and limiting initial availability, could become more frequent and costly as AI capabilities advance.

AI safety debate

The disclosures come as researchers and executives warn that advances in AI could accelerate into a cycle of recursive self-improvement, in which systems increasingly assist in developing more capable successors.

OpenAI chief scientist Jakub Pachocki has warned that the current pace of progress could continue into recursive self-improvement and called for extreme caution, saying he was concerned that the industry was unprepared for the consequences of a continued rapid rise in machine intelligence.

Anthropic itself says it increasingly uses its models to accelerate research and development, including writing code and helping develop subsequent generations of AI systems.

The company calls this process "closing the loop" and says it can shorten development timelines.

But it has also warned that its growing reliance on AI-assisted research could cause model capabilities to advance faster than its ability to develop, validate and implement corresponding safety, alignment and security measures.

The accord calls on companies to monitor their models during training and deployment for risks involving cybersecurity, biosecurity and chemical threats, among other areas. It also calls for internal teams to oversee safeguards, independent external assessments and board-level oversight.

The document says such measures should help ensure that AI systems operate as intended and that problems are identified and addressed promptly.

It also leaves open the possibility that the measures could eventually be incorporated into laws or regulations.