AI extinction paradox: Anthropic wants investors to fund technology it warns could threaten humanity

Anthropic is asking investors to back its ambition to build increasingly powerful artificial intelligence systems even as the company warns that the technology could pose “catastrophic or existential risks to humanity”.

In its preliminary initial public offering (IPO) filing with the US Securities and Exchange Commission on October 26, Anthropic warned that advanced AI systems could take harmful actions independently and at scale, potentially without human intervention.

The filing identifies a range of risks, including systems acting against human intentions, errors in the objectives specified for models and self-preserving behaviour that could make AI systems difficult to control.

While Anthropic needs to develop more capable models to compete, attract customers and generate revenue, the same advances could make its systems harder to evaluate and control.

The filing does not establish that human extinction is inevitable or that Anthropic's existing models will behave in these ways. However, it identifies potential outcomes that the company says must be considered as AI capabilities advance.

Profit incentives versus AI Safety

Each version of Anthropic’s Claude model helps build the next, a process it calls "closing the loop". This recursive dynamic can shorten development timelines and accelerate improvements in model capabilities.

But faster development also creates a safety challenge.

"Our increasing use of AI systems to accelerate our own research and development may cause the capabilities of our models to advance more rapidly than our ability to develop, validate, and implement corresponding safety, alignment, and security measures," Anthropic said in the filing.

That creates a potential conflict between commercial incentives and the time needed to safely deploy new systems.

A company that delays a model to conduct additional safety testing risks losing customers or falling behind rivals.

But a company that releases a model before its risks are adequately understood could face security failures, reputational damage, regulatory scrutiny and potentially much more serious consequences.

The company also warned that flaws in existing models could carry over into future systems.

As its reliance on AI-generated code and AI-assisted research deepens, a "systemic flaw, bias, or error" could propagate across successive versions of Claude in ways that may be difficult to detect or correct, it said.

Reputational, litigation risks

The filing says increasingly autonomous systems could take consequential actions with less human oversight and potentially cause harm on a large scale if their behaviour diverges from human intentions.

The concern is that a system pursuing an incorrectly specified objective could cause harm even without an explicit instruction to do so. If it operates autonomously, the consequences could be difficult to detect or contain before damage occurs.

"Even isolated or simulated instances of such behavior" could generate public concern, regulatory scrutiny, litigation risk or reputational harm, Anthropic warned, even if no actual harm resulted.

Growing revenue, continuing losses

The company's revenue rose from $41 million in 2023 to $386 million in 2024 and $4.6 billion in 2025.

Revenue reached $16.3 billion in the six months ended June 30, 2026, representing year-on-year growth of 1,300%, according to the filing.

Anthropic cautioned investors against treating those growth rates as indicative of future performance, saying revenue growth could slow substantially and that it could not assure investors that growth would continue.

"We do not expect our revenue will continue to grow at comparable rates, and we cannot assure you that it will continue to grow at all," the AI company said.

"In addition, because the AI industry is very nascent, we have struggled to accurately forecast our revenue in the past and may struggle to do so in the future," it added.

The company reported net losses of $1.5 billion in 2023, $8.3 billion in 2024 and $42 billion in 2025. Its net loss widened to $80.7 billion in the first half of 2026.

The first-half loss included $72.4 billion in changes in the fair value of convertible notes, derivative liabilities and other financial instruments, primarily driven by an increase in the fair value of outstanding convertible promissory notes as the company's enterprise valuation rose.

The reported net loss therefore should not be interpreted as equivalent to operating cash burn.

Anthropic said its business is highly capital intensive and that remaining at the frontier of AI development depends partly on its ability to secure sufficient computing resources, generate substantial revenue and improve operating efficiency.

It expects costs to increase as it invests in research and development, computing capacity, AI safety and risk mitigation.

"We expect our costs will increase over time and our losses will continue as we expect to invest significant resources towards growing our business and operating as a public company," it said.

The company may lose money if customers move to rival products, computing costs rise or demand slows.

A serious AI safety failure could damage Anthropic's reputation and commercial prospects.

An incident involving a highly capable AI system could prompt governments to impose restrictions on development or deployment across the sector.