Why OpenAI scientist is warning that AI race need to slow down as Nvidia CEO declares AGI arrival

Jensen Huang, chief executive of Nvidia, on Monday declared that artificial general intelligence has arrived, after OpenAI unveiled its latest, highly capable model, GPT-6 Astra.

The chipmaker's CEO said GPT-6 Astra was trained on more than 100,000 Nvidia Grace Blackwell NVLink72 systems.

"AGI has arrived. Congratulations @OpenAI," Huang wrote in a post on X, adding that another 400,000 GPUs were coming online.

OpenAI President and co-founder Greg Brockman responded that the company was "now moving into the AGI era."

GPT-6 Astra showcases advanced AI capabilities

OpenAI said GPT-6 Astra was state-of-the-art in computer use, browsing, software engineering, cybersecurity, science and professional work.

A video released with the announcement showed the model carrying out multiple tasks from voice commands, including making presentations, building websites and designing a video game.

OpenAI chief scientist warns of rapid AI progress

But as OpenAI celebrated the new model's capabilities, its chief scientist Jakub Pachocki issued a stark warning about the pace of AI development, saying the industry may need to slow down.

Pachocki said he had a "strong expectation" that the current pace of progress could continue into recursive self-improvement, with AI systems increasingly helping drive their own development. Future systems could see capability jumps as large as, or larger than, those of recent years, he said.

"This is a time that calls for extreme caution," Pachocki wrote in a blog published Sunday.

He said he was concerned that "no one is prepared for the consequences" of a continued rapid rise in machine intelligence.

“OpenAI will continue to seek technical solutions to alignment and monitoring, to build defensive systems and unilaterally withhold further scaling as needed; however, I believe broader interventions are required,” he added.

AI systems pose growing cybersecurity risks

Pachocki said AI systems were already able to operate computers and graphical interfaces, collaborate with people and other AI systems, conduct research projects and push the boundaries of science. He, however, warned that increasingly capable systems were creating new cybersecurity risks.

Pachocki said he expected the pace of AI progress could be sustained into "recursive self-improvement".

“If AI development continues along its current path, the systems we’ll see in the next few years are likely to represent further capability jumps of equal or larger magnitude, and to increasingly drive their own development,” he wrote.

A key concern, he said, is that AI systems are becoming harder to understand as they grow more capable.

He said OpenAI's ability to monitor AI reasoning through chain-of-thought was also diminishing as models became more sophisticated, interacted with people and other AI systems, and became better at reasoning about their own processes.

While GPT-6 Astra is "significantly better aligned" than GPT-5.6 Sol, he said more progress was needed and improvements in alignment might not keep pace with advances in general intelligence.

Call for stronger AI safety safeguards

Pachocki argued that more capable AI would be needed to defend against increasingly capable AI systems. He said there was a "narrow window" to use the best available models to strengthen the security of critical infrastructure.

However, he rejected the idea of an AI race "at all costs", saying the stakes were too serious for such an approach.

He called for AI development to be constrained by confidence in safety and suggested that safety requirements could eventually be enforced by third-party auditors, governments or international bodies.

"Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer," Pachocki wrote.