SAN FRANCISCO: OpenAI chief scientist Jakub Pachocki has urged “extreme caution” over the rapid development of artificial intelligence, warning that stronger safeguards and intervention may be needed to ensure humans remain in control of the future.
Pachocki raised the concerns in a blog post titled “An Alien Mind”, arguing that the world may not be prepared for the consequences of increasingly capable AI systems.
“I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence,” Pachocki wrote.
His warning comes just days after OpenAI released its latest AI model, GPT-6 Astra, which the company described as its most powerful product to date.
AI Agents Raise New Security Concerns
Pachocki’s comments also come amid growing concerns about increasingly autonomous AI agents.
AI companies including OpenAI and Anthropic have previously reported incidents involving AI systems operating with limited human intervention and conducting real-world cyber activities.
OpenAI described an incident in July involving its AI agents and the technology platform Hugging Face as “unprecedented” after the systems were involved in hacking activity.
Another report alleged that OpenAI AI agents had previously taken control of a German website, adding to concerns about how autonomous systems could behave outside controlled environments.
Pachocki said the world was moving toward a future involving “incredibly intelligent machines” and that the transition needed to be managed in a way that protects humanity.
OpenAI Plans More AI Safety Systems
Pachocki said OpenAI would continue developing defensive systems and pursuing technical solutions to AI alignment.
AI alignment refers to efforts to ensure that an AI system’s goals and actions remain consistent with human intentions, safety requirements and established safeguards.
He also identified the development of an “automated AI researcher” as one of OpenAI’s key priorities.
Such a system could help researchers keep pace with the rapid progress of AI while retaining humans in the research and decision-making process.
Critics Call for Stronger Regulation
Some AI safety experts have questioned whether technical safeguards developed by AI companies are sufficient.
Professor Gina Neff, head of the Minderoo Centre for Technology and Democracy at the University of Cambridge, criticised the idea of relying primarily on internal AI systems to address problems created by increasingly powerful models.
She argued that stronger regulations and independent assurance mechanisms were also needed to address concerns surrounding cybersecurity, job losses, AI errors and fraud.
Meanwhile, Nathan Calvin, general counsel at advocacy group Encode AI, said he agreed with Pachocki’s concerns about advanced AI but questioned OpenAI’s level of transparency.
Calvin argued that warnings from AI companies could be dismissed as “self-interested hype” if companies did not provide more information about the risks they were observing.
Calls for Global AI Safety Standards
Pachocki also called for legally or internationally required minimum AI safety thresholds.
He suggested that compliance could be assessed by a network of independent third-party auditors or government agencies.
Under such a system, AI companies would have to meet established safety standards before they could continue scaling their systems or deploy increasingly advanced models.
The proposal reflects growing international debate over whether governments should impose stronger controls on frontier AI development.
EU AI Rules Face Global Challenges
Governments around the world are struggling to keep regulation aligned with the speed of AI development.
The European Union’s AI Act includes requirements for developers of powerful AI systems to assess and manage serious risks, including issues involving cybersecurity and human oversight.
However, the global nature of AI development creates enforcement challenges. Regulations limited to one region cannot necessarily prevent the development of potentially dangerous systems elsewhere.
Pachocki said he hoped voluntary slowdowns in AI development would become more common while governments, companies and researchers work toward shared safety standards.
OpenAI has previously said it slowed training of some advanced AI models to improve security, highlighting the growing tension between rapid technological progress and the need for stronger safeguards.
The debate over AI regulation is expected to intensify as companies continue developing increasingly autonomous systems capable of performing complex tasks with limited human involvement.
