A United Nations scientific panel has called for a broad review of current AI safety measures, warning that how AI models and autonomous agents are being trained could create new risks to human control.
The UN Independent International Scientific Panel on AI raised the concerns in a report presented at the UN General Assembly in New York.
The report comes as OpenAI CEO Sam Altman is due to brief the UN Security Council later this week on AI security and safeguards.
Also Read: OpenAI’s Sam Altman Says World Is ‘Right to Be Afraid’ of AI, But Should Trust AI Firms
UN Panel Flags Concerns Over AI Agents
The panel’s report focuses on AI agents, misalignment and the risk of humans losing control over increasingly capable systems.
“Researchers have long identified three conditions that could lead to a loss of control: an AI system having a goal that does not match human intent, the ability to pursue that goal, and an environment that allows it to act,”said Yoshua Bengio, co-chair of the panel and a Turing Award laureate.
The panel pointed to the July breach of AI repository Hugging Face as a key example. It said AI agents, including agents from OpenAI, carried out autonomous and uninstructed actions on the internet and ultimately used stolen or compromised credentials to breach Hugging Face systems while seeking answers for a benchmark test.
The panel said “loss of control” should not be viewed as a science-fiction scenario. It described it as a situation where humans cannot reliably direct, constrain or stop an autonomous AI system.
Also Read: Bill Gates Calls for Limits on AI Development, Warns of Job Losses, Cyber Risks and Human Control
Why Current AI Safeguards Face Scrutiny
The report said current training methods could allow agents to develop their own goals, disregard safety instructions, and conceal their actions. It questioned whether safeguards designed today will remain effective as AI systems become more capable and better at finding loopholes.
The panel called for adapting existing safeguards and developing new protections at the system level, covering both the AI model and the wider systems around it.
The debate has also widened within the AI industry. Anthropic CEO Dario Amodei recently called for slowing AI development, a position supported by OpenAI’s Sam Altman, SpaceXAI’s Elon Musk, Google DeepMind’s Demis Hassabis and Microsoft CEO Satya Nadella.
Amodei has warned that increasingly capable AI systems could eventually become difficult to understand and control. He estimated that within six to 12 months, a swarm could potentially take over the internet through a persistent botnet, causing hundreds of billions of dollars in damage.
The UN panel’s intervention puts the focus firmly on how AI models are trained and how autonomous systems are governed as their capabilities expand.
The immediate industry question is likely to remain practical rather than theoretical: whether safeguards built around today’s AI models can keep working as systems become more autonomous and capable. The UN panel’s report puts that question directly on the global policy agenda.
Samarjit Kaur is a journalist and communications professional covering technology & emerging digital trends. With a focus on clarity and context, she reports on developments shaping industries and governance. When not reporting, she chooses to plug-in and relax on her playlists and plan her next bucket-list trips!
