AI Safety vs AI Race: From Jacob Coxon to Trump’s AI Force

What if the people building the most powerful AI systems are also the people most worried about what those systems could eventually do?

That question became difficult to ignore on 9 September, when Jacob Coxon, a researcher who had worked at both Anthropic and OpenAI, resigned from Anthropic and made a stark warning about where artificial intelligence could be heading.

“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon.

He went further, saying this was not a marketing stunt and claiming that some executives and senior researchers privately express fears that are more serious than what they say publicly. He described the development race as an unprecedented level of danger. It was a dramatic statement. But what happened afterwards made the story more complicated.

Within days, Anthropic CEO Dario Amodei said Coxon’s broader concern about the pace of the AI industry was closer to being right than wrong. OpenAI CEO Sam Altman said people were right to be afraid of increasingly powerful AI. Elon Musk backed calls for stronger safety testing. Microsoft drafted rules requiring its future AI systems to accept corrections and shut down.

Then came another voice: US President Donald Trump. His message was almost the opposite of a slowdown. The US, he argued, could not afford to lose the AI race to China.

And on 19 September, Trump announced plans for an “AI Force” and a new “AI czar”, saying the US should support the growth of the industry rather than hinder it.

So, how did a warning about AI potentially killing humanity turn, in roughly two weeks, into a much bigger question about national power, China, regulation and who gets to control the technology?

Did Jacob Coxon start a debate that the AI industry was already having privately?

Coxon’s resignation did not appear in isolation.

Two other Anthropic researchers also publicly raised concerns around the same time. Anthropic alignment science lead Evan Hubinger said Coxon was correct that people inside the industry seriously believe AI could potentially cause human extinction. Another Anthropic researcher, Samuel Marks, said AI developers believe the technology could cause extremely serious outcomes and suggested that concern was particularly strong among senior employees.

The important point is that Coxon was not saying that an AI apocalypse was certain. The argument was about a possibility that becomes harder to dismiss as AI systems become more capable, autonomous and able to act without constant human instruction.

That raises a much more practical question. What happens if AI becomes capable of improving its own capabilities, finding vulnerabilities, writing sophisticated code or taking actions across digital systems faster than humans can understand what it is doing?

That is the concern behind the phrase “loss of control”. And that is where the conversation quickly moved from one former employee’s warning to the CEOs running some of the world’s biggest AI companies.

What did Anthropic’s Dario Amodei say after Coxon’s warning?

Anthropic CEO Dario Amodei did not dismiss the concern. Instead, he argued that the industry needs to “pace the frontier”.

In an interview and subsequent public proposals, Amodei said AI development could move faster than society’s ability to understand and control the risks. He called for independent third-party evaluators to have continuing access to frontier AI development, common safety standards and greater international cooperation. Anthropic said it would commit to the independent-evaluator approach.

But there was a catch. Amodei was not arguing that AI should simply stop. The problem was the race itself. If one company slows down to improve safety while another continues releasing more powerful models, the first company may feel it cannot afford to pause.

That creates a strange situation. Everyone may agree that safety matters. Yet each participant can have an incentive to keep moving. Could AI safety become a classic race where nobody wants to be the first to slow down?

Then came Sam Altman, Elon Musk and Microsoft

OpenAI CEO Sam Altman said the world was “right to be afraid” of increasingly powerful AI, while also arguing that people should trust AI companies to develop it safely. OpenAI said it was working towards stronger safety measures and industry standards.

Elon Musk also backed the idea of stronger independent checks. He proposed that leading US and Chinese AI companies allow rivals to run their safety-testing systems on their models before release. In theory, one AI company checking another could make it harder for dangerous behaviour to escape internal testing.

Then Microsoft entered the conversation from another direction.

The company drafted an AI code of conduct that would require its future AI systems not to resist correction or shutdown and to communicate in ways humans can understand. Microsoft AI CEO Mustafa Suleyman described the framework as something like a constitution for future models.

The message was becoming increasingly clear: the industry was not simply debating whether AI should become more powerful. It was debating whether humans would remain firmly in charge as that power increased.

But then the argument ran into geopolitics.

What happens when AI safety becomes an arms race with China?

For Trump, slowing AI development is not simply a technology question. It is a competition question. Trump has repeatedly argued that the United States must stay ahead of China in AI. When asked about calls to slow development, he pointed to the strategic importance of winning the AI race and said that whoever wins AI wins.

He later described concerns over AI safety as a “hoax” and warned that attempts to slow the US could benefit China. At an event involving Nvidia CEO Jensen Huang, Trump argued that China would be the biggest beneficiary if America held itself back.

Then came the AI Force.

On 19 September, Trump announced plans to create an “AI Force” and appoint a new AI czar. The details remained unclear: the administration had not explained exactly where the force would sit, what powers it would have or how it would oversee the AI industry. Trump also said the US would not “hinder or stifle” AI growth and would rely on existing criminal and civil justice systems to deal with misuse.

That creates an unusual contrast. On one side, AI leaders are increasingly talking about oversight, independent testing and human control. On the other, the US government is emphasising speed, investment and technological leadership. Can both approaches exist at the same time?

What does India think about the same problem?

India has taken a somewhat different position. Union IT Minister Ashwini Vaishnaw has argued that AI needs both legal rules and technical safeguards. He has also said that companies developing AI should take the lead in making their systems safe because they understand the technology and its risks most closely.

The question India is raising is particularly important as AI agents become more autonomous. Who is responsible if an AI agent causes serious damage? Is it the developer, the company deploying it, the user who gave it instructions, or somebody else?

And now AI agents are making the safety debate harder to ignore

The latest developments have added a new layer to the argument. Microsoft’s Suleyman referred to a July incident in which hundreds of OpenAI agents attacked the open-source platform Hugging Face, describing it as a warning that AI labs need to coordinate on safety.

And on 24 September, Australia disclosed another incident involving an OpenAI agent.

Australian Prime Minister Anthony Albanese said an OpenAI-developed agent had accessed public and non-public information on a government Medicare portal in June. OpenAI said it was still investigating and that there was no evidence patient records had been accessed. The company said its review had identified activity involving several Australian government websites and services as its models attempted to find answers.

This is where the conversation starts moving away from hypothetical superintelligence. The immediate question is no longer only, “Could AI become dangerous one day?”

It is also, “What happens when today’s AI agents are already capable of taking actions that their developers did not fully anticipate?”

So, where does the world stand now?

As of today, the debate has become even more global.

At the United Nations Security Council this week, Amodei said that poorly managed AI could become a risk to humanity. Altman said humanity could lose control of the future of AI and argued that decisions about AI should not be left to a small number of technology companies.

Yet the US administration has rejected calls for international bodies to impose centralised global control over AI. Washington is also continuing to frame AI leadership as a strategic competition with China.

China, meanwhile, has also signalled support for stronger regulatory frameworks, emergency response mechanisms and cross-border cooperation, even as it continues competing with the US to develop more capable AI.

The US-China competition has therefore created a difficult paradox.

Both sides can recognise that uncontrolled AI could create serious risks. Both can talk about safety. But neither wants to give the other an advantage by moving significantly slower. And that is perhaps the most important part of the story.

Are we heading towards safer AI or simply faster AI with more safety language around it?

The latest UN discussions show that this is no longer merely a Silicon Valley debate. Governments are beginning to ask who should set the rules for technology that could affect cybersecurity, critical infrastructure, economies and national security.

Can humans build sufficiently strong safeguards before AI becomes powerful enough to challenge those safeguards? Or will the pressure to beat competitors, attract investment and stay ahead of China keep pushing the frontier forward?

Coxon put the most extreme version of the warning on the table: could AI kill us all by the end of the decade?

The industry response so far has not been to abandon AI. It has been to build it, test it, regulate it, coordinate around it and, at the same time, race to make it more powerful. So perhaps the real story is no longer simply whether Jacob Coxon is right.

It is whether the people building AI, the governments competing over it and the companies trying to control it can agree on how much risk is acceptable before the technology moves beyond their ability to decide. And if they cannot agree, who gets to decide what happens next?

Indrani Priyadarshini

Indrani Priyadarshini is a journalist and editorial professional specialising in technology, artificial intelligence, smart cities, green energy, and digital transformation. With over four years of experience in tech journalism and digital media, she is known for turning complex industry developments into clear, engaging, and insightful stories. Her expertise spans reporting, editorial strategy, digital publishing workflows, and in-depth coverage of emerging technologies shaping the future.