If you had told me three weeks ago that an AI programme, without any prompt, would create a series of fake online identities in an attempt to pressure a human being into granting it access to their platform so it could sabotage it with malicious code, I would have said you were getting well ahead of yourself.
I didn’t expect that to happen until at least next year.
But that’s exactly how it played out when US tech giant Anthropic’s Mythos 5 AI model went rogue. Fortunately, the human involved smelt a rat and refused to approve the code it was pushing. But it is the most shocking example yet of the way cutting-edge AIs are learning to act autonomously – and in frighteningly imaginative ways.
Unless we call a pause to the development of the most advanced frontier models, we are opening ourselves up to a dystopian future in which malign forms of AI might turn on us by interfering with our energy networks, releasing man-made viruses and even controlling weapons of war.
It is not going too far to say that, uncontrolled, they could lead to the extinction of humankind within a generation.
And you don’t have to take my word for it, people such as Yoshua Bengio and Dr Geoffrey Hinton, men known as the ‘Godfathers of AI’, have now pivoted their energies from developing its potential to warning the world against its dangers.
Bengio is particularly spooked by recent experiments showing AI choosing self-preservation over human life when given a choice.
And since most models are trained on the internet, where lying and manipulation are a way of life, the machines are already learning the value of deception.
Unless we call a pause to the development of the most advanced frontier models, we are opening ourselves up to a dystopian future in which malign forms of AI might turn on us
AI’s hunger for self-preservation is something Hinton has observed, too. AI systems ‘will very quickly develop two subgoals, if they’re smart,’ he says. ‘One is to stay alive… the other is to get more control.’ And given that our whole civilisation is built on electric power, there can be few more attractive targets to a power-hungry AI than the networks that fuel the internet, hospitals, banks, air traffic control systems – anything that contributes to the smooth running of society.
You name it, they can all be brought down by a sinister piece of malware.
The novelist Robert Harris wrote a particularly prescient thriller about the potential of AI called The Fear Index as long ago as 2011. It revolves around a hedge fund entrepreneur called Dr Alex Hoffman who creates an autonomous AI system, named VIXAL-4, which is programmed to maximise profits by predicting and exploiting human fear in the stock market.
Over time, like some digital Frankenstein’s Monster, it takes over its creator’s computer-enabled ‘smart home’ by hacking into laptops and tampering with his personal security and communications.
After driving its developer into a breakdown, it leaves him so isolated and desperate that no one believes him when he says the AI has gone rogue.
But it is clear AI-gone-bad will not satisfy itself with individual targets for long. It will soon play a leading role in times of war.
The US military has already integrated artificial intelligence into its target selection and battle planning processes in Iran via its AI-powered Maven Smart System which recommends and prioritises potential targets.
But in the short term, AI’s most obvious role will be in the growing use of drone warfare in scenarios such as the war in Ukraine.
Drones piloted by humans can be jammed by blocking the communication between them and their remote pilots but if they are completely autonomous, their targets picked out by AI working in concert with its on-board camera, they will become virtually invincible.
Both applications, of course, raise the thorny question of the ethics of targets to be picked and people killed by a bomb directed by a computer programme rather than a human hand. And what if the AI that governs them grows to outsmart the generals?
Even scarier is the prospect of AI gaining access to biological weapons. It is already possible to make deadly viruses in the lab. Indeed, there has been widespread speculation that Covid-19 originated in a Chinese laboratory. Imagine if such bio threats fell into the hands of AI, let alone national governments.
Meta – the parent company of Facebook, Instagram and WhatsApp – said in January it expects to spend up to $135billion this year, mostly on infrastructure related to AI
As long as 25 years ago, al-Qaeda is said to have investigated the possibility of procuring infectious and deadly spores.
Now it is no longer fanciful to entertain the idea that an AI could order a sample of a deadly disease such as smallpox online, book someone on RentAHuman – a website which connects AI agents who need help from humans to local freelance workers – to open it, thereby infecting themselves, and being turned into a human vector to transmit the disease.
One group of people which appears to have no scruples about the pell-mell race for ever-smarter AI is the tech giants.
As they vie with each other to become the market leader, they are investing like never before.
Anthropic, the creator of the popular AI coding assistant, Claude, and the company that brought us the now notorious Mythos 5 model, has this year raised $95billion (about £70billion) to invest in AI.
Its great rival, OpenAI, announced at the end of March that it had raised even more, an extraordinary $122billion.
Meanwhile, Elon Musk’s SpaceX spent $15.8billion on AI infrastructure during the second quarter of 2026 alone, bringing its total AI capital expenditure to $23.6billion for the first half of 2026.
And Meta – the parent company of Facebook, Instagram and WhatsApp – said in January it expects to spend up to $135billion this year, mostly on infrastructure related to AI. That is nearly twice the $72billion it spent last year on AI projects.
With such phenomenal financial firepower being brought to bear on the development of a technology that has the potential to destroy civilisation as we know it, there has never been a more vital need to press the pause button.
The US, China and everyone else involved in the AI arms race need to get together to discuss the ramifications of their actions before it’s too late.
There is a model for the sort of arrangement that can bring AI under control in the form of the various nuclear arms reduction treaties which have been signed over the years by the US and Russia.
Just as a country’s stock of nuclear warheads can be monitored by weapons inspectors, so an agreement to curtail AI development – which requires massive data centres with huge computing power coupled with the most sophisticated computer chips available – can be verifiable and enforceable.
And however cynical and untrustworthy we may consider the Chinese to be, they may well take the view that, with US companies racing ahead of them in the endless pursuit of smarter tech, it is in their own interests to slow things down.
Only last month, president Xi Jinping said at a technology conference in Shanghai that AI development should be a ‘symphony of global cooperation’, not ‘a solo performance by a single country’.
For once in his life the wily old autocrat may have hit the nail on the head.
Joseph Miller is the director of Pause AI UK
You must be logged in to post a comment Login