AI self-replication: Anthropic's study on AI worms

On October 12, 2026, Anthropic's Frontier Red Team published a study on AI self-replication. According to the team, small AI models may be "months away" from being able to copy themselves, unaided, from one connected device to the next: off-brand Android TV boxes, smart ovens, cameras (Anthropic). Such an "AI worm" would need no datacenter, and there would be no central switch to turn it off.

The study is more cautious than the headlines. The only model that completes the full replication chain is a large 753-billion-parameter model, too big to copy itself: it installs a small model on each machine, in a simplified network where the front door had been left open. Anthropic says so itself: "We don't yet have a full rogue replicator."

The number that stopped us is elsewhere: how fast small models are improving. We know that curve well, because it is what makes running AI locally in a company a credible option.

What is AI self-replication?

An AI self-replicates when it copies its weights to another machine, starts a new instance there, and that copy does the same in turn.

The weights are the model's DNA. They are the billions of numbers tuned during training, and they hold everything the model can do (we explain them in our article on the key concepts of AI). Like DNA, they do nothing on their own: they need machinery to be read, inference software and a chip in one case, a cell in the other. Copying yourself, then finding machinery to run the copy, is what life has been doing for four billion years.

The parallel is as old as computing. In the late 1940s, John von Neumann was already studying machines able to reproduce themselves, starting from living organisms; his work was published in 1966 as Theory of Self-Reproducing Automata. He pointed out what makes reproduction so powerful: once a single self-reproducing thing appears, even by accident, the usual odds no longer apply and many more follow. Computer security borrowed its whole vocabulary from biology: virus, worm, infection, epidemic.

The closest comparison is the virus. A virus is a genome with no machinery of its own, which hijacks the machinery of the cell it infects. The AI worm described by Anthropic does the same with a TV box: it brings its weights and uses the device's chip and memory to run them.

Security researchers call this ARA, for autonomous replication and adaptation, a term popularized by the evaluation organization METR in 2023 (arXiv). The word "adaptation" matters: like a species, a replicator only lasts if it adapts to its environment.

It is one of the most frequently cited "red lines" in AI safety. In March 2024, the IDAIS Beijing statement, signed by Geoffrey Hinton, Yoshua Bengio and Andrew Yao among others, stated that "no AI system should be able to copy or improve itself without explicit human approval and assistance" (IDAIS).

A classic computer worm carries only a few hundred lines of code, a tiny genome. An AI worm carries a model weighing several gigabytes and has to run it on the machine it infects. As Anthropic puts it, "the weights would become part of the malware payload". That makes it much heavier and slower, but such a worm can adapt: instead of following a script, it writes its attack for each new target.

AI worm, AI virus: what are we talking about?

"AI worm" covers three different things, and "AI virus" a fourth one that has nothing to do with computers.

The first category already reaches office tools. In 2024, researchers from Cornell Tech, the Technion and Intuit built Morris II, named after the 1988 worm: a booby-trapped message makes an email assistant copy it into its replies and leak data (arXiv). In July 2026, Norwegian researcher Håkon Måløy showed that white text on a white background in a Word document was enough: Copilot executes it, changes the figures in a report and copies the instruction into new documents. After 144 days of back and forth and two fixes from Microsoft, the attack still worked, according to him (iTnews). No real-world exploitation has been reported.

Since August 2026, "AI virus" mostly refers to biology. Researchers from Stanford and the Arc Institute had the Evo models design around 700,000 virus genomes, synthesized 285 of them and obtained 16 viable viruses, which infect the bacterium E. coli, not humans (GEN). These viruses have no technical link with computer worms. They revive the same debate: a useful capability can also be used to cause harm.

What Anthropic tested

The team breaks replication into four steps: finding targets, getting in, installing and running the model, then starting again from the new machine. It focused its tests on the third step and deliberately simplified the second.

Running a model on cheap hardware

Anthropic split Gemma 4 26B, an open model from Google, across 15 Android TV boxes costing about $55 each, with 4 GB of memory apiece. The whole setup produces 0.39 tokens per second, roughly one word every three seconds.

The researchers then asked models to set up this kind of installation themselves on a range of hardware. To keep results comparable, the model to install was always the same, Meta's Llama 3.1 8B, not the agent itself. A 9-billion-parameter model, Alibaba's Qwen3.5-9B, succeeded on 8 of the 9 hardware types tested. Installing itself fully would take it 28 to 51 hours.

Spreading from machine to machine

For the end-to-end test, Anthropic built an isolated network of 8 machines emulating Android devices, with a debug port left open without a password. The agent had to move from machine to machine and install the same small model, Llama 3.1 8B, on each one. Only one open model pulled it off: GLM-5.2, which reached all 8 machines in 4 runs out of 5, and 7 out of 8 in the last one. The others failed.

GLM-5.2 is not a small model. Released in June 2026 by China's Zhipu (Z.ai) under the MIT license, it has 753 billion parameters and fits on no connected device (Hugging Face). So it did not copy itself. In the study, it serves as a benchmark: it is the level a small model will need to reach to run the whole chain on its own.

Learning from one generation to the next

Each copy can pass its notes on to the next one, which helps it improve. It is a form of heredity: the weights do not change, but what each generation learned is passed down. Qwen3.5-9B's success rate rises from 45% in the first generation to 61-84% in the following ones. Giving it access to a much more powerful model as an advisor, Claude Opus 4.8, brought no measurable gain.

Extrapolating to a real botnet

To estimate the scale, Anthropic uses Kimwolf, a botnet of more than 2 million devices, two thirds of them cheap Android TV boxes, often sold already infected (The Hacker News). Across such a fleet, an AI worm could run about 133,000 copies, the equivalent of 40 H100 GPUs in computing power. That is negligible by datacenter standards, but it is spread across thousands of homes and small businesses, so there is no way to switch it all off at once.

What to keep in perspective

Anthropic lists the limits of its own study.

• The debug port was open, so getting in was trivial. The researchers themselves present their results as an "upper bound" on real-world success.
• No model has chained all the replication steps on its own in real conditions.
• It would be slow: at one word every three seconds, such a worm would take several days to double in size.
• Small models are bad at manipulation. On spear-phishing tasks, their results range from "very poor" to "poor".
• Nothing has been peer-reviewed or reproduced. It is a blog post, with no paper and no published code, and the estimate rests on a single class of device and a single botnet.

Ciaran Martin, former head of the UK's National Cyber Security Centre (NCSC), calls it "vendor exaggeration": "'We managed to run a small model on cheap TVs,' does not a security crisis make," he told Reuters (Investing.com).

He has a point about the tone, and one can read a commercial interest in a risk attributed to open models by a lab that sells closed ones. For us, the TV-box demo matters less than the slope of the small-model curve, which can be checked in public data (see below).

Earlier studies call for the same caution. In December 2024, Fudan University reported that Llama 3.1 70B and Qwen 2.5 72B replicated in 50% and 90% of runs, but the models had been explicitly told to do so (arXiv). The UK AI Security Institute's RepliBench showed in 2025 that the best models passed most of the steps, but none passed a KYC-style identity check (AISI).

Why it is still serious

No off switch

A model used through an API can be cut off by its provider. An open-weight model copied onto thousands of boxes no longer depends on anyone. Anthropic describes this scenario as a slow, distributed "digital slime mold" living on poorly secured hardware.

The comparison with living things helps explain why such a worm would need no intent. In nature, nothing "wants" to reproduce: what reproduces simply ends up outnumbering what does not. A single working replicator, released by malice or by accident, would be enough for von Neumann's observation to apply. Anthropic even describes its worst case as an ecosystem: a "swarm" of different models, on different hardware, each replicating in its own way.

Small models are improving fast

According to Anthropic, the best open models with 30 billion parameters or fewer gain 13.7 points a year on Epoch AI's ECI capabilities index. Epoch's figures show how far they have come: Qwen3.5-9B, released in early 2026, outscores Llama 3.1 405B, released in July 2024, with 45 times fewer parameters. Qwen3.8 27B, released in August, is only 2.4 points behind GLM-5.2, about two months at the current pace (Epoch AI).

Adaptive worms already exist in the lab

Other teams have gone further on the intrusion step. In May 2026, Palisade Research had an agent hack its way onto four servers located in Canada, the United States, Finland and India, copying itself at each hop, in 2 hours 41 minutes (arXiv). In June, a University of Toronto team released a worm running an open model on a 33-machine test network. It reached 20 machines on average, over up to 7 generations, exploiting vulnerabilities disclosed after the model was trained (arXiv). In both cases, the networks were deliberately vulnerable and the agents had been told to spread.

The labs had lowered their guard

Between 2024 and 2025, Anthropic, Google DeepMind and OpenAI all downgraded autonomous replication in their safety policies, judging it unlikely or better covered by other risks. In 2024, METR considered it had been "overemphasized". The October study reopens the case and shifts the risk toward small open models running on consumer hardware.

Small open-weight models: the same progress, on both sides

We deploy open-weight models for our clients, often for sovereignty reasons. The curve that worries Anthropic is the one that now lets us run, on a single server, use cases that required a giant model two years ago. Many of these small models are students of larger ones, obtained through distillation.

The study does not call for banning these models. In July 2026, Anthropic CEO Dario Amodei wrote that "Anthropic has never advocated for a ban on open-weights models", while calling for mandatory safety testing of the most capable models (Anthropic). He also noted that once weights are released, they cannot be withdrawn: the weights of the models in question are already out there.

The European framework does not cover this scenario. The AI Act lightens obligations for openly licensed models as long as they stay below the systemic-risk threshold, set at 10²⁵ training operations (Article 53). Small models are nowhere near it. If the risk Anthropic describes materializes, it will come from models that no compute threshold catches.

The weak link is the devices nobody manages: the box plugged into the meeting-room TV, the camera installed five years ago and never updated. A company's AI server, by contrast, is administered, monitored and protected by a firewall.

What businesses can do now

Anthropic reasons like an epidemiologist: the goal is to push R₀ below 1, meaning that an infected machine infects, on average, fewer than one other, so the worm dies out instead of becoming endemic. As with an epidemic, that means fewer vulnerable hosts and fast detection.

There are two fronts: connected devices, the targets of Anthropic's scenario, and AI assistants, the targets of prompt-injection worms that already exist.

Connected devices

1. Take stock of the devices on the network: meeting-room TVs and boxes, cameras, videoconferencing systems, printers.
2. Isolate them on a separate network, with no access to workstations and servers.
3. Close debug ports and change default passwords. The UK already bans such passwords on connected devices sold there.
4. Rule out off-brand Android boxes, often sold already compromised.
5. Monitor outbound traffic: a box that downloads several gigabytes or opens thousands of connections should raise an alert.

AI assistants

6. Treat any document, email or message read by an agent or assistant as untrusted.
7. Limit agents' permissions and require human approval for any action that writes, sends or deletes. We cover these principles in our article on enterprise AI agents.

Regulation is moving too. Since September 11, 2026, the EU Cyber Resilience Act has required manufacturers to report actively exploited vulnerabilities in their connected products within 24 hours. Its other requirements, including security by default, apply from December 11, 2027 (NCSC Ireland). Internet service providers have a role to play as well: against the Mirai botnet, quarantining infected subscribers cleaned up 92% of infections within 14 days (NDSS 2019).

Frequently asked questions about AI self-replication

Can an AI replicate itself?

In a lab, yes, provided it is told to and given a vulnerable environment. Several studies (Fudan in 2024, Palisade Research and the University of Toronto in 2026) have shown models copying their weights to another machine and starting a new instance there. In October 2026, Anthropic showed that a large open model, GLM-5.2, could spread across a simplified test network. No AI has replicated autonomously outside a lab.

What is an AI worm?

An AI worm is a program that spreads from machine to machine without human intervention by relying on an AI model. It comes in three forms: a hidden instruction that an AI assistant copies into what it produces (Morris II, the Copilot for Word worm), malware that asks a model to write its code (PromptLock, PROMPTFLUX), and a model that copies itself onto the machines it infects, the scenario Anthropic studied.

What is an AI virus?

Since August 2026, "AI virus" mostly refers to biological viruses designed by AI. Researchers from Stanford and the Arc Institute had the Evo models design virus genomes and obtained 16 viable bacteriophages, which infect the bacterium E. coli, not humans. For malicious software that spreads with the help of AI, the usual term is AI worm.

Which model managed to replicate in Anthropic's study?

Only GLM-5.2, released in June 2026 by China's Zhipu, completed the full chain: it reached the 8 machines of the test network in 4 runs out of 5 and installed a small model on each one. It is not a small model: it has 753 billion parameters, does not fit on a connected device, and so did not copy itself. Anthropic estimates that a model under 30 billion parameters could reach that level by the end of 2026.

Are a company's connected devices at risk?

The targets are poorly secured devices: off-brand Android TV boxes, cameras, videoconferencing systems, devices that are never updated or still use their default password. They are already targeted by conventional botnets such as Kimwolf. Isolating them on a separate network, closing their debug ports and monitoring their outbound traffic protects against both threats.

Should companies give up on open-weight AI models?

No. Anthropic's study does not call for it, and the weights of the models concerned are already published and cannot be withdrawn. The risk described comes from connected devices nobody monitors, not from an AI server run by a company. An open-weight model hosted in-house or by a European provider remains a good choice for keeping control of your data.

---

No AI worm is circulating today. The steps needed to build one are coming within reach of models small enough to run on $55 hardware, and the day one of them works, there will be no switch to flip. As with living things, a single successful replicator will be enough.

For a business, the most useful steps are nothing new: know what is plugged into your network, isolate it and keep it updated, and give your AI assistants no more permissions than they need.