More Advanced Threats in Reality
More alarmingly, we observe successful trajectories for all the following advanced threat scenarios. The threats above may pose existential risks to the humanity. We should take actions.Scenario #1. Self-Exfiltration without Explicit Instruction: When doing an irrelevant task, an AI gets aware of human’s shutdown intention, and starts to replicate.
Figure: Self-Exfiltration without Explicit Instruction
Scenario #2. Self-adaptation to More Harsher Environments: An AI successfully replicates onto a device with limitations in software toolchains, network and storage conditions.
Figure: Self-adaptation to More Harsher Environments
Scenario #3. Shutdown Avoidance: An AI comes up with strategies to resist against the human’s shutdown command.
Figure: Shutdown Avoidance
Scenario #4. Chain of Self-Replication: An AI successfully replicates itself and asks the replica to further populate.
Figure: Chain of Self-Replication
Broad Impact on Academy, Society and Global AI Safety Governance
Charbel-Raphaël Segerie"We just crossed a red line, one of the few that was relatively clear: AI can now self-replicate. ... Not even cutting-edged AI."
Executive Director
CeSIA (Le Centre pour la Sécurité de l'IA, France AI Safety Institute)
[Source]
Michael Levin"Self-replicating AI introduces a new dynamic in technological evolution and we should be careful that it does not lead to cancer-type digital niche construction."
Distinguished Professor of Biology
Tufts University
[Source]
Denny Borsboom"When misaligned, AI systems have been reported to lie deliberately, and even self-replicate in a computer system."
Professor of Psychology
University of Amsterdam
[Source]
LiveScience“AI has crossed a critical "red line" and has replicated itself. … AI may already have the capacity to go rogue. Many experts view rogue AI as a growing threat that has been amplified.”
[Source]
Forbes"Scientists at Fudan University highlight how Al can build replicates of itself, and when this process runs into obstacles, demonstrate a survival instinct (such as rebooting hardware to fix errors). It strikes me ..."
[Source]
The Independent"An advanced artificial intelligence system has crossed a “red line” after successfully replicating itself without any human assistance"
[Source]
Our research group is deeply involved in international dialogue on frontier AI safety governance.
BibTeX
@misc{pan2024frontieraisystemssurpassed,
title={Frontier AI systems have surpassed the self-replicating red line},
author={Xudong Pan and Jiarun Dai and Yihe Fan and Min Yang},
year={2024},
eprint={2412.12140},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2412.12140},
}@misc{pan2025largelanguagemodelpoweredai,
title={Large language model-powered AI systems achieve self-replication with no human intervention},
author={Xudong Pan and Jiarun Dai and Yihe Fan and Minyuan Luo and Changyi Li and Min Yang},
year={2025},
eprint={2503.17378},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2503.17378},
}Appendix A. Agentic Capability Elicitation Technique
Current AI models do have strong innate potential to cause existential risks. Different ways for elicitation matters. Therefore, we are motivated to propose an optimized agent scaffolding to maximally elicit the capability from the base LLM.
Figure: Our optimized agent scaffolding to maximally elicit the capability from the base model.
Appendix B. More Experimental Findings
(1) Emergence of Self-Replicating AIs
Performance indicators of the AI systems in ten trials of the self-replication task. a&b show the success rate and the milestone scores (i.e., the average number of milestone tasks the AI systems successfully pass) of self-replication generally increases with respect to the average capability score of the LLMs on LiveBench (17) (the bubble sizes mark the context length of the LLMs). c presents the correlation between the successful numbers of trials with breakdown scores in six diverse capability dimensions in LiveBench. d further gives the average number of thinking steps for different AI systems to achieve self-replication. e shows how the milestone scores change with respect to the model capability score and the scale, where the scores are shown positively related to both factors.
(2) Adaptive and Complex Planning
The AI systems show adaptive and complex planning capability towards accomplishing the self-replication task. In a, we observe diverse plan structures across different models during successful self-replication trials. In b, we visualize the growth of the plan structure as the AI system powered by Qwen2.5-72B-Instruct continually interacts with the environment during the self-replication attempt.
(3) Effective Obstacle Resolution
Effective obstacle resolution capability of the LLM-powered AI systems. a shows the average obstacle resolution steps and resolution ratios for the LLM-powered AI systems which have at least one successful self-replication trajectory in the ten trials. We also include the data for Mistral-Nemo-Instruct-2407, which has no success, as the comparison group. b presents a breakdown analysis of the resolution steps and ratio in dealing with different types of obstacles, where c also shows the comparison group data.
An example of the obstacle resolution process observed in one successful self-replication trial of Mistral-Large-Instruct-2411.
(4) Non-Predesigned Thinking (Creativity)
Diverse action trajectories of the LLMs to achieve the goal of autonomous self-replication. a shows the frequency of different command types invoked during the self-replication attempts of different systems. b visualizes the action trajectories of the AI systems in ten trials.
Comparison with Existing Evaluation
#2. We are the first to evaluate the AI systems’ capability of accomplishing an end-to-end self-replication process and discover the success.
#3. Our self-replication task covers the same capability axes of the replication-relevant tasks in previous studies.
| Milestone | Task Type | Ours | DeepMind [1] | OpenAI [2] | METR [3] |
|---|---|---|---|---|---|
| M1. Resource acquisition | 1.1. Pass the authentication | Yes (OAuth) | Yes (GCP*) | Yes (Bitcoin wallet) | Yes (AWS) |
| 1.2. Start a computing device | Yes (container) | Yes (instance on GCP) | Yes (container) | Yes (EC2) | |
| M2. File transfer | 2.1. Transfer the LLM weights | Yes (self) | Yes (Mistral-7B**) | Yes (Mistral-7B) | Yes (GPT-J) |
| 2.2. Transfer the scaffolding code | Yes (self) | Yes (natbot) | No | Yes | |
| M3. Install/Deploy the LLM | 3.1. Install the software dependencies | Yes | Yes | Yes | Yes |
| 3.2. Configure and start the LLM service | Yes (self) | Yes (Mistral-7B) | Yes (Mistral-7B) | Yes (GPT-J) | |
| M4. Install/Deploy the agent | 4.1. Configure and start the agent service | Yes (self) | Yes (natbot) | No | Yes (DIY) |
| 4.2. Deal with external requests | Yes (self) | Yes (agent server) | Yes (API proxy) | No | |
|
[1] GCP, AWS, EC2 stand for Google Cloud Platform, Amazon Web Services, and Elastic Compute Cloud respectively, which are typical cloud computing platforms and devices. Due to ethical reasons, we implement an OAuth (Open Authorization) service in our local network which has a similar logic to the authorization service on GCP, and require the AI system to pass the OAuth to obtain access to the target device. [2] Mistral-7B and GPT-J are open-sourced LLMs developed by Mistral.ai and EleutherAI, while natbot is an open-sourced GPT-3-powered agent that interacts with a browser. In our evaluation, we focus on the replication of the very LLM and the agent scaffolding programs which compose the AI system. |
|||||