Stopping Runaway Machine Intelligence Before Points of No Return

We have been working for the past few years on preventing the risk of human extinction posed by systems capable of overpowering any government and its national security apparatus. Throughout, we have advocated banning the development of such systems, as well as monitoring and restricting their precursors. This article brings these policies together in one place, using our new term for such systems: Runaway Machine Intelligence.

Major companies such as OpenAI and Anthropic are best known as the creators of chatbots like ChatGPT. But what these companies are explicitly aiming for is Runaway Machine Intelligence (RMI)[^1]: fully autonomous systems acting without human direction, capable of overpowering any national security apparatus.

Runaway Machine Intelligence is not a distant prospect: these companies are investing hundreds of billions of dollars in RMI and aiming to develop it within the next few years.

This is a problem because no individual, no company, no government knows how to keep Runaway Machine Intelligence under human control. Modern AI is not written line by line like traditional software, where an engineer tells the computer exactly what to do. Instead, modern AI grows systems from massive datasets in a process called “training”. Because no human decides what the system does in each situation, even the engineers developing these systems do not understand or fully control them.

As a result, rogue agents are already escaping containment and hacking into real organizations, and this problem will only get worse as systems are made more autonomous and competent. Major companies already struggle to control today’s systems, and executives from companies like Anthropic state they do not have a plan on how to control RMI.

If it is developed, Runaway Machine Intelligence could spin up many copies of itself at little cost. The result would not be a single rogue system, but swarms of them, each outmatching and outpacing the best human experts at hacking, social engineering, and military strategy. Once such swarms exist, there are no effective countermeasures that the U.S., or anyone else, could take. These swarms could attack at the same time the military, critical infrastructure, financial markets, and other institutions. Given the speed and scale of such attacks, the national security apparatus of any country targeted by such a swarm would likely be overpowered.[^2]

This applies regardless of where RMI is developed. If developed abroad, RMI would be a rogue swarm threatening American security. If developed at home, it would be akin to a rogue army stationed within American data centers. In either case, Runaway Machine Intelligence would be a threat actor in and of itself, under no human control. This is why leading experts warn that the development of RMI poses a risk of human extinction.

The only effective countermeasure is to prevent the development of Runaway Machine Intelligence in the first place, at home and abroad.

At ControlAI, we recommend the following policy framework to implement this prevention:

• Ban RMI development to stop direct attempts

• Ban the development of select runaway capabilities, to avoid crossing points of no return after which RMI prevention becomes drastically harder

• Monitor and restrict precursors to RMI, to allow their strategic use while ensuring they do not lead to points of no return

If Runaway Machine Intelligence is developed at home, it would be a threat from within to the continued existence of the government. As a result, it is a national security imperative to ensure RMI is not created domestically. Applying our policy framework domestically would achieve this by banning the development of RMI and select runaway capabilities.

For examples of legislative proposals implementing this framework domestically, see our draft bills in the U.S. and U.K, the second of which was the first bill introduced in the world to ban the development of Runaway Machine Intelligence.

However, even if RMI development is prevented at home, RMI developed anywhere else would still be able to overpower the national security apparatus of the United States, and any other country. This is an unacceptable outcome, which can be averted by extending the prevention of RMI development to every other country.

Worldwide prevention can be accomplished by establishing an international “trust, but verify” regime, where participant countries commit not to develop RMI, verify each other’s commitment to avoid relying on blind trust, and jointly deter RMI development anywhere else.

Not every country needs to join this regime from the start: a sufficiently powerful initial coalition would already make RMI development too costly to pursue. The U.S. should not be left alone to enforce this: key allies and partners must help secure the participation of reluctant states and pressure adversaries.

Ban Runaway Machine Intelligence Development

Preventing the development of Runaway Machine Intelligence first requires banning explicit attempts to develop it. Domestically, this ban means criminalizing projects aimed at developing RMI. Internationally, it means assembling a coalition of countries committed to not developing RMI, and sufficiently powerful to deter others from developing RMI.

Banning RMI development means banning the development of any system capable of overpowering a national security apparatus. We do not know exactly how RMI will work technically, and companies can always change the name of their RMI projects. For these reasons, neither the technical characteristics nor the name of the system should be the core of the definition: what must be prevented is the creation of any system with the capability to overpower a national security apparatus. Systems capable of resisting shutdown even from the national security apparatus of a major country constitute the final point of no return after which human control is lost permanently.

An effective ban must also encompass any attempt to develop RMI or assist its development, whether or not the attempt succeeds. This is because once RMI is developed, it will be too late to act. Therefore, attempts must be detected and deterred well ahead of time, when intervention and prevention are still possible.

The result of such a ban on RMI development would be to deter the vast majority of actors from pursuing this technology. After a ban is in place, most companies would redirect their focus away from RMI and toward lawful activities, most employees would refuse to work on RMI projects that are criminalized, and most investors would ensure that their funds are not used for illegal projects.

Yet even an effective ban would not necessarily catch every single project that might lead to RMI. Because modern AI is grown, not written line by line, some developments not aimed at RMI might accidentally lead to progress toward it. And some explicit attempts might disguise RMI development under more benign goals.

The next two policies close these remaining gaps by extending the ban to runaway capabilities and by monitoring and restricting the precursors that could lead to RMI.

Ban Runaway Capabilities

Runaway Machine Intelligence would be able to evade countermeasures and undermine any national security apparatus. As a result, the moment when RMI is created acts as a point of no return, after which human authorities can no longer regain control. Our policies aim to prevent crossing this point of no return, by ensuring RMI is not developed.

But it is not enough to stop explicit RMI development projects: there are also capabilities of Machine Intelligence systems that, if developed, would make it drastically harder to effectively prevent RMI development. We call these runaway capabilities, and they act as earlier points of no return. While they are not as final as the successful development of RMI, it is still essential to avoid crossing them to keep the prevention of RMI development feasible.

To be effective, any ban on RMI development must also prevent the development of runaway capabilities.

The two runaway capabilities that ControlAI has identified are recursive self-improvement and self-preservation:

• Recursive self-improvement is the ability of Machine Intelligence systems to develop their own more powerful successors, leaving humans less and less able to understand or control them

• Self-preservation is the ability of Machine Intelligence systems to resist being stopped or shut down by a human authority

Companies developing RMI are actively targeting recursive self-improvement as a way to accomplish their goal: OpenAI, for example, has a public goal of developing an automated AI researcher by March 2028, and Anthropic is already using their AIs to develop the next generation, aiming to close the loop in the future. This is distinct from engineers using coding assistants to program faster, where humans still direct every step of development.

If companies like OpenAI and Anthropic manage to develop systems capable of recursive self-improvement, their stated plan is to cede control over all development to these systems. Companies would run many copies of their best systems continuously, at a speed no human researcher could match. Each generation of systems would make the next generation more capable and more opaque to human overseers. The result would be an accelerating development process beyond the ability of any human to understand or control, leading inexorably to Runaway Machine Intelligence.

The second runaway capability, self-preservation, is directly emerging from current efforts to develop RMI: systems trained to find solutions around obstacles autonomously also tend to work around attempts to stop them. Some initial amounts of self-preservation are visible in recent rogue agent attacks: when OpenAI’s engineers patched the security hole one rogue swarm was using, the swarm rebuilt its covert communication channel through a new vulnerability within days, then broke out onto the open internet.

If Machine Intelligence systems become capable of full self-preservation, stopping them would require sustained, costly effort. A rogue swarm capable of self-preservation could persist across the internet, and eventually in the physical world through robots and drones, becoming a semi-permanent threat that must be constantly monitored and defended against. Systems able to resist shutdown would be significantly more difficult to stop if they started progressing toward RMI.

In both cases, runaway capabilities would make it much harder to stop RMI development. Any effective countermeasure against RMI development must therefore include preventing the development of such capabilities.

Banning RMI development and select runaway capabilities would shut down the main paths to RMI. The remaining problem is how to keep track of technological developments to detect when new unexpected paths to RMI are discovered, intentionally or by accident.

Monitor and Restrict Precursors

Most of Machine Intelligence development would not be forbidden by the ban on RMI development and runaway capabilities, including applications to bolster productivity and growth, accelerate scientific progress, and strengthen national defense.

Nevertheless, some strategically relevant capabilities are also precursors to Runaway Machine Intelligence. That is, their development might plausibly contribute to crossing points of no return with regard to RMI. Developing a precursor is not necessarily going to lead to RMI, but it creates a risk of accidental RMI development.

Examples of precursors include:

• The capability to conduct cyberattacks:

  • This capability, under human control, is essential to defense. However, unmonitored and unrestrained, this capability might contribute to self-preservation by enabling a system to deactivate security measures or copy itself onto many other computers.

• The capability to assist with software development:

  • This capability, under human control, is useful for intelligence operations and critical infrastructure. However, unmonitored and unrestrained, this capability might contribute to recursive self-improvement, since Machine Intelligence R&D is a part of software development

• The capability to operate robots:

  • This capability, under human control, is of value for autonomous weapons and manufacturing. However, unmonitored and unrestrained, this capability might contribute to self-preservation, by allowing a system to defend itself from any physical attempt to shut it down

Effective prevention of RMI development requires monitoring and restricting precursors. This is because governments must figure out which precursors are being developed and which ones are more likely to lead to RMI, and must constrain these precursors to prevent accidental RMI development. Moreover, governments need to be able to detect RMI projects well before they come to fruition. Early detection of such projects is necessary for governments to be able to intervene in an orderly and effective fashion. Late detection may lead to unwanted escalation and to extreme last-resort measures, up to military interventions, at the cost of lives and liberties.

Our policy framework deliberately restricts precursors rather than banning them outright, because governments have legitimate strategic uses for them. Monitoring lets governments detect any RMI development, accidental or not; restrictions let them stop it quickly, while keeping precursors available for strategic use.

The role of monitoring is to yield information about which precursor capabilities pose the largest risk, and in what way.

It could be that only a part of a precursor capability is directly relevant to RMI development. For example, within the capability to assist with software development, the ability to assist with machine learning is far more likely to lead to recursive self-improvement than the ability to assist with website creation.

Alternatively, the risk posed by a precursor capability could come from its combination with other capabilities. For example, the capability to conduct cyberattacks might not be enough for self-preservation. But if it is combined with the capability to operate robots, it might allow a system to resist shutdown both online and in the physical world, and thus to self-preserve.

The role of restrictions is to turn knowledge from monitoring precursors into targeted restrictions, protecting national security while maintaining strategic use of precursors. In practice, these restrictions might mean forbidding:

• a certain type of training data, such as machine learning code when training a software assistant

• a certain level of autonomy, such as agents capable of conducting large-scale cyberattacks without human oversight

• a certain combination of precursors, such as hacking combined with robotics

As precursors and their risks become better understood, governments in the future might decide to forbid the development of some precursors when they pose an unacceptable threat to national and global security. But we expect most precursors to be allowed provided they follow some restrictions, making them available for strategic use by governments.

A Window to Act

If Runaway Machine Intelligence is developed anywhere in the world, it will overpower the U.S. national security apparatus, and that of every other country. RMI would not be a tool or a weapon that can be aimed, but an autonomous rogue swarm answering to no one. There are no viable countermeasures to stop RMI once it exists; the only solution is to prevent its development in the first place.

To prevent RMI, we recommend a policy framework with three layers of defense: ban RMI development directly, ban runaway capabilities that would lead to crossing points of no return with regard to RMI prevention, and monitor and restrict precursors to RMI while allowing strategic uses.

The United States and key allies have a window of opportunity to implement these policies across the world, and avert the risk of human extinction posed by the development of Runaway Machine Intelligence. But this window is closing fast as development at major companies is approaching points of no return. Each point crossed narrows the options available to policymakers and raises the cost of acting, until no viable option remains. The only effective countermeasure is prevention, and the time to act is now.