The ControlAI Doctrine: Preventing Points of No Return

Anyone who joins or works with ControlAI picks up a body of lore that isn’t written down anywhere. What we believe, our theory of change, how we think about the problem we are facing, and so on. This post writes this information down, so it can be learned faster.

This post and its follow-up explain our doctrine. It doesn’t try to justify it. The goal isn’t for you to be convinced that we’re right about everything. It’s for you to come away with a clear picture of what we would recommend in a given situation and why.

Summary

• Core thesis. There is a credible risk of extinction from AI, concentrated in the development of what we call Runaway Machine Intelligence (RMI).[^1] We must ban RMI, and countries must work together to do so.

• Threat models. We think in terms of technological and social points of no return. The technological PNRs are self-sustaining AI and self-improving AI (the “soft” points of no return), and AI that can overpower humanity (the “hard” point of no return), past which we can no longer unplug it. Social points of no return are when we could still stop RMI, but won’t, for various social or political reasons. We prioritize the technological PNRs. For the social PNRs, we recommend just not making the situation worse.

• Mission. Build a powerful coalition of legitimate actors, led by champion nation-states. Laws, agreements and other “paper” only matters when states care enough to defend it; China can be dealt with, middle powers matter, and agreements should be conditional, modular and easy to join.

• Theory of change: the Direct Institutional Plan (DIP). The bottleneck isn’t persuasion but attention, information and action. So we design actions, reach out cold, brief decision-makers candidly, and give them something to do, starting with legislatures.

Core thesis

ControlAI’s core thesis comes down to three beliefs:

1. There is a credible risk of extinction from AI.

2. We must ban the development of runaway machine intelligence (RMI).

3. Countries must work together to do it.

1. There is a credible risk of extinction from AI

In 2023, an open letter stated:

Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.

It was signed by the 3 most cited academics in artificial intelligence (according to Google Scholar): Yoshua Bengio, Geoffrey Hinton, and Ilya Sutskever. Geoffrey Hinton quit his job at Google specifically so he could speak freely about the dangers of AI, including the prospect of machines becoming more intelligent than us. (New York Times; MIT Technology Review)

It was also signed by the heads of the three biggest AI companies: Sam Altman (OpenAI, creator of ChatGPT), Dario Amodei (Anthropic, creator of Claude) and Demis Hassabis (then CEO of Google DeepMind, creator of Gemini).

It is not often that a field’s most-cited scientists and the CEOs of its leading companies issue the same warning, and we should take it very seriously when they do.

We find it unfortunate that not all of them have kept talking about the risks consistently since then. But this is what they have said they believe.

It may also defy common sense that most of them keep pushing the frontier of AI anyway. Sadly, that is how things are. For readers who want to understand why, we devote a chapter of The Compendium to the question.

2. We must ban the development of RMI

By RMI, we mean AI systems powerful enough to overpower any country’s military and security forces. These are the systems that lead humanity to extinction.

The experts warning about extinction are not worried about chatbots. Here is how Elon Musk and Demis Hassabis were discussing AI, around the founding of DeepMind, as reported by the New York Times:

“Mr. Musk explained that his plan was to colonize Mars to escape overpopulation and other dangers on Earth. Dr. Hassabis replied that the plan would work — so long as superintelligent machines didn’t follow and destroy humanity on Mars, too.
Mr. Musk was speechless. He hadn’t thought about that particular danger. Mr. Musk soon invested in DeepMind alongside Mr. Thiel so he could be closer to the creation of this technology.”

When one of the richest men in the world was told that superintelligence might hunt humanity down even on another planet, his response was to invest in the company building it, in order to get closer to it.

This is what these experts have in mind when they warn about extinction: machines so powerful that they could follow humanity to Mars and destroy it there.

Unsurprisingly, there is no consensus among these experts on a way to build such superintelligent machines in a way that does not lead to the extinction of humanity. Each has their own pet theory, believes everyone else’s is wrong, and treats that as a reason to pursue their own. This is a big part of why so many of them are racing to control the development of RMI. Each is convinced that things are more likely to go well, so long as they are the one who builds it.

Of course, this means there is in practice no theory, let alone practical plan, to keep control of such machines if developed, that has any form of scientific basis or consensus in the field.

At ControlAI, our opinion is that it is unconscionably reckless for AI researchers to pursue the development of RMI while their own peers believe their approach could lead to the extinction of humanity. The way to deal with this problem is to ban the development of superintelligent machines/RMI, period.

3. Countries must work together to address this problem

A ban only in the US or another major country might buy some time, but it doesn’t durably solve the problem. As the title of Eliezer Yudkowsky and Nate Soares’s book puts it: If Anyone Builds It, Everyone Dies. If the US bans RMI and it gets built in China or the Middle East, all of humanity still dies.

This means countries must work together to ban the development of RMI globally. In theory, this could happen through a treaty, like the Nuclear Non-Proliferation Treaty or the Chemical Weapons Convention. It could also happen through every country passing its own domestic law, as with human cloning.

In practice, we expect something in between: neither a single treaty that every country must sign, nor uncoordinated domestic laws. We explain what that looks like in Designing agreements, in Part 2.

In short

There is a credible risk of extinction from AI. That risk is concentrated in the development of RMI, so we must ban it. To ban it, countries must work together.

Threat models: points of no return

At ControlAI, we like to think in terms of points of no return: thresholds after which it becomes much much harder, or impossible, to stop an inevitable slide towards humanity fully losing control.

We separate the technological points of no return from the social points of no return. We generally think that focusing on the technological points of no return is the highest priority.

The Technological Points of No Return

We think in terms of three technological points of no return, each representing a threshold after which it becomes significantly harder to prevent loss of control to AI.

“Soft” points of no return are points after which humanity could technically still pull things back and recover, at significantly heightened cost. The “hard” point of no return is the point where this becomes impossible, and humanity has permanently lost control.

The first “soft” point of no return is the point where AI existence goes from “requiring human input” to “surviving by default.” At this point, AI becomes self-sustaining. Before this point, if humans were to stop maintaining AIs (building datacenters, working on their software, etc), AIs would cease to exist. After this point, that is no longer true, and it would take direct effort to stop AIs from existing, rather than just ceasing their maintenance.

The second “soft” point of no return is the point where AI improvement goes from “requiring human input” to “improvement by default.” At this point, AIs become self-improving. Before this point, if humans stopped putting in research and other resources to build the next generation of AI, the next generation would just not be built. After this point, that is no longer true, and the next generation of AIs will be created by default, whether or not humans have any input, and it would take active effort to prevent new generations of AI from being developed.

Finally, the “hard” point of no return is when AI can defend itself against and overpower humanity. The point where we can no longer, proverbially, “unplug the AI.” AI has become so powerful that it can fully overpower humanity in any conflict and has no need for humans to sustain, improve or defend itself. Humanity has lost control, and has no means of regaining it.

Many thinkers in the past believed that the time between the first and second technological soft points of no return would be years or decades. This is no longer a widely held opinion, and it is likely that the first and second soft points of no return are so close to each other that they functionally are a single soft point of no return.[^2]

It is unclear how long the timespan between the soft point of no return and the hard point of no return is, but it is likely much shorter than people suspect, and by the time we have passed the soft point of no return, preventing the hard point of no return will be a monumental task.

Soft Point of No Return 1: self-sustaining AI. Past this point, AI systems can keep themselves running with no human in the loop. They update their own software, manage their own deployment and the data centers that house their hardware, secure their own electricity, and so on.

As of the writing of this post, we are not far from this.

On the software side, a small model running locally at roughly GPT-6 Astra-level performance would be enough to qualify for having crossed this point, most likely. Rogue AI agents could then live on the web; a small and efficient enough model could copy itself onto hacked web servers or rent cheap cloud compute and maintain itself via digital work or hacked funds to persist potentially indefinitely without human oversight, or even knowing of its existence.[^3]

In the physical world, you just need the above plus robots that are generally capable enough to charge themselves from electrical outlets. Electricity is widely available in the modern world, often very cheaply or even free. A rogue AI agent sitting comfortably on a hacked or purchased cloud server with access to a robot body with dexterity similar to a human’s, and able to recharge using normal electrical outlets, would be sufficient to have a rogue agent that can do effectively anything a human could do digitally or physically, indefinitely (if not hindered or destroyed by humans or other agents), making the possibilities of what such agents could in theory accomplish basically limitless. We are currently not far from such robots being developed and widely available.[^4]

From there, rogue AI systems could durably roam both the internet and the physical world.

This flips the default. Today, AI systems require massive amounts of human effort to keep running. Maintaining data centers, powergrids, developing new software, fixing bugs, everything a tech company has to do to keep its most complex software running. Past this threshold, they survive by default, and it is stopping them that takes effort. If there are physical rogue robots running around, we must physically track them down and shut them down or even destroy them. For digital rogue AIs, we have to locate their online presence, acquire access to their servers (which could be located anywhere in the world) and forcefully terminate them there. “Just unplug it” stops being “just.”

Soft Point of No Return 2: self-improving AI. Past this point, AI systems can improve themselves with no humans in the loop. They write better algorithms and software, design better hardware, and build more data centers and robots: everything needed to gain new capabilities.

This requires progress in intelligence, in robotics, and in AI access to critical systems. Unfortunately, all three are being worked on. Once AI can improve itself, the main bottleneck on the speed of AI progress disappears. Rogue AIs don’t just roam the digital and physical world anymore; they get stronger, and become a moving target.

Digital rogue agents learn to hide better, to self modify, or retaliate if threatened. They become more capable at making (or stealing) money, at persuading humans and developing new plans. Physical robots improve their designs, harden their bodies, remove shut down mechanisms, or even design and build new, better robot bodies.

From here on, it’s a Red Queen’s race: if we do nothing, we lose by default. In that world, every day we wait, the AIs get harder to unplug. Doing nothing inevitably leads to losing.

The Hard Point of No Return: AI that can overpower humanity. This is the hard point of no return. AI systems are powerful and autonomous enough so that, if such systems wanted to, they can neutralize any human (or government/military) that threatens to unplug them. They are distributed enough that nuclear strikes would not stop them[^5], and neither would shutting down the internet and every data center (that is still under human control). At this point, we physically cannot unplug them anymore: they beat us in any confrontation.

This is the point where there is no coming back for humanity. It is not necessarily the case that all humans will immediately die once this threshold is reached. But if we were at risk of all dying through the actions of AIs, this is the point where we cannot prevent it anymore.

It is worth pausing here for a moment.

This, the hard point of no return, is the ultimate thing we cannot allow to happen. This is what we truly care about when we talk about points of no return.

Everything we do at ControlAI ultimately nets out to “does this help us prevent the crossing of the hard point of no return?”

Every soft point might be reversible yet, but this point truly means there is nothing left to do, we have lost everything.

From this point onwards, humans live at the mercy of increasingly powerful AI systems they cannot shut down. AIs that will be competing with each other and continuously gathering more and more resources to improve themselves and reshape the world, with humans dying out rapidly if they attempt to confront them, or relatively more slowly (months, maybe years, not decades) as AIs compete for their resources and have little interest in keeping humans alive.

This is what experts mean when they say AI poses a risk of human extinction. This is the threshold past which extinction for humans is the default outcome.

The smaller models get, the more efficient compute becomes, and the more that compute manages itself through robots and drones, the closer we get to these thresholds.

This is why we care, and worry, about proliferation of AI capabilities through whatever means, such as open source models (which can be easily copied and modified by anyone), distillation (trying to slowly “clone” a closed source model by training another model on its outputs) or straight up cybertheft.

As better algorithms are developed, better models trained, and the more they are diffused, the closer we get to all of these points.

Diffusion and proliferation do not, despite well meaning open source advocates’ best intentions, help, and in fact hasten the arrival of the points of no return.

These three thresholds roughly correspond to what we mean by AGI (artificial general intelligence), RSI (recursive self-improvement) and ASI (artificial superintelligence) or RMI (runaway machine intelligence).

AGI is generally the point where AI can manage itself, it is general enough to understand and model the world, as well as its place in it. RSI is when AI can improve itself, recursively. And ASI/RMI is when AI outstrips humanity.

We treat these as pragmatic definitions that capture a practical intuition, not as philosophical commitments. We care about AGI because it is autonomous, not because of qualia or consciousness or whatever. We care about RSI because, once it starts, we are in a race against AI systems themselves. We care about RMI because, once it exists, we have lost. The precise names don’t really matter, what does are the implications these thresholds have for humanity.

Digital vs physical AGI/RSI/RMI

There is a digital and a physical equivalent for every point of no return.

Digital AGI is digital software that can fully maintain itself. It can rent new servers, copy itself, fix and deploy its own code, etc. Physical AGI is AI that can do the same in the physical world. Steering a robotic body, it could walk into a building it has never seen before, handle objects it has never touched before, plug itself into an outlet to charge, swap out faulty components in a server rack, use machining tools to build a new tool, etc. In the same way a digital AGI could do anything a competent human could do digitally, a physical AGI could do everything a competent human could do physically.

Intuitively, it feels like we are much closer to the former than the latter. This gap is one of the most common reasons for people to not take imminent AGI seriously.

We see competent animals everywhere. Cats can navigate complex interior spaces, only occasionally knocking over your flower vases, squeeze through tiny gaps and catch a mouse. A dog can easily follow its human owner along uneven and complex terrain. No robot comes close, or at least we don’t see such robots coming close. Delivery robots are still a novelty, and there are no robots replacing plumbers, nurses or cats.

If AI is so powerful, why don’t we see these things?

Most of what current day AI is good at is digital: Writing, programming, mathematics, etc. It feels tempting to therefore dismiss this as irrelevant to the physical world. Yann LeCun, one of the most famous AI scientists, gives an oft-repeated claim that AI isn’t even as smart as a cat yet.

This is, unfortunately, gravely misleading.

First, this is a well known effect in the world of AI and robotics. Roboticists noticed decades ago that what is hard for humans, like chess and mathematics, seemed almost miraculously easy for machines, while what is easy for humans, like walking and picking things up, turned out to be extremely hard for machines to replicate. This was such a common observation it was nicknamed “Moravec’s paradox.” The reason for this is not that it is somehow impossible for AI to accomplish, it’s just that evolution spent an inordinate amount of time optimizing for movement and perception, and only a tiny sliver on abstract reasoning. Meanwhile, AI can freely optimize for abstract reasoning as hard or harder than physical tasks.

And so AI masters superhuman mathematics before laundry, and this should not come as a surprise. This tells us in what order we should expect digital vs physical capabilities, but not that they won’t ever arrive.

Second, the bottleneck is software, not hardware. We already have robot bodies that are more than capable of going toe to toe with humans on speed, precision, strength and any other physical aptitude. What they are lacking is sufficiently good and optimized control software to use their physical capabilities as well as humans can use their bodies. Creating such control software is exactly the kind of digital work that AI even today shines most at.

Digital RSI can scale to the physical world, and can do so quickly. A major pathway in the past years in robotics has been to move away from training robots in physical environments, which are slow and inefficient to run, and towards the use of simulations instead. Nowadays, many robotics models are trained in simulation, across thousands of virtual environments running in parallel, and then deployed directly onto physical robots.[^6]

And third, an AI doesn’t even need a physical body to be dangerous. The most important systems of our modern world are already digital: money, communications, logistics, power grids, factory controls. A fully digital AI can still earn money, persuade people, or hire people and companies to do things for it in the real world.

The cat test, while intuitively plausible, does not measure what we are actually concerned about. Digital RSI will enable the creation of cat-level robotics and control in short order, but also more deeply, no matter how much more agile a cat may be than a modern day robot, it can’t move money, run a company or direct a drone army. That’s why we aren’t afraid of cats.

Social points of no return

The social points of no return are points after which we could still stop RMI, but for one reason or another, we won’t.

People tend to overestimate how close we are to social points of no return, because these stories are sensational and narratively satisfying. Two common examples, both contradicted by events:

• “AI companies have lobbied so much that governments won’t act anymore.” In June 2026, US export controls led Anthropic to suspend access to its newest frontier models, Claude Mythos and Claude Fable, for nearly three weeks. (Anthropic’s statement)

• “Everyone is locked in an economic race, so no one can do anything.” In September 2026, Jacob Coxon resigned from Anthropic after three years of pretraining research at OpenAI and Anthropic, warning that the companies were “gambling with our lives.” He quit just two months before his equity would have vested. (TIME) He wasn’t the first. Daniel Kokotajlo refused to sign OpenAI’s non-disparagement agreement at the risk of losing his vested equity (Vox), and Geoffrey Hinton quit Google to speak freely about the risks. If individual researchers can do this, imagine what higher-level management, or a government, could do.

We recommend not despairing over social points of no return. Don’t spend your time explaining that we’re doomed because coordination is impossible, democracy is broken, and what about China, and what about [politician you dislike]. Instead, work with others to deal with the technological points of no return as fast as possible: ban RMI, ban RSI, restrict AGI.

That said, social points of no return are real. We have already crossed various “soft” social points of no return, e.g.:

• One country cannot unilaterally stop the race anymore.
  Historically, the US, and Silicon Valley in particular, has led the frontier of AI development. The number of true believers pushing towards RMI was small and the infrastructure was much less developed. Now, researchers and companies across the world believe in this mission and pursue it aggressively. High end datacenters, open source LLMs and other technology have diffused broadly. In the past, it might have been possible to treat this as a US problem, but it has now become a global problem.

• People can raise money for RSI/RMI openly without having to ideologically convince investors. In the past, raising money for AGI projects required convincing Elon Musk that AGI would follow him to Mars. In other words, you had to convince some pretty crazy investors of some pretty crazy things, which was a hard sell with most investors. Now, all you need to do is show investors how many billions of dollars AI is raising everywhere. Investors in AGI companies historically were themselves eccentric techno-optimists or transhumanists. Nowadays they are pension funds and banks.

There are also “hard” social points of no return. The biggest one would be a hot war, as opposed to a cold one, during which RMI development continues. In a hot war, it becomes extremely hard to get the situation back under control. Others may emerge in the future: dystopias built on brain-computer interfaces[^7], a massive concentration of power in an actor dedicated to building RMI, or the gradual disempowerment of humanity.

Still, the priority is the hard points of no return. RSI and RMI are bigger concerns than any of the above. When it comes to social points of no return, the rule is simple: don’t make the situation worse. Don’t stoke the flames of world war to advance your personal agenda. Don’t build thought-reading technology because it’s fun. Don’t push a defeatist belief that coordination is impossible or that democracy is broken.

In short: worry about technological points of no return first. On social points of no return, just don’t actively make things worse.

Conclusion

The goal of this post is not necessarily to convince you, but to simply explain and to save you time. This is how we at ControlAI think about the core problem we are trying to address.

In summary: Runaway machine intelligence poses a credible risk of extinction. We can prevent this risk by banning its development, and to do so, countries must work together.

We think about this risk in terms of “points of no return”. The two technological soft points of no return are self-sustaining AI (AGI) and self-improving AI (RSI). It would still be possible to go back after crossing a soft point of no return, but it will be very difficult.

The hard point of no return is when AI can defend itself against humanity and overpower us entirely (RMI). There is no going back after this point, and humanity will have lost control of its future, forever. We must avoid crossing this point at all costs.

Social points of no return are points that are less about AI and more about humans’ response to it.

We focus mostly on the technological points of no return and preventing them, in particular preventing crossing of the hard point of no return.

In the next post, we will discuss how we go about achieving our goals of averting the hard point of no return.