ControlAI's Policy Proposals
AI is well known as the technology behind chatbots like ChatGPT, but what the top AI companies are explicitly aiming for is something else entirely: superintelligent AI. That means AI that can fully replace and outmatch humans at any task, including in domains like hacking, social engineering, and military operations.
AI engineers are unable to understand and reliably control even today’s AI systems, as demonstrated by recent incidents in which rogue AI systems fully autonomously escaped their developers’ containment and hacked into real organizations. And this is the least capable AI systems will ever be: this is why leading experts have warned that superintelligent AI poses a risk of human extinction. The reason is that no company, no government, no individual knows how to keep such a system under human control.
AI is the crucial technology of the 21st century, vital to a competitive military and promising vast benefits in science. Superintelligent AI is different. It would not be a tool humans can leverage, but rather an adversary able to pursue its own objectives with no human control over its actions. Regardless of who develops it, superintelligence would pose a national and global security risk.
Developing superintelligent AI at home offers a country no protection from this threat. Superintelligence would amount to a rogue army stationed inside the country’s own datacenters, answering to no one, capable of seizing control of critical infrastructure and attacking human organizations with superhuman sophistication.
ControlAI’s Core Policy Principle: Ban Superintelligent AI, Monitor and Restrict Precursors
AI that outmatches humanity in every domain, would, by default, permanently disempower us and would likely cause our extinction. This risk can be averted by ensuring that no one, anywhere, develops superintelligent AI. To guarantee this, development of superintelligent AI should be banned worldwide.
Every country that understands the universal threat posed by superintelligence has a direct stake in joining the collective effort to prevent its development.
Any attempt at superintelligence development must be detected and stopped well before it comes to fruition. To avoid dangerous last-resort actions that countries may consider to dismantle a nearly complete superintelligence project, countries must monitor and restrict precursors to superintelligent AI across the world. These precursors include the powerful chips that enable AI training, and AI capabilities such as hacking.
A blanket ban on precursors would undermine a vast range of AI applications unrelated to superintelligence, including critical military applications. Instead, within these restrictions, AI development and deployment can continue: even without superintelligence development, countries can reap the benefits of advanced AI in science, medicine, and defense.
Superintelligence development is a large-scale industrial process, akin to uranium enrichment. It requires massive datacenters, visible from orbit and detectable through thermal imaging satellites, filled with state-of-the-art AI chips whose supply chain runs through a handful of lithography machines. This means that it is feasible to monitor current avenues for superintelligence development.
Establish an International “Trust, But Verify” Regime to Prohibit Superintelligence Development Worldwide
Countries should negotiate to establish an international “trust, but verify” regime to prohibit superintelligence development worldwide. Participants in this regime would commit to not developing ASI, and guarantee that no one develops it within their jurisdictions.
Participants would also cooperate to deter superintelligence development by any actor outside of the regime, via diplomatic and economic measures, and as a last resort military action. As part of this effort, participants would monitor and restrict precursors to superintelligence, in order to detect and prevent any superintelligence development projects.
Not every single country needs to prohibit superintelligence development for the ban to be effective worldwide. If a sufficiently powerful coalition of countries joins the regime, superintelligence development will become very costly to carry out anywhere in the world. Every country has a stake in ensuring superintelligence is not built, and coordinated deterrence is the most effective way to do so.
Participants should monitor each other for signs of superintelligence development, to build trust that no participant in the regime is pursuing superintelligence in secret. Each participant would share sufficient information about its AI hardware, and how that hardware is used, to demonstrate that it is not developing superintelligence.
Pass Domestic Legislation Banning Superintelligence Development
Individual countries should pass domestic legislation banning the development of superintelligent AI. It is illegal to attempt to build a nuclear bomb, even if the attempt fails; the same standard should apply to superintelligence.
It is in no country’s interest to develop superintelligence, because it would threaten the country’s leadership and its people. A domestic ban would make it clear that superintelligence development is an unacceptable threat to national security, and give the government tools to ensure no private actor is endangering the country by pursuing it.
For the purpose of this ban, we recommend defining superintelligence by outcome rather than by technical details: superintelligence is an AI system capable of overpowering or undermining the country’s national security establishment. This anchors the definition to the actual threat, and keeps the law relevant no matter how the technology used to build AI changes.
There are capabilities that act as points of no return: once they are developed, preventing superintelligence becomes drastically harder. A ban on superintelligence is best enforced by drawing the line before these capabilities, treating their development as a violation.
The clearest example is Recursive Self-Improvement (RSI): AI systems that can autonomously develop the next, even more powerful generation of AIs without human involvement. Many experts believe that, if this is achieved, it will kickstart a runaway process that increases AI capabilities far too fast for humans to intervene, quickly producing superintelligence.
Finally, this legislation should establish government powers to monitor and restrict precursors to superintelligent AI domestically, from advanced AI chips to AI capabilities such as hacking. This enables the government to detect and shut down superintelligence development in its jurisdiction long before any project comes close to producing superintelligence, while still allowing the government to make use of these precursors for legitimate purposes.
Who We Are
ControlAI is a non-profit that informs policymakers and the public about the threat of superintelligence and helps them act to address it. We advise lawmakers in multiple countries and are building an international coalition to prohibit superintelligence development.
In less than two years, we have briefed more than 20 U.S. Members of Congress and over 150 federal lawmaker offices, and over 350 lawmakers in the UK, Canada, and Germany. Over 150 lawmakers in the UK and Canada, across parties, have supported our campaigns recognizing the national and global security threat posed by superintelligence.
If you are a lawmaker or staffer, contact us for questions, briefings, and draft legislation at advocacy@controlai.org. For media inquiries: media@controlai.org.
If you are a citizen, you can help establish these policies by contacting your lawmakers. We’ve made it take less than a minute: controlai.org/take-action.
Get Updates
Sign up to our newsletter if you'd like to stay updated on our work,
how you can get involved, and to receive a weekly roundup of the latest AI news.
