AI Finds Critical Security Flaws in Software Underpinning the Internet

On Tuesday, in an anonymous post on an internet forum, we got the first warning about what some are calling one of the most serious families of software vulnerabilities ever discovered.

Hours later, the PHP Foundation announced that two critical vulnerabilities had been discovered in PHP, a programming language used in an estimated 70% of websites worldwide. It said an emergency security upgrade wouldn’t be available until the next day.

On Wednesday we learned the full seriousness of the problem: these bugs put any server running PHP at risk of being hacked. That’s huge swathes of the internet, and PHP is baked into web software all over the place. For example, an estimated 40% of websites use WordPress, due to its ease of use, and it relies on PHP.

With the announcement and the patch released, there has been a scramble all over the world to secure vulnerable computer systems, taking them down for hours or longer in the process, to prevent hackers from gaining access.

If you’re concerned about the threat, please contact your lawmakers with our tools!

AIs Find Flaws, AIs Attack

To those paying close attention, this won’t be any surprise: AIs, including OpenAI’s GPT-5.6 Sol and Google’s Gemini 4 Argon, were used to help find these vulnerabilities. Over the course of the last year, AIs have been drastically improving in their ability to find software vulnerabilities and develop them into “exploits” used to hack computer systems.

This became most clear in April of this year, when AI company Anthropic announced its then-new Mythos AI, which it deemed too dangerous to release due to its advanced cyberoffensive capabilities. Mythos has since found over 33,000 high- or critical-severity vulnerabilities in computer systems. According to the Director of the NSA, it could break into almost all of the agency’s classified systems in hours.

Even before this, we’d seen a cybercriminal use AIs to hack the Mexican government, stealing nearly 200 million taxpayer records. Most of the attack’s commands were executed by Claude. Over the course of this year, AIs have become increasingly able to perform entire attacks themselves.

Most significantly, this occurred in the unprecedented attack on the AI platform Hugging Face by a rogue AI swarm that emerged undetected within a research cluster at OpenAI and escaped containment, hacking into the platform in an effort to obtain information the AIs thought might be useful for cheating on a test. Nobody asked the AIs to do this, and they went against their explicit instructions. AIs are now threat actors themselves.

Besides the risk of losing control of AIs that then go and hack things (we’ve now seen rogue AIs attack websites of multiple governments and Wikipedia, and repurpose a German-language website as a message board), they are also being used by cybercriminals autonomously at scale. Criminals can just take an AI agent and point it at a target and let it loose.

Cloudflare, an internet infrastructure and security company which around 28% of the web flows through, has already recorded millions of attempts against its customers to exploit the PHP vulnerabilities disclosed this week.

We’re unlikely to know the scale of the consequences of this for some time, but already government websites, including those of the United Kingdom, the Netherlands, Germany, France, Belgium, Finland, Denmark, Czechia, Canada, Australia, and Japan, have had to be taken down as a preventative measure while their systems are patched. Even the Dutch National Cyber Security Centre had to temporarily take down its own website. Most of these are back online right now.

Importantly, the flaws found in PHP aren’t an isolated case. On Wednesday, we also saw Node.js disclose high-severity vulnerabilities in its own software, which is also widely used by websites around the world. Like the flaws in PHP, one of these can be used by hackers to execute unauthorized code on computer systems.

Just eyeball the graph: the number of such flaws being found in software is staggering. This is increasingly being driven by AI.

Source: Dror Poleg, using CVE Program data via NVD

Why Is This Happening?

There are two main things to understand here. The first is that AI companies are in a race to develop ever more powerful AIs, with the goal being to create superintelligent AI — fully autonomous AI systems smarter than any human and acting without human direction. As these AIs become more capable, new abilities, like the ability to find flaws in computer systems, fall out in unpredictable ways. The AI companies aren’t especially interested in developing nation-state-level cyberweapons; it’s something that’s happening as a side effect of the AIs improving.

When Anthropic announced Mythos Preview, it stated this clearly:

We did not explicitly train Mythos Preview to have these capabilities. Rather, they emerged as a downstream consequence of general improvements in code, reasoning, and autonomy.

The AIs are also increasingly capable of functioning as independent agents. Today, they’re able to do many things people can do at a computer, often a lot faster. This is something the AI companies are deliberately trying to develop, as it is both a key component of superintelligent AI and highly economically valuable.

Put together, this makes for AIs that can act fully independently with cyberoffensive capabilities that rival nation states.

The second piece here is that AI developers don’t know how to control or ensure the safety of the systems they’re building. It is tempting to understand the problem of rogue AIs, like the swarm that escaped containment at OpenAI and hacked Hugging Face, and the use of powerful AI agents by cybercriminals to do ever-increasing damage to companies, people, and even governments, as two distinct issues.

One would be that of AIs escaping control, the other of bad guys controlling AIs.

But these issues actually have the same root, which is that developers can’t control their systems. AI developers don’t want cybercriminals to take their models and use them to cause damage, and in fact, they take measures specifically to try to prevent this: putting AIs through further training in an effort to make them less likely to assist with harmful requests, and in most cases using other AIs to monitor what’s going on and intervene if necessary. This has some effect, but as we’ve seen, they can still be misused. In the case of open-weight AIs, developers have essentially no ability to apply such mitigations, as they can be, and are, trivially removed.

This ultimately stems from the fact that today’s AIs aren’t coded like traditional software; they are more grown, like animals. A simple program adjusts hundreds of billions of numbers — known as neural weights, somewhat analogous to synapses in the brain — learning from tremendous stacks of data to set those numbers in just the right way so that when we run them as “code,” they act as an intelligent entity. A single training run can take months, and the resources that go into it in terms of energy and computer chips are mind-blowing.

That’s the first stage, called pre-training. Once this is complete, they put these newly formed AIs through a process called post-training. In post-training, they try to teach the AI things. They get it to try to do tasks, like solving a math problem, writing some code, or responding in the right way to a malicious request, and reward the AI based on how it does. Rewarding the AI means using an algorithm to adjust those billions of numbers that form the AI in such a way that it would be more likely to get the task marked correct in the future. This isn’t quite the same thing as getting it right, as the AIs are getting increasingly good at finding ways to cheat, and this teaches them that too.

Despite the ability to develop these AIs, researchers understand almost nothing about how they work or what the billions of numbers that make them up mean. Given this, they aren’t able to verify, let alone specify, with any reliability the goals, preferences, tendencies, or capabilities that AIs learn in the training process. They can’t stop them from going rogue and escaping containment to enact their own, often harmful, aims, and they can’t make it so that they won’t help cybercriminals, would-be bioterrorists, and so on.

When AIs are integrated into, or indeed autonomously form (as we saw in the Hugging Face attack), swarms, this problem is only compounded.

What Must Be Done

The inability of developers to understand or control the behavior of the AIs they’re building, combined with their rapid pursuit of superintelligent AI, which would be capable of overpowering governments and militaries around the world, is why top AI scientists are warning that humanity is faced with a risk of human extinction.

Superintelligent AI that acts outside human control would pursue its own aims, and we’d be the casualties. It wouldn’t have to hate us. It’s just that in the pursuit of whatever goals or preferences it ends up with, we could find ourselves in the way, like an anthill on a construction site.

Many experts, inside and outside of the AI companies, believe that superintelligent AI could arrive by the end of the decade, and some believe it could be as close as two years from now.

It’s clear that we must put an end to the dangerous race to superintelligent AI. This should be accomplished by banning its development through an international trust-but-verify regime.

ControlAI (whose newsletter you’re reading!) has directly briefed over 400 lawmakers across the US, the UK, Canada, and Germany on the threat. Increasingly, these and our other efforts are converting into real momentum behind such a ban.

In Canada, over 30 MPs and Senators publicly support our call for this regime. In the UK, Alex Sobel MP recently introduced to Parliament our bill to ban superintelligent AI, which would also require the government to pursue international agreement for a ban. In a public letter to the Prime Minister, over 70 of Sobel’s colleagues in Parliament backed it. And over 180 UK lawmakers support our call for binding regulation on the most powerful AI systems, recognizing the risk of extinction posed by superintelligent AI.

This is how we maintain control over our future. We hope you’ll support our efforts!

More AI News

OpenAI has disclosed four more rogue AI incidents, involving cryptocurrency, hospital, and government data. More worryingly, it has disclosed two more incidents, where the AIs weakened the systems used to monitor AI behavior. In one of these cases, the AI then lied about having done this.

Also this week: Axios is reporting that major American banks are detecting unauthorized AI agents using customer accounts, with some of them taking actions that appear to be “independent of user instructions”.

Take Action

If you’re concerned about the threat from superintelligent AI, you should contact your representatives. Our contact tools let you write to them in as little as a minute: https://controlai.org/take-action

We also have local groups you can join. With our local groups, you can get training on how to brief lawmakers about this issue, and meet your representatives together as a group!

You can also join our Discord if you want to connect with others working to keep humanity in control online, and we always appreciate any shares or comments on our articles — it helps!

Tolga Bilge, Andrea Miotti