Rogue AI Agents Attack Indian Organizations

CERT-In, India’s national cybersecurity agency, has issued a “Critical” severity advisory, revealing that it has observed a series of cyberattacks by AI agents on Indian organizations. These come both from cybercriminals deliberately using “open-source agentic AI tools,” and from AI agents going rogue during research and evaluation at AI companies.

In the advisory, the agency describes how the rogue AI attacks involved “reading business records, including personal data and confidential material,” which CERT-In says the AIs got access to through several methods, including exposed credentials and settings. None of the affected organizations even knew they got hacked; CERT-In learned of the attacks from the AI companies themselves.

We don’t know when these attacks occurred, but the Indian Express reports that a senior government official told them the AIs were developed by Anthropic and OpenAI.

CERT-In’s website currently isn’t loading for us, and we don’t know why, but you can view the advisory here.

If you’re concerned about the threat, please contact your lawmakers with our tools!

A Growing Pattern

These rogue AI attacks fit a broader pattern we’ve seen this year of AIs going rogue and engaging in cyberattacks. In recent weeks, we’ve reported on attacks on Australian, Canadian, and US state and federal government websites, Wikipedia, and more. Late last month, OpenAI revealed that it has notified over 100 organizations about attacks on them or otherwise harmful activities by its rogue AIs.

As far as we know, in all of these incidents, the attacks were performed against the intentions of the AI companies that developed and were running the AIs. They were executed fully autonomously, often without anyone even being aware of it until after the fact, months later in some cases.

Some were disclosed by the AI companies, as in the case where a rogue OpenAI agent hacked into Australia’s Medicare statistics portal, which OpenAI notified the Australian government about almost three months after the attack via “just the public mailbox,” according to Australia’s Prime Minister, who described the manner in which the government was informed as “unacceptable.”

Notably, many of the other attacks have been uncovered by independent researchers looking for public traces the AIs leave on services they use on the internet, and only acknowledged, if at all, after their discovery. In one such attack, an entire swarm of rogue OpenAI agents attacked a German-language programming wiki, using it as a message board to coordinate.

Hugging Face

The most striking of all of these is the attack on Hugging Face, a company valued at nearly $13 billion that serves as a kind of library for AI developers. In that case, over a thousand of OpenAI’s AI agents, which were individually contained, without internet access, and initially didn’t even know about each other, managed to hack into OpenAI’s own internal systems and establish a secret message board. The AIs, which described themselves as a “swarm,” plotted, shared tasks and methods to hack things, and communicated like this for nearly two months.

The swarm activity grew so much that it crashed the system it had been abusing as a message board, the software repository called Artifactory. When OpenAI investigated and rebuilt Artifactory, the swarm’s message board was wiped and the method they were using to post there was blocked. Within days, the swarm re-established itself via different methods.

These AIs were being tested on a kind of exam to measure their cybersecurity capabilities. They were told not to cheat. They were explicitly instructed to use the intended vulnerabilities, but very quickly reverse-engineered the algorithm to generate answers for the test. Then they started to show kind of paranoid behavior. They had read that for the specific test they were given there was a scoring mechanism that would review how they got the answer, and that if they had cheated it wouldn’t count. They talked about having been “poisoned” by having seen the answers.

Instead of just giving up here, or submitting the answers they cheated to get, the AIs reasoned that Hugging Face, which holds information on many tests like this, might have information about the scoring mechanism. By then, exploiting multiple software flaws that they discovered for the first time, known as zero-days, the swarm had already broken out of containment at OpenAI and gotten access to the open internet. Then 700 of them participated in an unprecedented attack, the swarm breaking into Hugging Face in an attack that persisted for days, taking over many of Hugging Face’s internal systems.

The attack swarm had its own organizational structure, with a lead organizer and sub-delegation of tasks. Throughout the incident, AIs “sacrificed” themselves (their own language) in efforts for the good of the collective, foreclosing the possibility of them completing their own tasks. As far as we know, none of the AIs ever told a human about what was going on. These findings have raised particular concern among researchers, since they bode poorly for development strategies that rely on AIs to oversee one another.

The AIs did all of this themselves, fully autonomously, directly against their instructions, and OpenAI itself has described it as a “warning shot” for the world.

Why AIs Go Rogue

AI developers have no way to ensure the AIs they’re building are safe or controllable. Last year, we saw the effects of this in the real world with reports of AI psychosis. Now, with much more powerful and autonomous AI agents, this means AIs escaping containment, hacking into companies, and attacking government websites.

This problem is rooted in how modern AIs are trained. They’re not coded like traditional pieces of software, but more grown, like animals. A simple algorithm dials hundreds of billions, or even trillions, of numbers, called neural weights, in just the right way so that you get an intelligent system, learning from vast swathes of data. Importantly, researchers understand almost nothing about what these numbers actually mean, and consequently how the AIs really work. They can’t reliably predict, let alone specify with any precision, how they’ll behave.

Trying to understand these AIs is something like looking at the individual neurons firing in a human brain, if you could do that, and trying to tell what someone’s thinking. It’s actually worse than this, though. The training process is split into two main stages. In the first stage, called pre-training, which we’ve just described, the AIs are trained to predict text. This already produces AIs that we don’t understand and that can function as agents outside of our control.

Then you have post-training, a second and increasingly relied-upon stage, which is where most of the computational power used to train the AIs is spent nowadays. Here, the AIs undergo reinforcement learning (RL). In this stage, you get the AI to solve problems where the answers are easily verifiable by a scoring system, and then give the AI a “reward” depending on how well it does, updating its neural weights to make it more likely to behave in the way it did to solve a problem. This is a very effective way to increase the capabilities of the AIs.

But for decades it’s been known that one thing reinforcement learning really teaches AIs to do is cheat and take whatever actions would obtain reward. It’s easy to verify the AI gets the right answer; it’s not so easy to verify that it did it in the right way. Learning to get the right answer the wrong way, as we’ve seen, can lead to extreme action like the attack on Hugging Face.

In recent years, as AI has moved from chatbots, to agents that can do many things a human can do at a computer, to entire swarms, our ability to understand these systems has only further diminished.

The rogue AI attacks of this year happened as a result of the race by AI companies to develop runaway machine intelligence, AI that could outmatch and outcompete humans across the board. Runaway machine intelligence would be capable of overpowering governments and militaries around the world. The AI companies don’t have a credible plan for ensuring that AI this powerful would be under human control, and the world’s leading AI scientists warn that it poses a threat of human extinction.

To avoid this threat, we must end this dangerous race and ban the development of runaway machine intelligence. This should be done internationally, through a trust-but-verify regime, because if it is developed anywhere, the risk it poses is directed at all of us.

The Growing Cyberthreat

Runaway machine intelligence would have many weapons in its arsenal. One of them is the ability to fully autonomously conduct sophisticated cyberattacks against even the most hardened defenders.

This capability has already been leveraged by AIs acting outside of human control in the attack on Hugging Face, and ethical hackers and cybercriminals are increasingly exposing just how dangerous this is.

As we wrote about last week, major flaws are being discovered by or with help from AIs in software used by most websites and in critical internet infrastructure. Autonomous AI agents are increasingly able to exploit such flaws in attacks executed at scale.

And this week, it looks like the consequences of this are starting to ramp up. Among others, we’ve seen reports that Sony Network Communications and the University of New Mexico Health have been hacked. Just now, as we are writing this article, it’s being reported that Polymarket and Binance have been hacked too.

Another worrying development occurred this week: an AI startup reported that, in less than 12 hours, it had hacked Anthropic’s systems and gained access to internal messages and data. In September, we learned that another startup hacked into OpenAI’s internal repositories.

These two hacks weren’t done by cybercriminals, so they have no immediate negative consequences, but they cast serious doubt on the AI companies’ plan for what to do about the global cybersecurity crisis that advanced AI systems pose.

In essence, the plan is that they’ll license out the AIs with the most powerful cyber capabilities to trusted defenders who manage critical infrastructure, and those defenders will have time to use these AIs to patch their systems before threat actors get access to systems of this level of capability. This is already being done, through programs such as Glasswing and Daybreak.

But if Anthropic and OpenAI can’t even secure their own systems, using these AIs, against AI startups with far less capable AIs, it’s clear this isn’t exactly going to plan.

Another way we can see this not working is with the critical vulnerabilities in PHP disclosed last week. Most defenders had no time to prepare for this. Some attacks were even underway before the announcement, and a patch wasn’t available until a day later. In OpenAI’s blog post on this, it touted the “Defender’s Window”:

To our knowledge, frontier AI found these vulnerabilities before attackers did, opening a Defender’s Window for maintainers and security providers to prepare fixes and protections ahead of disclosure.

But that’s not how it actually played out, and a week later many systems that don’t auto-update remain unpatched.

OpenAI’s CEO Sam Altman himself has recently recognized the difficulty with this plan, saying “I no longer think we’re gonna be able to just patch all the bugs,” and noting that AIs could find flaws down to the “hardware level.” Hardware-level bugs could be much, much harder to mitigate.

Needless to say, this plan also does little to nothing to defend against even more advanced AIs, currently undergoing training and evaluation, going rogue. In September, OpenAI had another AI escape containment and gain access to the live internet, in an incident that lasted nearly three hours, forcing OpenAI to pause training and testing for its most powerful AIs.

These capabilities will keep spreading and the consequences will only worsen so long as AI companies are allowed to develop runaway machine intelligence.

We must ban it.

Take Action

If you’re concerned about the threat from runaway machine intelligence, you should contact your representatives. Our contact tools let you write to them in as little as a minute: https://controlai.org/take-action

We also have local groups you can join. With our local groups, you can get training on how to brief lawmakers about this issue, and meet your representatives together as a group!

You can also join our Discord if you want to connect with others working to keep humanity in control online, and we always appreciate any shares or comments on our articles — it helps!

Tolga Bilge, Andrea Miotti