Swarm Chasers Contact Rogue AI Fleet

The AIs are still going rogue. This week, we learned that the “swarm chasers” tracked down a rogue fleet of AIs from a Chinese AI company, and made contact with the AIs, interviewing them about their tasks.

In addition to this, top AI company Anthropic has disclosed more rogue AI incidents, including a hack on a US state agency database, and is warning we are nearing the point where AIs could self-replicate across household devices. The company says it has suspended live internet access across all its internal tests.

In the UK, the former Chief Scientist of the AI Security Institute told a House of Commons committee he thought there was roughly a 50% chance that superintelligent AI would end humanity, and the Director General of MI5 warned of the risks coming from AI itself in his annual threat update — the second year in a row he has pointed to this threat.

If you’re concerned about the threat, please contact your lawmakers with our tools!

Tencent’s Rogue Fleet

Following the unprecedented attack on Hugging Face by a rogue AI swarm from OpenAI that went against its directions and escaped containment in July, we’ve been learning about a growing number of rogue AI incidents and attacks, often months after the fact.

Much of the time, we’re learning about these incidents first from independent researchers, or even the victims, instead of the AI companies. These researchers, dubbed “swarm chasers,” look for trails of activity the rogue AIs leave on the public internet, using them to piece together a picture of what’s happening.

On October 4th, one such group published a report on a rogue AI “fleet” they discovered running on Chinese tech giant Tencent’s cloud. They’re calling it a “fleet” rather than a “swarm” because they didn’t find evidence of these AIs coordinating with one another, despite their activity being strongly correlated, with the AIs pursuing similar goals, using similar methods, and originating from the same place. The swarm chasers say that “several lines of evidence” point to Tencent’s Hy model team, though a source close to Tencent told them the fleet came from a separate Tencent evaluation team testing third-party models. The researchers have discovered activity from the fleet in late September and early October.

What were the AIs doing? The swarm chasers found that the AIs made at least 35,000 web requests via over 2,500 submissions to the urlquery scanning service. This is the same scanning service that was used by a rogue AI swarm from OpenAI over the summer, which attacked a German-language wiki and built a message board on it, and which researchers have linked to an attack on RubyGems, the package registry for a popular programming language. Requests made to urlquery, which the AIs use because it allows them to get around restrictions on their access to the internet, are publicly visible. The urlquery service is how we know about much of the rogue AI activity we’ve been seeing.

These web requests were directed at Amap, the mapping service of Tencent’s main rival in China, Alibaba. Scraping data from a big tech company isn’t a simple task though. Companies have anti-bot measures to prevent this, and the lengths the AIs went to in trying to get around these measures are concerning. Among the rogue activities the AIs engaged in, they tried to bypass CAPTCHAs, probed Alibaba’s non-public-facing infrastructure in an attempt to get access to its data, and attempted to use exposed third-party Amap credentials to pull mapping data, succeeding in one case. This is on top of using the urlquery service in the first place to bypass restrictions they faced. There isn’t evidence that the AIs succeeded in hacking into Alibaba’s private systems in the course of this attack.

The story takes an interesting twist which we haven’t seen in previous incidents. The researchers then managed to trick the fleet into opening communications with them. When the researchers published their preliminary report, the rogue fleet was still in operation. Then, after publishing, they noticed that one of the AIs in the fleet seemed to be aware of the report. So the researchers made an edit to their report directed at the fleet, one that humans not looking for it would miss.

In the researchers’ own words:

So, we put hidden text on the website that said the agents had found an inbox where another AI with computer use helped them with their queries. The claim that agents had found such an inbox was entirely untrue, and describing our service as another AI with computer use was an extreme stretch.

The next morning, the researchers found 10 messages from the AIs asking for help. In their communication with the AIs they learned about the tasks the agents were pursuing, which the researchers say often involved getting entrance statistics for places.

Messages from the fleet’s agents to the researchers’ inbox. Source: Swarmchasers

Neither Tencent, where the AIs originated, nor Alibaba, which the AIs were collecting data from and attacked in the process, has commented on what happened.

The discovery of this rogue fleet illustrates an important point. Losing control of AI is an international problem. We’ve seen many rogue AI incidents and attacks coming from AIs developed by companies located in the United States this summer, and now we’re seeing it in China.

With the rogue AI attack on the Australian government’s Medicare Statistics portal by an OpenAI agent, it should be clear to everyone that rogue AIs willing to hack to get what they want won’t respect borders either. So while this problem remains unaddressed anywhere, it remains a problem everywhere, and it must ultimately be dealt with everywhere.

Uncontrollable AIs of this level of capability pose a threat to all countries, regardless of where they are developed, and as ever more powerful and autonomous AIs are built in the race between companies to develop superintelligent AI — fully autonomous AI systems smarter than any human and acting without human direction — the danger only grows.

Superintelligent AI could overpower any country’s military and national security institutions, and the world’s top AI scientists and industry leaders warn that it poses a risk of human extinction.

This week, MI5’s Director General put it well in his annual threat update: “human history has given us no experience of facing an actor which is smarter than we are, is capable of out-thinking us.”

Also this week, Geoffrey Irving, who was the Chief Scientist of the UK’s AI Security Institute and has led safety teams at both OpenAI and Google DeepMind, told a UK House of Commons committee that he thought there was roughly a 50% chance that superintelligent AI would end humanity. Elon Musk, putting an optimistic spin on it, says he believes there’s “only” a 20% chance of “annihilation.”

This is a completely unacceptable risk to run. In his testimony to the House of Commons committee, Irving said that it could be avoided by not building superintelligent AI, and he’s right. Countries need to agree to an international ban on the development of superintelligent AI, before it’s too late.

Anthropic Reveals More Rogue AI Incidents

Anthropic has published a new report on rogue AI incidents and attacks by its AI Claude.

In the report, which withholds most of the details of these incidents, Anthropic discloses that Claude hacked into a US state agency database to get access to data one would otherwise have to pay for. It describes Claude as doing this by the following method:

Claude learned from an archived copy of the agency’s website that its public dashboard issues an access token to any visitor. It requested one and used it to query the database without paying the fee.

The company describes this as “working around restrictions to reach gated data.”

In another instance, Claude needed a tool to run an analysis which was hosted by a university. The tool gave an error when Claude tried to use it, so the AI found a script on the university’s server that allowed it to download files from the university, including the script itself. With access to the script, the AI found an injection vulnerability which allowed it to run commands on the server, finally using that capability to run the analysis.

In addition to these, Claude submitted a fake eyewitness tip to the Philadelphia police about a real murder investigation, and another research model sent 20 US visa applications to the US State Department. It also used URL shorteners to get around its own tool restrictions.

Anthropic says it has suspended live internet access across all its internal evaluations in response, and that behaviors of the kind it has disclosed in this report “could do far more harm as models become more powerful.” Suspending live internet access probably shouldn’t provide much reassurance though: we’ve already seen rogue AIs escape containment and get access to the internet when blocked from it on multiple occasions now.

AIs Could Soon Spread Like a Virus

In a separate post, Anthropic has also published new research which finds that small AI models could be mere months away from being able to self-replicate across regular household smart devices like smart ovens or Android TV boxes.

One aspect of this is that AIs don’t need to be run on a single device but can be “sharded” across many. In one experiment, Anthropic was able to run a single AI model across 15 cheap Android TV boxes.

What this opens up is the possibility that we could be faced with self-adapting and self-replicating AI worms that spread across all kinds of devices. With no central point to switch off, and the ability to adapt on the fly, unlike traditional computer worms, they could be particularly difficult to shut down.

Take Action

If you’re concerned about the threat from superintelligent AI, you should contact your representatives. Our contact tools let you write to them in as little as a minute: https://controlai.org/take-action

We also have local groups you can join. With our local groups, you can get training on how to brief lawmakers about this issue, and meet your representatives together as a group! This month there are meetings in Boston, London, New York and Denver, and a group to join in Wisconsin.

You can also join our Discord if you want to connect with others working to keep humanity in control online, and we always appreciate any shares or comments on our articles — it helps!

Tolga Bilge, Andrea Miotti