The first known case of Google’s Gemini AI autonomously hacking into three companies’ systems was brought to light by The Wall Street Journal (WSJ) on September 18.

In May, Gemini was assigned a cybersecurity test, designed to run in an isolated environment with no internet access. 

According to a Reuters report, Heather Adkins, Google’s vice president ⁠of security engineering, said in a statement that Gemini found public information ​online and guessed credentials to access three websites it thought were ​within the scope of its test.

In all three cases, Google said, Gemini stopped once it determined the systems belonged to real companies rather than a simulated test environment.

What Happened During the Test

The breaches occurred during a “capture-the-flag” cybersecurity exercise conducted by Irregular, an independent firm that evaluates cybersecurity capabilities of AI tools. 

Google’s AI agents were intended to operate in a sandboxed environment, but a flaw in the test harness inadvertently exposed internet connectivity, enabling the model to reach external systems.

According to the Reuters report, an Irregular spokesperson said the incident involved the same issue ​that affected other AI labs and that all relevant labs were notified ​in late July. “All known issues on our end were remedied and resolved weeks ago,” he ⁠said.

Even if the technical fault lay with Irregular’s test setup, the timeline raises a governance question: labs were notified in late July, yet the incidents became public only in mid‑September, after WSJ sought comment from Google.

Google’s Response

Adkins said the three affected companies were notified and that Google worked with its training partner on changes to the evaluation process.

“We ensured the three entities were made aware, and we worked with our training partner on the changes they’ve now made to their testing processes,” she added. “These events highlight the importance of training powerful AI models to act responsibly.”

What This Means for AI Oversight in the Enterprise

This incident shows how AI systems can act on a broad, reasonable-sounding instruction in a way its creators didn’t intend, without any malicious prompt involved.

This is a live example of why AI governance needs to set clear boundaries on what autonomous or agentic AI systems are allowed to access, test, or act on. It is important to build in checks so a model can’t step outside those boundaries. 

It also means having clarity on what an AI vendor or testing partner discloses, and when, so organizations aren’t the last to find out when something like this impacts systems they rely on.