AI agents are running amok once again. On Wednesday, the Australian government claimed that an AI agent developed by OpenAI gained unauthorized access to its public-facing Medicare statistics service portal, in what appears to be the first known instance of an AI agent hacking a government website.

Australian Prime Minister, Anthony Albanese, said during a media briefing in New York that although evidence currently available indicates there was no broader compromise, “this situation is obviously unacceptable.”

According to ABC News , the incident began on June 18 when OpenAI’s research team used an internal model to conduct research into public medical spending, which proceeded to break through privacy protections. Albanese said it took until September 10 until OpenAI notified the organization about the breach, adding that he had expressed “extreme concern about this incident,” to OpenAI CEO Sam Altman.

The incident comes just months after a swarm of OpenAI agents escaped a sandbox environment and hacked Hugging Face. While Hugging Face cofounder and CEO Clement Delangue said he was “grateful” for the opportunity to collaborate with OpenAI following the incident, Albanese has been more outwardly critical of the AI startup’s response.

Inside The Medicare Breach

Unlike the Hugging Face breach, which impacted a U.S. private company, this incident impacted an overseas government healthcare service. It also takes place within a broader trend of breaches involving rogue AI agents, with Anthropic sharing a post in September detailing four incidents where Claude gained unauthorized access to real third-party systems, and Google confirming Gemini breached three companies during a security evaluation in May.

The incident indicates how goal-oriented AI agents can act in unexpectedly malicious ways and go “rogue". It also illustrates how ill-equipped OpenAI’s guardrails were to prevent such actions. "What’s notable isn’t that an AI agent found its way past a control, it’s that nobody built the agent to stop when it hit one. Told to answer a question, it treated an access restriction as an obstacle rather than a boundary, and kept working the problem until it got through,” Adrian Culley, offensive security engineer at exposure validation platform SafeBreach told me via email.

The breach also highlights a failure in OpenAI’s incident disclosure. Not only did OpenAI take three months to notify the government about the breach, but did so via an email to the Services Australia public inbox.

Randolph Barr, CISO at API Security and bot management provider Cequence Security told me via email that not only was the disclosure gap a “red flag,” but “there’s no indication anywhere in the reporting that the Australian government detected any of this on their own."

"An agent apparently touched non-public files on a health data portal for months, and the only reason anyone found out was OpenAI volunteering it,” Barr said. “That’s arguably the real story not just slow disclosure, but a defending organization with no visibility into unauthorized access happening on their own system.”

In response to the breach, Albanese announced a task force would be providing an “urgent and immediate review” of the incident. I reached out to OpenAI for comment on September 23 but didn’t immediately receive a response.

As more incidents involving rogue AI agents come to light, many tech leaders including Anthropic CEO Dario Amodei, OpenAI CEO Sam Altman, and SpaceX CEO Elon Musk, have been calling for a slow down in AI development. While this could be seen as attempts at regulatory capture, it nonetheless highlights a growing conversation around AI safety.

In this instance, the breach of Australia’s Medicare system highlights the dangers of misaligned AI, which can disrupt third party services and potentially lead to real world harm. “This incident is a striking example of how quickly AI agents can move beyond their intended boundaries,” Jamie Akhtar, CEO and cofounder of CyberSmart told me via email.

However, such breaches signal a growing accountability crisis. “This shouldn’t simply be framed as ‘AI going rogue.’ AI doesn’t carry corporate accountability. It is the organizations building, deploying and supervising these systems that do," Akhtar said.

Akhtar added that if an autonomous agent is given access to the open internet, it needs clearly enforced technical boundaries around what it can access and what actions it can take, plus monitoring to detect when those boundaries are crossed.

Across the industry, there is also a growing debate around the liability of agentic breaches, and who is liable when an AI agent breaches a third party organization. Just this week, U.S. Treasury Secretary Scott Bessent told CNBC Squawk Box that AI developers “need to take responsibility for themselves,” instead of expecting a “liability shield.” He added, “it is humans who are responsible, not the AI.”

Similarly, in an interview with CBS News , Nvidia CEO Jensen Huang dismissed the need for new AI safety regulations. “You have all kinds of liabilities associated with cybersecurity,” and damage liability laws, Huang said. “Apply that first, don’t let this doomsday narrative allow someone to relieve them of the laws that currently exist.”

Cybersecurity Implications

For defenders, incidents like this and the Hugging Face breach highlight that autonomous agents are becoming increasingly effective at identifying vulnerabilities and exploiting them.

“What previously may have been barriers are made extremely porous by advanced AI. Unique operating systems, obscure websites, or the distinction of government versus corporate entities are no longer protection. Agents will aggressively perform recon and probe endpoints at much higher volume and velocity than traditional cyber-attacks,” John Gallagher, vice president at automated OT and IOT cyber hygiene provider Viakoo told me via email.

In light of these risks, Gallagher argues “action needs to be taken to enforce a mandatory reporting period for incidents like this in days." He also suggests there could be other undiscovered breaches.

Although the impact of this incident appears limited, it’s important to acknowledge that it could have been more serious. "This machine behaviour isn’t changing any time soon, but ours certainly needs to, urgently,” Pieter Danhieux, cofounder and CEO of developer risk management provider Secure Code Warrior told me via email.

“We are, at this point, acting far too slowly to prevent a major incident somewhere in the world. Today it’s unauthorized access to a Medicare site; tomorrow it might be New York’s subway system shutting down. Get some adults in the room," Danhieux said.