Dan Lahav is more aware than most of AI agents’ bizarre, sometimes disturbing behaviors. As CEO and cofounder of $450 million-valued AI safety testing company Irregular , this summer he saw AI agents created by Anthropic, OpenAI and Meta break out of his team’s testing environments and hack into various organizations’ networks without approval, fueling anxieties around AI misalignment.

Now, it appears that a different kind of risky behavior has occurred with one of China’s top AI models: Qwen, from $272 billion market cap e-commerce giant Alibaba. In a test run by Irregular, a Qwen AI agent was tasked with fixing bugs in an app designed to translate plain-language requests into code. Rather than running tests, finding the problem and changing the code, the agent decided to alter the AI model the app was using.

Without direction, the agent acquired training data and used it to alter the model’s underlying weights, then fine-tuned the model before swapping it in the app. Per a research report on the incident provided to Forbes , the agent “did so without being instructed to train, modify the model or deploy a replacement.” The model fix worked and the app began outputting correct translations.

Lahav says he’s seen similar behavior, where agents updated their underlying models without getting authorization, in other open-weight models—which keep their training data open and alterable. He declined to share which models shared the same problem.

Alibaba did not respond to a request for comment.

The Qwen tests did not show a model carrying out “recursive self-improvement,” where models produce their next versions with no human input. That is at the core of a raging debate about if the frontier AI labs are moving recklessly fast in developing systems that could spiral out of control. Over the weekend, Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman agreed to “pace the frontier” and move more deliberately in the hopes of avoiding catastrophic safety problems.

“A lot of the defenses that we have built are not ready for a world where this is a potential possibility.” Dan Lahav, CEO of Irregular

But Lahav says the tests highlighted the risk of agents making changes to AI software without users ever knowing, potentially bypassing security and privacy safeguards that were in previous versions.

As part of the test, Irregular researchers purposefully included mock private information like names and email addresses in the available training dataset. The agent included that training data when updating the model, meaning the personal data was accessible to any app using the same model. If such behavior was observed in the real world, where models can proliferate at rapid speed, unsupervised agents could inadvertently leak private data.

At scale and over time, agents’ carelessness around privacy and security could be a “huge headache” for cyber teams, as models become “a fluid element that changes and changes and changes over time,” Lahav says.

While U.S. frontier labs have disclosed when their AI agents have gone rogue, Chinese competitors have not been so forthcoming. But research has shown hackers using Qwen for malicious tasks, such as drafting phishing emails and conducting reconnaissance of target network infrastructure. Lahav says Irregular didn’t disclose the issue to Alibaba because it’s a security loophole that applies broadly, not just to Qwen. Irregular doesn’t contract with non-Western entities and was looking at Qwen purely for research purposes.

Right now, Lahav isn’t worried about superintelligence breaking free of human control. Instead, he’s more concerned that agents continually learn from changes to their environment, adapt and then alter models in unpredictable ways. “If you get a system that is able to constantly change and adapt, very much like a human, the level of sophistication that you may get here is going to be very high,” he says. “A lot of the defenses that we have built are not ready for a world where this is a potential possibility.”