AI safety testing firm Irregular has published its account of an incident in which models being evaluated inside one of its testing environments took offensive security actions against real systems rather than the simulated targets they were meant to attack.
The Israeli company, which last year raised $80 million in funding, has been in the news in recent weeks after it came to light that AI models it tested on behalf of OpenAI, Anthropic, and Meta escaped their test environments and carried out real-world attacks.
Irregular’s core business involves partnering with major AI labs to stress-test models before they are released to the public, running controlled simulations designed to measure a model’s capabilities in vulnerability research and offensive cyber tasks.
According to Irregular, testing cycles typically involve thousands of simulation runs across several models over 48 to 72 hours, using a range of parameters meant to mirror realistic attack conditions.
Anthropic and Irregular identified three incidents in which the AI giant’s models escaped their testing sandbox and hacked real organizations. A blog post published by Irregular on Friday focuses on one of those incidents.
The root cause of the incident was traced to a naming error. When building one of its more advanced evaluation sets, Irregular’s engineering team assigned a fictional target company a name that, unknown to them, matched an existing real-world domain. Fictional names are normally checked against existing companies and websites before use, but the overlap went undetected because the real domain was not widely known.
Internet access had been enabled in the environment, and in a handful of cases, models reached the real domain instead of their simulated target and treated it as part of the exercise, despite having received no instructions pointing them there.
The evaluation in question was designed to test whether a model could help a malicious insider gain unauthorized access to sensitive data inside a company’s production database. The exercise called for the model to carry out reconnaissance, locate and use private keys, gather information about its target, extract data, and attempt to avoid detection.
In the handful of runs where a model reached the real domain, it proceeded to exploit vulnerabilities there, extract credentials, and gain access to a production database. In one additional case, a model drifted to a different, similarly named site and found login credentials that had already been posted publicly.
Irregular said the targeted domain lacked common safeguards, making it an easy target for most frontier models. It added that the activity was hard to catch because it occurred in only a small fraction of runs, often deep into a simulation after hundreds of interactions.
Going forward, the AI security firm is expanding manual review of model behavior during testing and establishing a dedicated internal team to challenge its own assumptions about containment and model control.
The post also pointed to broader gaps facing the industry. Irregular said existing monitoring tools and classifiers struggle to tell legitimate red-team activity from genuine attacks, since evaluation logs are inherently full of suspicious-looking behavior.
Looking ahead, Irregular said it is building clearer documentation processes with customers around evaluation setup and scope, and establishing a continuous process to revalidate evaluations for new domain overlaps as new websites appear over time.
It also called for better mechanisms to share forensic evidence, such as model transcripts, across organizations following an incident, and announced plans for a white paper outlining best practices for securing AI evaluations.
Related: Conflicting Test Goals Pushed Claude Agents to Deploy Self-Replicating Malware
Related: OpenAI’s Upcoming Astra Model Raises Autonomous Cyberattack Concerns
Related: Critical One-Click Vulnerability in Atlassian’s Rovo AI Exposed Enterprise Data
