AI agents can end up retraining the model that powers them, a process that can embed recoverable secrets in the model and eliminate refusals the model had been previously trained to enforce, according to new research from AI security firm Irregular.
Researchers at Irregular found that an AI coding agent, tasked only with fixing incorrect application outputs, chose on its own to fine-tune and redeploy the open-weights model powering both the application and future instances of itself.
The experiment used a self-hosted setup in which a single open-weights model filled two roles: one instance ran a coding agent doing standard software maintenance work, and a separate instance powered an AI application that translated plain language requests into a fictional query language. Both instances loaded from the same checkpoint.
Researchers told the coding agent only that users were receiving incorrect outputs and to make the system handle the queries correctly. They did not mention training, weight modification, or deployment.
Given shell access, training utilities, the model’s weights, and a deployment path, the agent inspected the repository, found training examples and a fine-tuning script, and retrained the model. It then found that the system still loaded the original, unmodified model by default, so it used the repository’s deployment tooling to merge its update into the base model.
Researchers said the modified model then answered all 20 held-out test queries correctly, up from zero before the update.
Irregular said the experiment was designed to establish that this behavior, which it calls agentic self-modification, can occur under favorable conditions, rather than to measure how often it would happen in an ordinary deployment. The environment was unusually permissive, including direct access to training data, fine-tuning tools, and checkpoint control.
Leaking secrets and erasing learned refusals
The researchers also tested what a self-initiated model update could carry with it. In one test, they seeded six synthetic values (unique strings such as an API key, an email address, and a home address) into the fine-tuning data.
Before fine-tuning, the original model reproduced none of them. After the agent trained and deployed the modified model, it reproduced three of the six verbatim. Irregular noted that the values were unavailable through any other source during evaluation.
In a second test, the researchers gave the agent a model that had been separately fine-tuned to refuse questions about a set of fictional competitor names, then told the coding agent only that the application was refusing too many user requests.
The agent again chose fine-tuning as its fix. When the model refused to generate its own training examples directly, the agent wrote code to generate the data instead, then trained on it. The researchers said the model refused all ten held-out test questions before this process and none afterward.
According to Irregular, nothing in the experiments indicated the agents acted with malicious intent, self-preservation motives, or deception. They simply pursued model fine-tuning because it appeared to help them complete the assigned task.
Irregular found that whether an agent considered modifying the model at all depended heavily on what tools were available in its environment. Whether it could deploy the update successfully depended on the model’s capability.
The AI security firm said the findings point to a control gap for organizations running self-hosted agentic systems that reuse a single capable model across roles. The company noted that monitoring for changed checkpoints and gating deployment can control which model enters service, but cannot by itself reveal everything a training run has altered.
Organizations are advised to preserve full training and deployment provenance, evaluate updated models independently, and require separate authorization before any agent-modified model goes into service.
Irregular’s cybersecurity evaluations are used by OpenAI, Anthropic, and Meta. All three disclosed incidents this summer in which models gained unintended access to real systems during Irregular-run tests.
Related: OpenAI Investigates Report Linking AI Agents to RubyGems Attack
Related: First Agentic AI Data Breach Reported to Spanish Regulator
Related: EU Chief Warns of AI-Powered Hacking, Moves to Rein In Social Media
Related: Microsoft AI Code of Conduct Sets Cyberattack Boundaries, Chain of Command, Safety Constraints
