If you think you’ve excluded all Chinese AI from your tech stack, you may need to think again.
The US government’s attitude toward Chinese AI is that it is a national security threat. This alone could persuade patriotic Americans to eschew any AI labelled ‘Chinese’ in favor of an AI without that label. Cisco has published details on new research from itself and VAIL demonstrating that national labels do not give an accurate picture of model lineage and characteristics.
In a blog post titled “The ‘U.S. vs. China’ AI Trap: An Incomplete Proxy for AI Security,” Cisco argues that country labels do not provide a complete picture of an AI model’s components because of a phenomenon it calls provenance entanglement.
Researchers used two AI model fingerprinting methods to analyze model weights and behavioral patterns, and found that they do not accurately or necessarily reflect the model’s publisher name and country of origin.
They suggest that AI models require their own form of SBOM but note there are difficulties: “the dependencies aren’t listed in a manifest file, they’re embedded in the learned weights themselves.” This is because a model producer is likely to fine-tune its AI with existing checkpoints rather than training it from a blank page. In short, it can inherit weights, biases and behavioral patterns from a different model originating in a different country to the one indicated on the label. A US model could contain behaviors inherited from a Chinese model while a Chinese model could inherit US characteristics.
The developer and country of origin on the AI label retains value, says Cisco, providing insight into the accountable developer, the applicable jurisdiction, and ‘authorized’ procurement process – but it does not provide an accurate assessment of what may exist inside the model.
Nemotron and Qwen models were chosen for study since it is known that some Nemotron models use Qwen base weights.
The researchers looked for surviving relationships between the two families, using Cisco’s Model Provenance Kit and VAIL’s Behavioral Fingerprinting. The former examines the artifact from the inside, while the later examines inference behavior from the outside.
“Both methods found Nemotron models built from Qwen base weights to be substantially more similar to Qwen models than chance would predict.” Their ultimate conclusion is that post-training and a new publisher name do not necessarily erase detectable relationships to an upstream model family.
This is important, since model lineage can create issues similar to the software supply chain threat for which SBOMs were created. For example, say the researchers, “If an upstream model is later found to contain a backdoor, systematic bias, or exploitable behavior, organizations would need to know which downstream models may warrant review.”
It suggests three areas that need improvement for the efficient use of AI:
Enterprises considering the use of a particular model should, “treat the publisher identity as one piece of the puzzle.” Due diligence should include lineage, training dependencies, behavior analysis, and operational control: the name on the label is not a proxy for potential risk.
Regulators need a better understanding of a model’s upstream dependencies to build a true picture regarding vulnerabilities, biases and restrictions stemming from model lineage.
AI developers should treat lineage disclosure as routine, not optional. As in all things, transparency is the best disinfectant, and it would allow users to understand upstream dependencies before integrating a model into their tech stack.
In a geopolitical context, a complete model is often known simply by the labeled country of origin. But this can produce blind spots and false equivalences.
“A model bill of materials could extend responsible AI adoption by recording base checkpoints, derivation methods, major datasets, synthetic-data generators, teacher and reward models, licenses, and entities with post-deployment access. Technical fingerprints can corroborate those disclosures or identify relationships that may warrant further review. The industry doesn’t need to wait for regulation to make that routine.”
So, conclude the researchers, while country of origin labels have relevance, they do not define a model’s technical lineage. “Models do not have passports. They have supply chains.”
Related: Cisco Releases Open Source Tool for AI Model Provenance
Related: AI Weights: Securing the Heart and Soft Underbelly of Artificial Intelligence
Related: Bias in Artificial Intelligence: Can AI be Trusted?
Related: AI and Cybersecurity – Everything You Wanted to Know, But Were Afraid to Ask
