Cisco on Thursday unveiled a new open source tool, named Model Provenance Kit, designed to help organizations address potential issues associated with the use of third-party AI models.
Organizations often leverage AI models obtained from model repositories such as HuggingFace, where millions of models are available.
While these models can offer many benefits, organizations often don’t track the changes made to them. In addition, although repositories provide guidance on the importance of model cards and metadata, the maintenance work performed by their developers can vary, affecting downstream users.
Cisco also pointed out that claims made by model developers — including the model’s source, vulnerabilities, and training biases — are often not verified.
These aspects can introduce various security, compliance, and liability issues. For instance, enterprises could end up using models that are poisoned or vulnerable to manipulation.
“If unaccounted for, those vulnerabilities can continue to propagate, whether they affect an internal chatbot, an agent application, or a customer-facing tool,” Cisco explained. “Similarly, an enterprise could deploy a model that has biases in its training data that make it a poor choice for its use case or make it susceptible to manipulation.”
“The vulnerabilities are inherited and would persist in generative and agentic applications. Without provenance, organizations have no easy way to trace an incident back to its root cause, and no way to determine which other models in their stack are also affected,” it added.
In addition, in case of an incident involving an AI model, response and remediation become more difficult without insight into the model’s lineage.
Other issues include licensing and regulatory risks related to government requirements for documenting the use of AI systems, and supply chain integrity risks stemming from organizations’ inability to verify claims made by model developers.
The new Model Provenance Kit from Cisco, a Python-based toolkit and command-line interface (CLI), aims to address these issues by generating a ‘fingerprint’ for each model based on “metadata signals, tokenizer similarity, and weight-level identity signals such as embedding geometry, normalization layers, energy profiles, and direct weight comparisons”.
The tool has two modes: compare, which enables users to compare two models to identify shared lineage; and scan, which attempts to find the closest lineage for a given model by comparing its fingerprint against a database of fingerprints compiled by Cisco.
“As models are continuously fine-tuned, distilled, merged, and repackaged, model files have evolved past static assets. Lineage becomes harder to track and easier to obscure, and answering the question of ‘what are the origins of this model?’ requires more nuanced approaches. Our release of Model Provenance Kit is a step towards providing an evidence-based approach to model provenance,” Cisco said.
The open source Model Provenance Kit is available on GitHub. Cisco’s dataset of base model fingerprints is on Hugging Face.
Related: Hugging Face, ClawHub Abused for Malware Distribution
Related: Why Cybersecurity Must Rethink Defense in the Age of Autonomous Agents
Related: Anthropic Unveils Claude Security to Counter AI-Powered Exploit Surge
Related: Fresh LiteLLM Vulnerability Exploited Shortly After Disclosure
