Details of eight vulnerabilities found in the open source supply chain used to develop in-house AI and ML models have been disclosed by AI cybersecurity startup Protect AI. All have CVE numbers, one has critical severity, and seven have high severity.
The vulnerabilities are disclosed via Protect AI’s February Vulnerability Report. They are:
- CVE-2023-6975: arbitrary file write in MLFLow, CVSS 9.8
- CVE-2023-6753: arbitrary file write on Windows in MLFlow, CVSS 9.6
- CVE-2023-6730: RCE in Hugging Face Transformers via RagRetriever.from_pretrained(), CVSS 9.0
- CVE-2023-6940: server side template injection bypass in MLFlow, CVSS 9.0
- CVE-2023-6976: arbitrary file upload patch bypass in MLFlow, CVSS 8.8
- CVE-2023-31036: RCE via arbitrary file overwrite in Triton Inference Server, CVSS 7.5
- CVE-2023-6909: local file inclusion in MLFlow, CVSS 7.5
- CVE-2024-0964: LFI in Gradio, CVSS 7.5
The nature and vulnerability of open source code is well understood. For standard code development using OSS libraries, SBOMs are designed and used to provide some security surety. But SBOMs don’t work with open source used for AI/ML development.
[ Learn More at the 2024 AI Risk Summit – June 2024, Ritz-Carlton, Half Moon Bay ]
“Traditional SBOMs only tell you what’s happening in the code pipeline,” explained Daryan Dehghanpisheh, co-founder of Protect Ai (and former global leader for AI/ML solution architects at AWS). “When you build an AI application, you have three distinctly different pipelines where you must have full provenance. You have a code pipeline, you have a data pipeline, and more critically, you have the machine learning pipeline. That machine learning pipeline is where the AI/ML model is created. It relies on the data pipeline, and it feeds the code pipeline — but companies are blind to that middle machine learning pipeline.”
Protect AI has called for the development of an AI/ML BOM to supplement SBOMs (software) and PBOMs (product) in a separate blog: “The AI/ML BOM specifically targets the elements of AI and machine learning systems. It addresses risks unique to AI, such as data poisoning and model bias, and requires continuous updates due to the evolving nature of AI models.”
Absent this AI/ML BOM, in-house developers are reliant on either their own expertise or that of third parties (such as Protect AI) to discover how vulnerabilities can be manipulated within the hidden machine learning pipeline to introduce flaws to the final model before deployment.
Protect AI has two basic methods for AI/ML model vulnerability detection: scanning and bounty hunters. Its Guardian product introduced in January 2024 can use the output of its AI/ML scanner (ModelScan) to provide a secure gateway. But it is the firm’s community of independent bounty hunters that are particularly effective at discovering new vulnerabilities.
The firm launched what it calls the world’s first AI/ML bug bounty program — which it calls huntr – in August 2023. “We now have close to 16,000 members of our community,” said Dehghanpisheh. “When we first launched in August, I think we were getting maybe three a week. Now we get north of 15 a day.”
The bounties are funded from Protect AI’s own capital (it raised $35 million in July 2023 just prior to launching huntr), but he believes it is money well-spent. Firstly, finding vulnerabilities highlights the complexity of a new technology that is neither well understood by the developers using the technology, nor the many traditional security companies who don’t understand the complexities of machine learning and artificial intelligence.
Secondly, the volume of vulnerabilities found and disclosed positions the firm as a leader in what he calls ‘AI/ML threat IQ’. “There are two components to this,” he continued. “The first is the Threat Reports on discovered vulnerabilities, and the second is our AI exploits tool that we give away for free.”
The third advantage is that having reached this position through the success of the huntr program, it is an effective lead generation tool. “The way I see it, paying those bounties is cheaper than paying a salesperson.”
The huntr program has proved successful in finding vulnerabilities and is likely to continue being successful. The monthly Threat Reports actively demonstrate the complexity of securing in-house developed ML/AI models by disclosing the vulnerabilities found by the hunters.
Related: Don’t Expect Quick Fixes in ‘Red-Teaming’ of AI Models. Security Was an Afterthought
Related: The Emerging Landscape of AI-Driven Cybersecurity Threats: A Look Ahead
Related: Addressing the State of AI’s Impact on Cyber Disinformation/Misinformation