In an analysis of bug bounty programs, a trio of academic researchers concluded that the programs were cheaper to run than hiring expert security researchers to find software vulnerabilities.
Vulnerability rewards programs can range anywhere from two to hundreds of times more cost-effective than hiring expert security researchers to find vulnerabilities, according to a study on vulnerability rewards programs. The paper (PDF) will be presented by Matthew Finifter, Devdatta Akhawe and David Wagner, graduate students in the computer science department at the University of California, Berkeley, at the USENIX Security Symposium in Washington, DC in August.
The “empirical” study focused on the programs Google and Mozilla have for collecting vulnerabilities for Chrome and Firefox, respectively. Over the past three years, Google has paid approximately $580,000 over 501 bounties, while Mozilla has shelled out $570,000 over 109 bounties, the researchers found. A little over a quarter, or 28 percent, of Chrome’s security advisories during this time period were the result of bugs identified through the program. For Mozilla, the figure was closer to 24 percent for Firefox security advisories.
Despite different participation levels, both programs were “economically efficient” and appear to be more cost-effective than hiring full-time security researchers, the authors wrote in the abstract.
“We therefore recommend that more vendors consider using them to their (and their users’) advantage,” the authors wrote in the study.
These programs are attractive precisely because the potential financial reward is an incentive for security researchers to look for vulnerabilities in the software. The increased attention generally improves the likelihood of finding latent vulnerabilities and zero-day issues before the attackers can exploit them for their malicious purposes. Vendors can also manage disclosures much more effectively and reduce the possibility of unexpected revelations.
While monetary rewards “provide an incentive for security researchers not to sell their research results to malicious actors in the underground economy or the gray world of vulnerability markets,” the researchers acknowledged the money may not be enough. The prices in the underground economy are “far higher” than those offered by the bug programs, according to the study.
Even so, there have been an “upsurge of interest” in these programs in recent months, with vendors expanding existing programs to include other products or collect other types of vulnerabilities, and others launching new programs. There are also vendors who act as an intermediary, or a broker, between the researchers and vendors so that the researcher doesn’t have to deal with the back-and-forth process of vulnerability disclosure.
Some vendors, notably Microsoft, still argue that vulnerability rewards programs do not represent the best return on investment on a per-bug basis, the researchers said.
Despite the difference in the number of bugs identified, the researchers found the total cost of paying out rewards for vulnerabilities affecting stable releases were similar for both Chrome and Firefox, at approximately $400,000. The average daily cost to date of each program differed, with Chrome costing $485 per day and Firefox costing $658 per day.
“If we consider that an average North American developer on a browser security team (i.e., that of Chrome or Firefox) would cost the vendor around $500 per day (assuming a $100,000 salary with a 50% overhead), we see that the cost of either of these VRPs is comparable to the cost of just one member of the browser security team,” the authors wrote.
However, the program’s benefits outweighs a single security researcher because the programs help uncover more vulnerabilities than what a single researcher is likely to find, the authors wrote. For bugs affecting stable releases of Chrome, Google paid 371 bounties. In contrast, the most prolific internal security researcher at Google found 263 vulnerabilities.
While Mozilla has a fixed payout of $3,000 for confirmed vulnerabilities, Google offers a tiered structure, with more serious issues earning bigger bounties. This means the large rewards, such as $10,000, attract researchers to participate, but most rewards are for smaller figures, such as $500 or $1,000, keeping total program costs fairly low. The median payout for a Chrome bug is $1,156.90, the study found.
“Finding vulnerabilities for VRPs typically does not yield a salary comparable to a full-time job,” the authors wrote in the abstract.
That’s not to say vendors won’t hire “unusually successful” independent security researchers. Google hired at least three researchers who regularly participated in the bug bounty program, and Mozilla has hired at least three, as well. The researcher’s past experience may be an indicator of how many vulnerabilities he or she may find, and the vendor may prefer to pay a fixed salary with benefits than to stick with bounties.
However, while this may be the case for Web browsers, the authors warned “the cost/benefit trade-off” may vary for other types of software. If the cost of the security incident is less for a vendor, then the bug program will not be as useful, they said.
“The higher-profile the software project is (among developers and security researchers), the more effective a VRP will be,” the authors concluded.
Related Podcast: The Story Behind Microsoft’s Bug Bounty Program
Related Podcast: Bluebox’s Adam Ely Discusses Android Master Key Vulnerability
