Security Experts:

Silver Bullets only Work in the Movies, Not Security

Have you ever seen those crazy late night commercials where a salesperson is trying to sell you things like the Snuggie or the knife that can cut anything? You’ll be pleased to learn that this indestructible blade is the last one you’ll ever need. Do you need to chop some onions? Do you have a stray brick lying about that you’ve been meaning to saw through? Look no further, here is your one solution, here is your silver bullet.

Silver BulletUnfortunately, silver bullets only work in the movies.

Marketing Crap is Everywhere

Now, let’s assume for a second that something similar goes on in the security industry. Let’s consider, for just a moment, that there may exist entire teams of people, perhaps living in caves and working under the cover of darkness, whose only job is to convince you that “the Securify 2000 will solve your insecure software woes”. These people who are trying to persuade you, we’ll call them “marketing”.

Now don’t get me wrong, I don’t think everyone in marketing is bent on misleading you. Quite the opposite, I think it’s a few bad apples that spoil the bunch. Many of the people who I trust the most in the security industry are in marketing. I often find that they rank among the most well-informed and well-connected people. I consider many of them my friends; I hope that’s still the case after this article! But as in any field, there are the good and the bad. Many companies will engage in what we can call “good marketing” – delivering legitimately valuable information, resources, and tools. Examples of this include (Security BSides, OWASP, OpenSAMM, Metasploit, and Microsoft’s SDL). These organizations should be commended. But with that in mind, we must also accept that there are many “bad marketing” organizations that can’t be distinguished from their payday loan cousins.

With bad marketing, the deck is stacked against you. It’s easy to fall prey to their traps; bad marketing teams will spin their webs in various ways. They’ll purport to give away “advice, resources, and research” to lure you into nothing more than a sales pitch. Let’s start by exploring each of these tactics, this way we’ll recognize them so that we can avoid being duped by them.

Like a Timeshare Pitch Man

One particularly annoying marketing tactic can be found at conferences when a “speaker” shows up pretending to be an expert in a particular area. They may actually be an expert, but you find yourself listening to a 30 minute sales pitch, or with a self-aggrandizing speaker, a detailed history of their accomplishments. They pay the conference organizers for this opportunity; like many of you, I despise the bait and switch tactic. There are few things that upset me more than when someone has wasted my time. Those 30 minutes of my life are 30 minutes that I’m not going to get back.

Security Vendor PitchesFortunately, we’re starting to see more and more of a backlash against these types of presentations – this is a good thing. No one wants to feel like they’ve been ambushed by a timeshare pitchman, so more and more conferences are explicitly prohibiting this type of behavior doing things like requesting the slide deck in advance.

On the other hand, sometimes we actually seek out opportunities to be pitched at. Those who have attended events such as the RSA Conference, can see how they tend to be a giant sales and marketing confab. Try to seek out those events that have legitimate, redeeming value. Avoid those companies and punish the conferences that have a history of bait and switch.

Lies, damned lies, and statistics.

As Mark Twain was fond of saying, “There are three kinds of lies: lies, damned lies, and statistics”.

Another common approach taken by companies is to try and persuade you with numbers. Many of us in the security industry are trained as engineers, scientists, or have worked with them long enough to appreciate empirical data. However, closer examination of the numbers will almost always lead you to find that the “statistics” produced by most organizations are nothing more than ad-hoc information gathering exercises lacking any semblance of scientific rigor.

It’s not uncommon to find that these statistics will support whatever product or service someone is trying to get you to buy. Having spent over a decade in jobs that required me to find problems in other people’s networks and applications, I’ve become quite attuned to looking for the little details that make all the difference. Thus my first reaction whenever I read yet another headline touting the “latest statistics about application vulnerabilities” is to try and understand their methodology. More often than not, you’ll find serious flaws.

For example, the Web Application Security Consortium has a project that tries to aggregate all the various vulnerabilities discovered from a number of security providers and product vendors. The fundamental problem with collecting this information is that they’re all using different tools, configured differently, against vastly different environments, and using people who have different levels of experience and training. The methods and techniques being employed are almost certainly wildly different. If one team is trained to focus more on SQL injection issues and another team is trained to focus on design level problems then you’re going to get different results. The use of different tools, some commercial and some proprietary, will generate results that are constrained by the same limitations as the tools being employed.

The publication of these types of statistics may be entertaining but rarely can you draw any actionable value from them. These statistics and the associated methodology would never pass peer review in any respected journal. You might as well make decisions based on charts from GraphJam.com.

So don’t just accept things at face value. Always think twice before making any decisions based on unverifiable information gathered from vendors utilizing a broad spectrum of tools and methodologies. Be especially careful of media outlets that blindly regurgitate these “statistics” into their articles, which only layers another level of abstraction to fundamentally flawed numbers.

An App Sec Menu at a Really Lame Restaurant

A clever approach that some bad marketers use, and can be difficult for inexperienced folks trying to learn about SDL, is giving away free resources. As I’ve mentioned several times in the past, there are legitimately useful models and methodologies (ref: MS SDL, OpenSAMM). However, there are also other resources that range anywhere from thinly veiled marketing whitepapers to a seemingly fully-fledged framework or methodology.

An astute software security architect once commented that BSIMM is “supposed to be a measuring stick that’s built on the aggregate best practices but winds up like an appsec menu at a really lame restaurant”. While they claim a long list of prestigious companies who participated in the survey, they didn’t all admit to adopting BSIMM either. He further commented that it was “like the Cheesecake Factory” in that there was something for everyone, but nothing being done with distinction.

My personal conversations with other software security experts, including one particularly vocal expert in the UK, confirmed this sense that BSIMM’s authenticity has lingering taint because it is provided and controlled by a single vendor. Those turned off by BSIMM usually opt to use OpenSAMM, a competing model.

Yet having worked with a number of these “prestigious companies” on their software security programs for several years, the one thing I can say for certain is that BSIMM (and possibly even OpenSAMM) may be okay for your company, but it’s unlikely that it’ll be great. There’s a good chance that it won’t be the right fit unless you’re one of the organizations that counts themselves among the Fortune 500. The best practices of a Fortune 500 organization are not always the same best practices that should be adopted by your company. So it’s more likely than not that you’re looking at the wrong measuring stick for your company. OpenSAMM can be a good place to start, but you’ll want to make sure that you tailor a model that suits your situation the best.

What about Product Companies?

Now, product companies are notorious for trying to tell you that a great way to start a program is to “lead with the tool”. Static code analysis vendors have historically been notorious for trying to tell prospective customers that their solutions are the one stop shop for solving your application security problems. Static code vendors are not, however, the end of the list.

But “leading with a tool” is a huge mistake, and I’ll explain why. As an industry, we need to get past the marketing and start looking at the SDL problem through a clear lens. Specifically, let’s understand why leading with a tool is for fools. To have a successful and practical program there must be a balance between the people, process, and technology.

“People, process, and technology”, I’m sure you’ve heard that line a million times before. But I’m not just going to say it – I’m going to show it.

Stay tuned for the part 2: the right tools, in the right place, at the right time.

view counter
Vincent Liu, CISSP, is a Managing Partner at Stach & Liu and a recognized expert in application security and secure development lifecycle. Before founding Stach & Liu, Vincent led the Attack & Penetration and Reverse Engineering teams for the Global Security unit at Honeywell International. He was also a consultant with the Ernst & Young Advanced Security Centers and an analyst at the National Security Agency. Vincent has presented at conferences including Black Hat, ToorCon, and Microsoft BlueHat. His publishing highlights include: Hacking Exposed Wireless 1st and 2nd ed, Ajax Security (technical editor), and Hacking Exposed Web Application 3rd ed. Follow him on Twitter @VinnieLiu