Connect with us

Hi, what are you looking for?


Application Security

Secure Development: Using the Right Tools in the Right Place at the Right Time

Are You using the Right Application Assessment Tools in the Right Place at the Right Time?

Are You using the Right Application Assessment Tools in the Right Place at the Right Time?

To develop any program, you always start by performing an assessment of your current situation in order to establish a baseline of where you stand. In developing a secure development lifecycle, this is no different. Many companies begin the baselining process with an assessment of one or more applications to determine the level of risk posed by your applications. Naturally, people first look for tools that can “perform an assessment.”

Types of Tools

Generally, there are two different types of application assessment tools on the market – static analysis tools and dynamic analysis tools. Static analysis tools (e.g. Fortify SCA, Armorize CodeSecure, etc.) are used to scan application source code (or, in some cases, a representation of it) for vulnerabilities. Dynamic analysis tools (e.g. HP WebInspect, IBM AppScan) are used to scan live applications such as a web application or a web service.

For a long time the vendors in each camp battled it out, and to a degree, they still do. Static vendors would claim the best way to find vulnerabilities in an application was by performing code level analysis, while dynamic vendors made the claim that what you really needed was something that worked against a live system because that’s what attackers were doing.

Then there were consultants who claimed tools were stupid and the best work was always performed by consultants. I remember one particularly horrific presentation in which the consultant was trying to explain to an audience how they once found four times as many cross-site scripting issues as a tool would during an assessment. Based on this single experience, everyone was encouraged to stop using of tools and use humans to assess applications instead.

They were all right, and they were all wrong.

When Humans Succeed!

Advertisement. Scroll to continue reading.

Let’s begin with a simple example. Below is a cleaned up version of a vulnerable password reset webpage that I found during an assessment a couple of years ago. Can you name three things that are wrong with this picture?

Secure Development: Password Reset Code

First, the web application tells the user which questions have been answered correctly. Second, the last two questions are not even questions. The application allowed the user to define their own questions, assuming that the users will know how to define security questions the right way . The answers to the last two questions can be easily Googled. Finally, the first question has a rather limited set of potential answers as there are only a handful of car manufacturers. A question with a larger set of potential answers would be, “What street did you live on in third grade?”

Even if we combined all the best commercial static and dynamic analysis tools, they would never find the problems of the security questions above. That is, until computers learn to read (all hail our robot masters). There are more issues, which will be left as an exercise to the reader.

If static and dynamic analysis tools can’t find the critical design flaws above, then how are they found? Well, the same way that we just found them – using humans. To identify design flaws (and more complex vulnerabilities) it’s necessary to have a human in the loop. In my personal experience, design flaws make up between 30 to 40% of the critical issues that are identified during our assessments .

If your organization has made the mistake of designing a secure development lifecycle around the use of a tool, then you may be missing critical design level flaws. An application architect with proper security training, adhering to secure design best practices, should be in a position to eliminate a poor design such as the one above. By eliminating the poor design before it is implemented, you avoid the time-consuming activity of code refactoring.

But this doesn’t mean that humans are infallible. On the contrary, humans have a long and well-documented history of failure.

When Dynamic Tools Rule.

When do people fail? All the time—as do static analysis tools.

Directory IndexingThere are many scenarios where humans simply cannot keep up with technology. The simple fact is, that for many vulnerabilities, the best way to locate them is using tools. Take for example the identification of directory indexing. Directory indexing is a situation where a web server has been improperly configured during deployment and it allows users to view a listing (or index) of the files within the directory.

When humans find this issue it is usually by chance. It’s possible, however, to find this problem by systematically crawling through the web site and truncating the path in an attempt to find directory indexing. For example, if a page exists at then an attacker might try to identify and access the directory listing by removing (truncating) userguide.htm from the URL and submitting only the directory:

Having to do this for an entire website? Haha. Depending on the website, it might take days, months, or even years for a person to perform these checks manually. Humans fail.

Clearly, the solution is to automate the process across the many tens, hundreds, and sometimes even thousands of potential requests this comprehensive undertaking would require. What would cause an expert to die of boredom could be accomplished by a tool in a matter of minutes.

It’s also important to keep in mind that a static analysis tool would never be able to find this issue because it only searches for vulnerabilities with source code. This particular vulnerability is introduced to an application when it is deployed on an improperly configured web server. So, no matter how many static analysis tools you ran against the application, this would never be found.

There are many issues like directory indexing where humans and static analysis tools fail, and dynamic scanning tools reign supreme.

Static Reigns Supreme!

The type of vulnerabilities where static analysis excels is when looking for vulnerabilities that require examining data flows within an application. For example, trying to identify SQL injection bugs (a data flow issue) in an application that has complex workflows, such as the steps to create a personal profile.

Secure Development Testing

Let’s assume that there’s a critical SQL injection bug, but it can only be found at the very end of a series of profile setup pages and the vulnerable feature only appears if you’re a non-smoking, dog lover, who wants kids. What this means from a technical standpoint is that a series of very specific options have to be selected from an extremely high number of possible permutations to expose the vulnerability through the web interface. It would be infeasible for dynamic scanners to test every possible permutation of options and checkboxes in an attempt to cause the application to expose the vulnerable feature, much less perform the testing required to identify the vulnerability. In any moderate to high complexity workflows, testing each branch cannot be performed by dynamic scanners because there are simply too many permutations.

In order to identify this deeply embedded issue, you must analyze the code using static analysis tools. The static analysis tool would be able to trace through the code, create a map of the data flow branches, and determine which paths would lead to the embedded SQL injection point. Visually, it would look something like the following:

Diagram of embedded SQL injection point

Of course, a human could identify the SQL injection vulnerability by either working through the workflows or by reading through the code, but it would take an inordinate amount of time to do this against any reasonably sized codebase. When reviewing code, the human brain essentially mimics the behavior of the computer as it processes code while simultaneously having to act like an attacker trying to break the code. This is extremely time-consuming, and humans are almost always the costliest resources. The economics of a fully manual code review usually rule it out as an option for the most critical applications.

In this case, the best tool for the job is a static analysis tool.

When to Use Tools and Humans

Let’s take what we’ve learned and summarize it into a nice table:

Table of Dynamic Code Analysis Tools vs Static

Dynamic analysis tools are well-suited for finding issues like directory indexing, but will fail at finding an embedded SQL injection issue and secure design problem. Static analysis tools can help identify issues related to data flow, but fail at finding deployment type issues and design problems. Humans can do it all, but you probably won’t be able to get them to do it quickly or willingly unless impressment comes back into fashion. You want to use the right tool for the problem.

In addition to the above, we also know that there’s a certain time that you want to use these tools. Applying everything at the very end of the software development process probably isn’t the best use of your time and effort.

Secure Development Lifecycle Process

Security question issues are a design level problem. It’s best to apply human analysis to find these types of issues during the design phase of the process. On the flip side, it would be silly to try to run a static or dynamic analysis against the application at this point because the application code isn’t deployed and the site isn’t running.

SQL injection vulnerabilities are introduced when the design is implemented in the application code, and the construction of the SQL query isn’t properly performed. Static analysis tools run during this coding process can eliminate the issues early on before more code is built around or using the vulnerable component. Still, you can’t run the dynamic tool because the application hasn’t been deployed.

Finally, directory indexing issues are introduced when the application is released and deployed onto a misconfigured web server. Trying to identify it before this point would be futile.


So what does all this mean? You need to use the right tools in the right place at the right time.

Written By

Vincent Liu is CEO at Bishop Fox, a security consulting firm. With over two decades of experience in cyber security, Vincent is an expert in security strategy, red teaming, and product security; and at Bishop Fox, he oversees firm strategy and client relationships. He has presented at Microsoft BlueHat and Black Hat; and has co authored seven books including Hacking Exposed Wireless and Hacking Exposed Web Applications. Prior to founding Bishop Fox, he led the Attack & Penetration team for the Global Security unit at Honeywell International. Before that, he was a consultant with the Ernst & Young Advanced Security Centers and an analyst at the National Security Agency.

Click to comment


Daily Briefing Newsletter

Subscribe to the SecurityWeek Email Briefing to stay informed on the latest threats, trends, and technology, along with insightful columns from industry experts.

Learn about active threats targeting common cloud deployments and what security teams can do to mitigate them.


Join us for an in depth exploration of the critical nature of software and vendor supply chain security issues with a focus on understanding how attacks against identity infrastructure come with major cascading effects.


Expert Insights

Related Content

Application Security

Cycode, a startup that provides solutions for protecting software source code, emerged from stealth mode on Tuesday with $4.6 million in seed funding.


Less than a week after announcing that it would suspended service indefinitely due to a conflict with an (at the time) unnamed security researcher...

Data Breaches

OpenAI has confirmed a ChatGPT data breach on the same day a security firm reported seeing the use of a component affected by an...

IoT Security

A group of seven security researchers have discovered numerous vulnerabilities in vehicles from 16 car makers, including bugs that allowed them to control car...


A researcher at IOActive discovered that home security systems from SimpliSafe are plagued by a vulnerability that allows tech savvy burglars to remotely disable...

Risk Management

The supply chain threat is directly linked to attack surface management, but the supply chain must be known and understood before it can be...


Patch Tuesday: Microsoft calls attention to a series of zero-day remote code execution attacks hitting its Office productivity suite.