Evaluating Security Tools – A Proof of Concept Can be Costly, But in the Long-term Could Save You Money and Your Job…
Organizations spend millions of dollars each year to maintain their IT environment and implement sophisticated computer defense systems. However, when selecting a new solution many businesses rely on product demos in their technical evaluation process. Given the increasing importance of using big data in security, are demos still sufficient? Or should proofs of concept (POCs) be conducted to validate the scalability and flexibility of the tool being considered? While the initial investment in a POC can be costly, the end results might not only justify the additional expenses, but in the long-term save you money (and your job).
Selecting the right security and risk management tools to defend against threats is a requirement to strengthening your company’s risk posture. At the end of the day, you might be spending thousands, if not hundreds of thousands of dollars for a security tool, its integration, and ongoing maintenance. Unfortunately, many organizations still rely on making product assessments based on vendor marketing materials, request for proposal responses, presentations, canned demonstrations, and friendly customer reference checks. But does it make sense to make major purchase decisions based on these decidedly subjective sources?
The industry has seen multiple examples of failed security and risk management technology implementations that subsequently led to the release of the staff members who were involved in the selection process. A proof of concept pilot project is a more reliable approach since it provides a controlled environment for evaluating the capabilities of shortlisted tools for a particular use case. As a result, POCs also enable organizations to mitigate the risks associated with deploying a sophisticated security or risk management platform.
We’re not talking about anti-virus software or firewalls, but rather systems that aggregate critical intelligence about risk and compliance postures with current, new, and emerging threat information to calculate impacts on business operations and prioritize remediation actions.
However, POCs come at a cost, since internal resources need to be allocated to test the product under simulated real-world conditions that include the set-up of an environment that matches your specific requirements and preliminary end user testing. Nevertheless, a POC can ultimately be a huge time saver.
Primarily, since the test settings can be repurposed when a tool is selected and deployed, and involving users early in the implementation process can assure better and faster adoption. Furthermore, on-demand cloud-based resources enable the provisioning of the necessary environment at far lower cost than in the past, which removes a major financial hurdle associated with POCs.
If you’re still not convinced or need further ammunition to convince your organization that it should invest the time, energy, and money in a POC, here are five reasons why it is a good idea:
#1 (Proof of) Capability – Every organization is different and has unique requirements. Therefore it is essential to validate that a tool supports all, and not just a sub-set of use cases. In this context it is also important to evaluate the tool’s ability to extend use cases over time and how easily this can be achieved (e.g., same underlying database and enablement of new use-case applications via software feature key). Further consideration should be given to the tool’s flexibility for adjusting to changes in the environment, cross-role usability and manageability, built-in reporting and analytics capabilities, etc.
#2 (Proof of) Integration – Most organizations have already invested in best-of-breed technology and therefore any advanced security and risk management application should be validated based on its interoperability with the existing product architecture. It’s also important to determine if the product is modern and secure; integrates across multiple use cases, data models, third-party data sources, and identities; and provides stable APIs, etc.
#3 (Proof of) Scalability – According to Gartner (see Information Security Is Becoming a Big Data Analytics Problem, written by Neil MacDonald) “the amount of data analyzed by enterprise information security organizations will double every year through 2016. By 2016, 40% of enterprises will actively analyze at least 10 terabytes of data for information security intelligence, up from less than 3% in 2011.” Taking these data points into account, scalability is one of the most essential factors when running a POC. So while canned product demonstrations are based on limited data sets, evaluating the product under full load and performance tests will reveal if it can handle the volume and velocity of data objects (e.g., assets, vulnerabilities, threats, incidents, tickets, etc.), concurrent users, and multi-geographic clustering often required by global organizations.
#4 (Proof of) Self-Sufficiency – New customers are always the vendor’s best customer. However, once the honeymoon is over and the product is up and running, organizations will want to limit their reliance on the vendor’s professional services team to maintain and advance the product. For instance, not being able to modify workflows, content mapping, reporting etc. creates a dependency that can be expensive and negatively impact total cost of ownership. The best way to determine if the user interface is intuitive and flexible enough to handle future changes is to allow end users to take the product for a test drive.
#5 (Proof of) Time-to-Value – Many advanced security and risk management systems have gained a reputation for taking a long time to get up and running and produce outputs to justify the investment. A POC can provide insight into expected time-to-value, based on purpose-built content and connectors, configuration wizards, and customization tools that can accelerate deployments to deliver results within weeks or months, and not years.
Using these five evaluation criteria will enable an organization to conduct a POC that will yield reliable evidence on whether products being evaluated will or will not meet their needs, integrate with their existing infrastructure, and deliver expected results in an acceptable time frame.
