Security Experts:

Monitoring Your Unknown Network Traffic

The recent New York Times hack was yet another high-profile attack that demonstrated the evolution towards multi-vector, sophisticated attacks. In this case, the mission of the perpetrators was very specific -- retrieving editorial information and data related to a particular story -- but it could easily have been nastier. With the extent of access the attackers had, the personal information of millions of New York Times subscribers could have been compromised.

The attack clearly demonstrated that a security strategy centered primarily on signature-based endpoint security isn’t enough to prevent against sophisticated attacks that use a cocktail of tactics, including advanced malware. We’ve seen this before. There is clearly a new cyber war being staged by a very different set of actors (nation states, political groups and criminal organizations) and much has been written about how to tackle these new sets of challenges.

Network Security Monitoring

What was interesting about this attack was that it emphasized the need for a rapid monitoring and response system. Hand-in-hand with the deployment of a robust security architecture is the need for a monitoring and response process that allows enterprises to continuously monitor and process security data efficiently and proactively act upon this data if something suspicious is found. But, delivering the proper monitoring and response infrastructure starts with feeding it the right information from your security appliances. Selecting the right enforcement model also makes the process of finding unknown traffic and potential malware more manageable.

Garbage In, Garbage Out

The first step is of course actually having useful security data to observe. The analytics of data is only as useful as the data itself, so logging capabilities must be enabled so there is enough data to capture network attacks or anomalies. Routers, switches and network security appliances generate logging and netflow data that can provide information about anomalous behavior within the network.

The challenge with a monitoring and response system is twofold. First, it must process large and complex security data sets in real time. Second, it needs to ensure it is processing useful, intelligent data. Recall the growth of the managed security services business just to handle the volume of IPS alerts generated by enterprises that had no idea how to react to them. Having too much data or data that is not easily actionable just brings operational headaches. Useful data includes information about applications, users and content that can shed light about traffic and user behavior.

A new breed of SIEM and big data vendors like Splunk understand this. They have the intelligence to process richer, premium feeds from next-generation firewalls that include information on applications, users and content for more flexible, analytic tools. Next-generation security appliances now also provide integrated reporting and logging tools that make the security administrator’s job a lot easier.

Looking for the Unknown Unknowns

With the richer data feed now driving more interesting and relevant dashboards and reports, how do you leverage this information to find the malware that may be in the network? What exactly are you looking for? Enter the Rumsfeldian theory applied to network security. In February 2002, Former Defense Secretary Donald Rumsfeld said this:

“There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say, we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know.”

Of course, his quote was referring to Iraqi weapons of mass destruction, but these categories apply to network security as well. In an ideal enterprise or data center network, you should know all your traffic, as any unknown traffic could include malware or a possible breach. But in order to easily categorize the unknown, you must start off with a positive enforcement model.

Positive enforcement means that you selectively allow what is required for day-to-day business operations and block everything else as opposed to a negative enforcement approach where you would selectively block everything that is not allowed. A negative enforcement approach requires you to keep track of any new applications and constantly adapt your policy to block them. This would be a never-ending, non-trivial task. The adoption of a positive enforcement model is therefore fundamental to identifying known applications so that unknown traffic becomes significant.

Here is how the categories apply to network security:

Known knowns - If you apply a positive security enforcement model, then you are allowing approved traffic and blocking everything else. Next-generation firewalls do this by analyzing all traffic and decoding known applications and protocols using a combination of signatures, decoders and heuristics. The “known knowns” therefore are the applications that you are allowing in the network, safely enabled for specific users and network segments. The “known knowns” also means blocking traffic you know is bad. This includes controlling traffic sources and destinations based on risks such as blocking known bad URLs, blocking DNS to known bad hosts and domains, known exploits, malware and command and control traffic.

Network SecurityKnown unknowns – The known unknowns include unknown traffic and unknown files. Any traffic that is not categorized via any of the next-generation firewall application identification technologies is categorized as unknown TCP or unknown UDP. This may include custom applications, enterprise applications that have not yet been classified or malware. Within the enterprise network, there may also be unknown files that are downloaded by users that could be infected by malware. The strategy with the “known unknowns” is to eliminate or inspect. For example, by creating custom identifiers for internal applications, a certain amount of unknown traffic can be eliminated. All unknown files should be inspected to ensure that they do not have advanced malware, accomplished via advanced malware inspection technologies such as cloud-based processing of unknown files in a virtual sandbox to look for malware behaviors.

Unknown unknowns – Assuming you’ve completed the first two steps above, you’re left with the unknown unknowns. This is where anomalous and malicious behavior can be observed. For example, look for concentrations of unknown traffic in one user or device, unknown traffic with a lot of bursty sessions (relative to bytes) across a lot of different ports or non-standard ports, or unknown traffic coming from certain countries. Encrypted unknown traffic that cannot be inspected can reasonably be blocked at this point. Further investigation can be launched on the rest of the “unknown unknowns” sessions such as drilling into the user or machine generating the traffic or whether the unknown application is transferring files.

In summary, just as building the right security architecture for your network requires the right security appliances, critical foundational elements like the right data feed and the right enforcement model are the building blocks to creating a robust monitoring and response system.

Danelle Au is head of product marketing at Adallom, a SaaS security company. Danelle has more than 15 years of experience bringing new and innovative security technologies to market, and is a frequent speaker at conferences. Prior to Adallom, Danelle was responsible for solutions marketing at Palo Alto Networks, driving growth in critical IT initiatives like virtualization, network segmentation and mobility. She was also co-­founder of a high-­speed networking chipset startup. She is co-­author of an IP Communications Book, "Cisco IP Communications Express: Operation, Implementation and Design Guide for the Small and Branch Office” and holds 2 U.S. Patents.