PCAP Enables Defenders to See and Capture Exactly What Has Happened Across a Network, But Comes With Challenges
PCAP, or full packet data capture for analysis, does what it says – it captures the entirety of every packet that comprises the network traffic (both metadata and content). If something happens on the network, PCAP knows about it. Whether it is malware moving data around, or staff arranging a private party, it can be captured and then analyzed.
PCAP provides what CISOs seek but rarely achieve – total visibility into the network.
The security potential for this type of traffic monitoring is clear, and probably explains the motivation for a number of U.S. federal agencies investigating their options. Toward the end of 2020, in the first flush of the SolarWinds debacle, the DHS, the Department of State, Aberdeen Proving Grounds, the U.S. Marine Corps (USMC), and the Missile Defense Agency (MDA) all issued requests for proposals (RFPs) and requests for information (RFIs) for PCAP solutions.
The Homeland Security Department’s Enterprise Security Operations Center stated that it considered “Full Packet Capture a cornerstone of the cyber security visibility stack enabling analysts to perform investigation analysis while also satisfying DHS security requirements.”
This sudden rush to PCAP poses a couple of obvious questions. If PCAP is such a powerful security tool, why hasn’t it already been widely adopted among the agencies? And is this movement within the federal agencies likely to migrate to the general business sector?
All-seeing benefits
“The packets never lie,” says Vectra’s EMEA director, Matt Walmsley. “Packet capture has long been the gold standard of primary evidence sources for network security forensics. It’s a bit-for-bit direct copy of the exact traffic that was transmitted across the monitored network. It’s not an interpretation, it’s not a summary description – it’s the raw truth.”
PCAP collects everything. It is not designed to provide real time – or any – analysis. Analysis is left to add-ons or other security tools. The value of PCAP is the ability to see and capture in detail exactly what has happened.
For this reason, many analysts believe that PCAP is best suited to (recent) historical analysis. “Full packet capture is purely for historical analysis,” says Oliver Tavakoli, CTO at Vectra. “It usually depends on some other detection capability to point the finger at packets of interest.”
Not everyone entirely agrees with this. “It is absolutely not only for historical analysis, “Joseph Carson, chief security scientist and advisory CISO at Thycotic, told SecurityWeek. “Full PCAP can also be used in real-time; however, this requires very well-crafted algorithms to help SOC Analysts determine which packets should be investigated and what can be fully automated.”
The addition of artificial intelligence to PCAP could well change the use and value of PCAP in future years.
Carson summarizes the value of PCAP. “Recently, I analyzed a severe ransomware incident. With the log data remaining it was only possible to get a partial view on how the attackers worked – but if I had full PCAP data then it would be possible to create a much more detailed attack path.”
Axellio SVP Stefan Pracht explains further: “Only packets,” he said, “can offer the insight into the timing and sequence of events, where the attack came from and which enterprise resources were involved with the malicious activity, what data was accessed in the attack or even exfiltrated and how the attack spread laterally through the network. Being able to play out the traffic of the actual attack also provides important insight into whether this happened before but went undetected and to determine whether any implemented mitigation is actually working.”
SolarWinds is a case in point. “We’d really like to know how whoever got onto the networks actually did what they did – and I mean exactly what they did,” says Sammy Migues, principal scientist at Synopsys. “Normal logs will capture that HostA talked to HostB and things like that; but what did they say? Enquiring minds want to know! What exactly did they change? How? How did they remain undetected? So many questions that might be answered with full packet captures.”
Brandon Pearce, AVP of federal and intelligence products at AT&T Cybersecurity (and former CISO at the National Geospatial-Intelligence Agency), goes into more detail. “Because PCAP allows direct examination at the packet level, government agencies can review it on their network forensically to find any anomalous behavior at this level. Compared to logging data, there is more of a chance to find previously unknown behavior like covert communication/exfiltration channels, command and control signals embedded in otherwise expected traffic, and so on.”
Finding previously unknown activity at the packet level can reveal not only that something has happened, but how it happened. This gives agencies the data to form an actionable plan to counter an unwelcome presence in their network, and the hard evidence to show what happened. Logging at higher levels of the OSI model doesn’t give that same forensic level of information.
The cost problem
Apart from being the gold standard for network forensics, Walmsley adds, “It’s also highly voluminous, expensive to store, and extremely arduous to search and analyze manually at any meaningful scope.”
Cost is a problem. Capturing everything that crosses the entirety of a network requires a huge amount of storage. While storage costs are coming down, network traffic is going up, and it remains inhibitively expensive to store more than a few days of PCAP data. “Performing full packet capture on a one gigabit per second link (note that fast links are now running at 100 gigabits per second or more),” explains Tavakoli, “can require upward of 10 terabytes a day in storage. This practically means that it becomes a race against time: if an organization can afford to store 10 days of full packet capture, it is effectively betting that it will find an attack within 10 days of its initiation.”
If a company does not have the resources to effectively analyze that amount of data quickly, it is hard to justify the expense of PCAP. “PCAP has a place,” says Richard Bejlitch, principal security strategist at Corelight, “but one must balance trade-offs of storage, ability to query, and other factors. I would encourage agencies looking into upgrading their network security monitoring infrastructure to first see if transaction logs could solve their problems, with targeted or ‘smart’ PCAP for edge cases and additional inquiry. A ‘full PCAP first’ approach can be costly and slow compared to the alternatives.”
Apart from the choice between full or just partial PCAP collection, there is a potential problem in staffing. “Storage limitations may necessitate choosing from which network segments to capture traffic,” says Joseph Salazar, technical deception engineer/ technical marketing engineer at Attivo. “Much like storage constraints result in lost evidence, lack of coverage leads to missing PCAPs and blind spots during analysis. Finally, analyzing PCAPs requires experience and training to extract the relevant data needed for an investigation, whether it is a binary payload or an exfiltrated data file. With the training and personnel challenges facing the cybersecurity industry, this is not a need that organizations can quickly fill.”
Balancing privacy rights issues
Privacy is another issue that needs to be considered. “PCAP files,” explains Matt Walmsley, EMEA director at Vectra, “can be used to reconstruct clear-text session and data payloads. This can open up the time-honored question of user privacy versus the defending organization’s need to protect. Where traffic is encrypted and the encryption keys are not available, then there are still insights that can be gained if one knows where and how to look. In fact, all network traffic, encrypted or otherwise, contains subtle high-fidelity signals within the vast quantities of legitimate communications that can betray the presence of a threat actor active inside the organization as they develop their attack.”
Carson adds, “Full PCAP on encrypted data does not allow the analyst to see the content. However, the metadata from the headers can be helpful when analyzing incidents or performing threat hunting activities.”
Nevertheless, the use of encryption on the network changes the calculations.
Kyle Huddart, lead principal analyst at Talion, explains: “The biggest crux (as with all monitoring solutions) is that for this to be effective for web traffic, you need to have SSL inspection available and access to the certificates to decrypt the information. This is where privacy kicks in. Full packet capture of users’ HTTPS traffic could reveal sensitive information should they be using personal websites within a corporate network (banking/medical etc).” This can be negated by applying the decryption certificate to only certain categories of website (for example, don’t decrypt banking or medical sites); but it is worth noting this method is reliant upon the web categorization of a proxy and adds complexity and therefore cost to the solution.
“In most modern communications,” says Vectra’s Tavakoli, “the packets would be encrypted. On such connections all that would be visible is that communication between two systems occurred and how much data was exchanged.” It must be asked, then, whether the cost of PCAP can be justified in an environment that effectively doesn’t allow full PCAP.
Here there are two options. The first is to abandon full PCAP in favor of one of the increasingly powerful AI-based network detection and response systems. The second is to break your own encryption. “Most organizations insisting on packet capture,” continued Tavakoli, “also look into schemes where the encryption can be broken via an active ‘man-in-the-middle’ scheme or by placing an agent on endpoints to access a copy of the encryption keys.” This, however, returns the privacy issues to front and center.
Some PCAP-based solutions attempt to solve the problem with additional technology. One example, explains Axellio’s Pracht, “requires packet capture solutions to carefully manage both data storage and access. Information can be identified and ‘masked’, meaning the information is overwritten. This can be done either permanently before writing it to disk or masked when displaying to the user. With some of this information being essential for analysis such as address information, access to this information needs to be carefully managed by limiting access on who can see and review the data, and managing user permissions at different authority levels.”
The collection of employee content is a thorny issue, and its acceptability may come down to the legal jurisdiction in operation. However, provided the collected data is kept safe and secure, and access to it is limited to the security team tasked with keeping the network secure, then it could be argued that the data collection is necessary to maintain the company in business.
The future of PCAP
PCAP is at an inflection point. Will decreasing storage costs and improved and automated AI-based analysis bring it within the budget of more companies? Or will improving AI-based network detection and response (NDR) solutions render full PCAP redundant?
“Full packet processing (but without capturing the packets for posterity) is the province of products which do Network Detection and Response,” explains Tavakoli. “These products provide an opportunity to apply AI to fields (referred to as metadata) extracted from the packets in service of detecting threats in real time.”
The only thing that is certain is that the closer the analysis remains to the totality of packet content, the more accurate the analysis will be. And the less automated that analysis, the longer it will take and the more effort it will require. Realtime NDR, however, will not provide the depth of forensic evidence available to full PCAP.
One final comment. If you collect and store full PCAP files, you had better store them securely. “Having an attacker steal all your PCAPs,” warns Migues, “might actually be worse than having that attacker run rampant in your organization for weeks and months. All the information is right there–emails, documents, browsing histories, proprietary code, and so on without them having to break into every computer to get it. Having your PCAPs used against you in court would harsh anyone’s buzz.”
This may explain the agencies’ late push to PCAP. “I think there could have been a lot of red tape for the US government to use full packet capture,” suggests Talion’s Huddart, “mainly because if they were, for example, sharing classified information across the network – that would then reside in the full packet capture information and could be reconstructed by potentially unwanted eyes.” SolarWinds has changed our thinking.
Related: Encrypted Network Traffic Comes at a Cost
Related: Group Seeks Investigation of Deep Packet Inspection Use by ISPs
Related: FireEye to Acquire Network Forensics Firm nPulse in $70 Million Deal
