Log Management in Different Cloud Environments can be Challenging – But Having Access to Log Data is Essential In Order to Get the Visibility You need to Optimize and Secure your Cloud Environment.
One of the biggest challenges for the enterprise is sorting out how to run its operations from the cloud in order to perform proper log management. On a local network, logging is easy. You point your devices to a local log management solution, and off you go, alerting, reporting and searching through your logs. Chances are, your local network has tons of available bandwidth for not only standard traffic, but log traffic as well.
Event Log Management /Security Information Event Management (SIEM) are considered IT best practice, and for regulated industries it’s a requisite for audit compliance.
A trend that we’ve been seeing in the log management and SIEM space is that SIEM and Log Management vendors are moving toward securing the cloud. It was inevitable.
Some equate log management to simply log aggregation, display, and storage– a simple approach that fails to address these complex challenges. Most SIEM products offer basic event consolidation, simple correlation rules, limited real-time analysis, poor reporting and investigation flexibility, and no identity or infrastructure context. Many still require special collectors, add-on modules, additional systems and significant expertise.
This raises a number of questions. How valuable is the data I’d like to store in the cloud? Is this data absolutely critical to my business? If so, will I use a private cloud to store and work with that data, where I have full control over how to access and manipulate that data? Or, will I push that data up to the cloud where you I have limited access and limited control? Should I segregate aspects of the toolset between private and public cloud? These are all legitimate questions. And if the cloud makes sense in ANY of these situations, it leads to the following question:
“How do we log this stuff?”
Now the SIEM correlation piece is easy. Correlation is nothing more than a bell to ring. The majority of SIEM products offer basic event consolidation, simple correlation rules, limited real-time analysis, poor reporting, zero investigation flexibility, and zero context around the infrastructure. Most require special collectors, add-on modules, additional systems and significant expertise. And don’t get me started on the professional service conversation.
Security logs for correlation make up a paltry 5-10% of log data. Forwarding the security events to the cloud is easy. Filtered and forwarded it’s very low overhead. But the big data, 90-95% of it, is the pain point. This is handled by the front end log management tool– the workhorse.
But what do we do with the rest of the log data? What do we do with that boat load of operational data? What can we do with the forensics data? If we’re to push this to the cloud, how do we get it there? Chances are it’s a ton of data! For hybrid clouds, will the cloud log management solution save me enough money to justify the bandwidth costs, just to get the data up there?
Well first, let’s examine the use cases for log management.
Most common use case is compliance. SOX, PCI-DSS, HIPAA/HiTech, NERC, GLBA, ISO, ITIL. If you’re bound by any one of these mandates, or others, there is generally a requirement to store all log data for one to 10 years. Why? Forensics. Accountability. Responsibility.
Second? Security. The security team wants not only alerting (correlated, targeted or behavioral) to tell them when something happens, but the forensics data to tell them the “who, what, when and where”. The alert will just announce that something happened. It’s an incomplete conversation if there is no context around that alert.
Third? Operational. Log data is the bees knees for not only notification of an operational alert (blown hard drive, overheating server, firewall policy changes, downed devices, restarting appliances, etc.), but is also fantastic for high-level reports to view anomalies. So, reporting would be the 50,000 foot view of the forest vs. searching raw log data which is extremely granular, and perfect for root cause analysis.
Now back to the cloud.
Whether you’re in the cloud now with your business tools and data, or you’re looking to move there, establishing how you’ll do the log management aspect is our topic of discussion.
Private Cloud:
Logging in a private cloud is business as usual. Where the enterprise controls the physical and virtual environments, log management and correlation engines can easily offer visibility into both virtual environments, which live in their private clouds. This is the easiest and most expensive route. Reliability, responsibility and accountability are that of the enterprise. It’s your cloud.
Public Cloud:
Logging in a public cloud, however, is much more challenging. Visibility is severely reduced when system access and system/application controls are limited. Although cloud-based applications can boost productivity and availability of data, they can’t offer the same activity level that more traditional data-centers and public clouds can offer.
Regardless of whether the IaaS or PaaS environments are segregated by some sort of organizational multi-tenant cloud solution, there remain complications in keeping track of all the activity that occurs at different virtualized layers. Regardless of physical or virtual, identity and access management are still important ingredients in the log management stew, even if the data and the applications exist outside of traditional network boundaries. So essentially by pushing log management and correlation to the cloud, the current offerings from Log Management and SIEM vendors offer a loss of visibility and control. This is compounded because of shared infrastructures across multiple enterprises in the dynamic ebb and flow of resource use.
Hybrid Cloud:
Hybrid clouds can give you the best of both worlds. The local, private cloud is often where the bulk of log data is created and managed. There, operational use is maximized as is the forensics side of the house. This would be the steak and potato, whereas, the forwarding of that 5% of security events for correlation, is the dessert. It’s a tiny use case, but when it works, it can be sweet. And there you can offer better evidence of regulatory compliance and government mandates.
In conclusion:
| Read Dimitri’s Other Featured Cloud Security Columns Here |
Regardless of whether you go hybrid, public or private with the cloud, it is critical to acquire the key log data that offers a clear view into the operational and security events that will benefit the enterprise. Many cloud providers are offering log management in the cloud. Use it. It’s not only IT best practices, but it can prevent operation and security risks from being diminished. And if your cloud provider doesn’t offer a log management service, push for one. You need transparency. If they can’t offer you what you need, perhaps this isn’t the right cloud vendor for you. Lastly, responsibility of a data breach still falls on the shoulders of the customer. That means it’s still your problem. Liability is still a risk. Mitigate that with log management and correlation in order to get the visibility you need to optimize and secure your environment.
Read More in SecurityWeek’s Cloud and Virtualization Security Section
