Strategies and Considerations for Securing Private Cloud Environments
On the back end of private cloud environments you’ll find multiple flavors of virtual software loaded directly onto hardware. This virtual software is essentially the host operating system. VM Host is the base hypervisor and hardware. Think of it as the house. The guest operating systems (Guest OSs) are the virtual machines living in the house.
As the basis for all public and private clouds, virtual infrastructure is how it’s done. So this conversation we’re about to have is related to the back end of the private cloud. If you’re building one, it is important for you and your organization to understand how to maximize the benefits and mitigate the risks of a private cloud infrastructure. There are several key things to keep in mind when trying to secure the virtual environment before even loading guest operating systems.
Most virtual solutions are transparent, by design, to the guest operating systems. The same way machines are secured in physical environments, they are secured in a virtual environment. This includes segregating networks, defining domain security policies and installing antivirus.
But unlike their physical hardware cousins, Virtual Machine infrastructure security seems to be lagging behind, and although this virtualization is consuming datacenters worldwide, many organizations fail to recognize that security basics are still security basics.
According to Gartner, 16% of server workloads were running on virtual machines by the end of 2009. Gartner expects this to grow 50% (to 58 million) by 2012. Unfortunately, Gartner also predicts that 60% of these virtual machines will be less secure than their physical hardware predecessors.
Why are Virtual Machines insecure?
The reason that VM servers are less secure than their traditional hardware counterparts are as follows:
Security isn’t considered at the beginning of the project, which is often the case. In many situations a public cloud project is begun, and from there each project becomes a knee jerk reaction.
If the VM host OS layer is compromised, all guest OSs can be compromised. This is called Hyperjacking. More on that later.
Although most public cloud vendors maintain adequate controls for admin access to the virtual machine monitor, many private clouds do not.
Segregate and separate. VM hosts create flat networks. You’ll need to change that. In a non-virtual world, traditional data-centers had segregation and network traffic could be inspected, filtered and monitored by a number of security products. In a virtual world, these are a rare commodity. The local communication between virtual servers is largely untouched and unmonitored. If the traffic runs through a virtual switch it’s practically invisible because it never hits the wire. It’s just traffic between virtual hosts on virtual links. So virtual traffic between virtual machines needs to be monitored.
Separation of duties is something that we in security often push. Unfortunately, in a virtual server environment, the back-end of a private cloud environment, you’ll often find that the server team and the operations team are the same people who do both provisioning of machines and managing virtual switches. So that means that there’s rarely any integration between the tools and security controls to be implemented for the network and security groups. And what THAT means, is that without visibility into configuration and policy changes, topology specifications and audits, the network and security team has zero view into what’s taking place at the access layer.
In security circles, we also talk about the “principle of least privilege.” This says that you should not give anyone more security than the minimum security they need to do their job. Defining roles that can be used to give different levels of security will make life much easier.
How do you secure a traditional server? First, lock down the server OS (usually Windows or Linux). Now, as you go virtual, instead of just securing the server OS, you also have to lock down VMKernel and the VM layer (the host OS), as well as the console. The same thing you’d do with your weekly Microsoft patch plan is what you should do with your VM Infrastructure. Although extremely secure, stay up to date on patches. There are security updates in there, not just bug fixes.
What you really want to avoid is Hyperjacking, which involves compromising the hypervisor. This is the lowest level of the OS stack, and the hypervisor has more privileges than any other account. At this level it’s impossible for any OS running on the hypervisor to even detect that a hack has taken place. So the hacker can control any guest VM running on the host.
When you go virtual, you add that other layer to the mix, the hypervisor. So again, the hypervisor needs to be secured at all costs. It’s mission critical because an attack on the hypervisor can lead to the compromise of all hosted workloads, and successful attacks on virtual workloads can lead to a compromised hypervisor. Another concern would be a VM Escape, which is an exploit that is run on a compromised guest OS to attack and take over the underlying hypervisor, which can then result in a hyperjacking.
Moving up the stack are those OS patches. Although it’s super easy to spin up another guest operating system, admins sometimes forget that the virtualization software is there. So make sure that just as fast as virtual machines are spun up, patch distribution software should also be installed, and antivirus, service packs and security policy changes should be made to all of those virtual guest operating systems.
The biggest security concern, however, is the insecurity of the underlying virtual guest OS. The VM software you use will separate the guests from both each other as well as the VM Host, so if one of the guest VMs does get compromised it’s unlikely it could affect the host, with the exception of using more memory/processor/network resources.
Be aware that the easier move for a hacker to steal data would be VM Hopping. This is a situation where an attacker to compromise a virtual server and use that as a staging ground to attack other servers on the same physical hardware.
The last threat to VMs themselves would be VM Theft. This is the virtual equivalent of stealing a physical server. Take the whole box and run off with it. Then fire it up later and steal the data. Same concept, however, in this situation it is theft of the virtual machine file electronically, and then attack it later.
How can you make the private cloud more secure?
Start with the base layer. The lower stack. The hardware and traffic. Force all traffic between hosts to be inspected by an IDS (intrusion detection system). Each VM Host should also have a different ingress/egress VLAN pair. Then the IPS should be set with VLAN translation to configure each ingress VLAN and egress VLANs. The goal is to define all VM-to-VM traffic being sent across the wire where it can be inspected, monitored and potentially filtered. Of course, as the private cloud grows, it can become complicated and costly between data-centers and DR sites.
Another option is to define an IPS and firewall on each VM Host, and policies be configured to inspect the traffic. This makes sure all intra-virtual communication is inspected. Of course, there are some performance hits you’ll take running all those additional VM IPS and Firewalls, plus the monitoring for all traffic, however, in the end security should be paramount.
Next up is a ‘love connection’ between the above two. It’s a mix of both. Route traffic to an actual IPS where it’s filtered, can be monitored, etc. Then send the traffic off to a destination VM.
Another security option as you move up the stack after traffic would be securing your hypervisor. Keep your hypervisor console patched. Just like any OS, VM Servers will have security patches that need to be deployed. The majority of these patches are related to the Linux-based OS inside most service consoles.
Additionally, ensure that virtual machines are fully updated and patched and all provisioning is done with security tools etc before they are turned on in a production environment. Although VMs are much easier to move around and faster to deploy than physical machines, there is a greater risk that a VM that is not fully updated or patched might be deployed. To manage this risk effectively, use the same methods and procedures to update VMs as you use to update physical servers.
Another tip would be to use a dedicated Network Interface Card (NIC) for management of the virtualization server. By default, NIC0 is for the parent partition. Use this for management of the Host machine. Don’t expose it to untrusted network traffic and please don’t allow any VMs to use this NIC card. Use one or more different dedicated NICs for VM networking.
Lastly, before installing the private cloud services, (the reason you build this infrastructure) I’d recommend using disk encryption to protect the VMs. If one is stolen, it’s worthless to the thief, and the data at rest is also safe.
Summary
When building a private cloud, it’s important to remember that going VM isn’t as easy as moving servers to a host machine. Security is a bit different, but security practices are still the same. Recognize the additional components. Recognize that going VM often ends up in a flat network, and you’ll need to change that for security’s sake. Lock down your console and hypervisor. And remember the same lock down procedure you used in traditional servers is also critical, if not paramount. Follow these tips and you’ll be ready to host those apps to your user community in no time. Securely.
