Security Experts:

How to Tap the Hardware Random Number Generator in Your Load Balancer

“Everything happens for a reason.” — Girlfriends 1, 2, 4, and 5.

It is part of human nature to see patterns where none exist. To see meaning where there isn’t any. There are words to describe this behavioral characteristic: apophenia, patternicity, and agenticity are three neutral examples. Go a little further and you encounter these familiar pejoratives: paranormal, delusional, and paranoid. Misperceiving meaning is not limited to a certain gender, age or subgroup; it exists across all cultures. People will fabricate meaning where it doesn’t exist.

Computers have the opposite problem—they periodically need to find entropy (randomness) but often they cannot even perceive it. Their very windows of perception are influenced by algorithmic agenticity:  time slices of a hypervisor, process scheduler, or interrupt handler. Cryptography needs entropy the most because cryptographic keys are supposed to be generated from random data, both for static keys such as SSL or SSH keys, and also for ephemeral session keys. Whereas humans generate keys via supposedly random-looking data called “passwords,” computers are supposed to use blocks of completely random data—except they can’t find any. So they fabricate entropy where it doesn’t exist.

Consider a virtual machine booting up for the first time—it needs to generate an SSH host key to enable other devices to connect to it. The ssh-keygen utility queries the operating system’s random number generator (RNG) for the entropy, but there isn’t any. The virtual machine has just come up and hasn’t seen any external data from which it could filter some entropy. So it makes up the random data by seeding a deterministic algorithm (such as RC4 or MD5) and drawing off the output like ground beef coming out of a hand-cranked meat grinder.

Hardware Random Number Generator The problem is, making up random data is a bad, bad idea. If the initial seed values can be guessed (often they are easy things such as a process ID or timestamp), then the rest of the stream can be determined.

“Anyone who considers arithmetical methods of producing random digits is, of course, in a state of sin.” — John Von Neumann (1951)

This fabrication of entropy goes on all the time. In 2012, Apple was worried about the amount of entropy in its iPhones on boot-up, so it switched to a different deterministic RNG algorithm during boot-up. This “early RNG” had problems, though, resulting in even worse entropy collection. The most popular computer of all time, the iPhone, has not escaped the problem of the lack of entropy.

Server-side software may be even more susceptible to this problem as more servers become fully virtualized and therefore even farther removed from their own hardware. Nadia Heninger and her colleagues wrote a disturbing paper, Mining your Ps and Qs: Detection of Widespread Weak Keys in Network Devices, in 2009. They found that between 1 and 2 percent of all SSL keys on the Internet are factorable (and therefore recoverable), largely due to bad entropy. In 2008, a screw-up in the RNG of one of the Debian distributions resulted in only 15 bits of entropy in their keys. This key generation weakness went unnoticed for years.

The problem is going to keep happening. According to security luminary Dan Kaminsky, there has been no progress in RNG technology for 25 years. Is there anything that an administrator can do about this, besides frantically patching his software and regenerating his keys the next time an open-source RNG turns out to be broken?

Actually, there is. Few people realize that a source of real randomness already exists inside nearly every data center. It’s in the load balancer. Years ago, market forces encouraged architectures in which load balancers, in addition to managing traffic load, also terminated the SSL protocol on behalf of the application servers. To compensate for this extra computational load, load balancers started including special chips that performed the cryptographic offload necessary to handle massive amounts of SSL. These chips require lots of randomness, so they produce it from quantum mechanics subsystems. The chipsets are typically Cavium’s Octeon chipset or Intel’s Ivy Bridge. Intel engineers had to build a new type of RNG to meet low-power requirements and, as a result, came up with a nifty all-digital quantum RNG.

Hardware-based load balancers exist in nearly every data center built after 2000. And many of the hardware RNGs within the load balancers can be tapped to provide real, hardware-based entropy for dozens, hundreds, or even thousands of machines within that data center.

A load balancer can be quickly configured to export a virtual server on the internal network that simply returns, say, one kilobyte of random data to any device that asks for it (a virtualized LAMP server, for example). The LAMP server can then dump that entropy into its own entropy pool in a boot-up script periodically in a cron job. Administrators will be happy to know that even if the process fails (due to network disruption or malfunctioning equipment), the system isn’t any more broken than it would be without it, because the operating system stirs the data into its entropy pool.

Tapping an already available hardware RNG at the load balancer can solve the problem for devices in the data center, but what about for devices in outside the perimeter? Or remote devices in remote offices?

There are actually several systems that already deliver streams of random data over the Internet. The most prominent is random.org, which purports to get its entropy from atmospheric scatter. The NSA, believe it or not, will also provide you with entropy as a service. I told this to a customer and he replied, “Isn’t that like having Sauron deliver your milk?” Joking aside, getting random data from external sources is not a trivial or safe exercise. That data could be non-random, or it could be observed as it’s being delivered to you, so use external data at your own risk.

But getting back to the solution of using the hardware RNG on the load balancer: a security architect can turn to this technique to provide real entropy for the servers under network function virtualization. It is free entropy, requiring only some scripts pushed out in the next DevOps cycle. When the next ridiculous RNG vulnerability comes around, the architect can sleep a little better knowing the data center’s entropy pools are primed via quantum mechanics. As he sleeps soundly, part of his mind will build those little stories (dreams), cooking up meaning out of somnolence.

At least we can improve the lack of randomness for our computers. On the human side, there’s still no quick fix for the lack of meaning, and we’ll have to keep making that up as we go.

view counter
David Holmes is an evangelist for F5 Networks' security solutions, with an emphasis on distributed denial of service attacks, cryptography and firewall technology. He has spoken at conferences such as RSA, InfoSec and Gartner Data Center. Holmes has authored white papers on security topics from the modern DDoS threat spectrum to new paradigms of firewall management. Since joining F5 in 2001, Holmes has helped design system and core security features of F5's Traffic Management Operating System (TMOS). Prior to joining F5, Holmes served as Vice President of Engineering at Dvorak Development. With more than 20 years of experience in security and product engineering, Holmes has contributed to security-related open source software projects such as OpenSSL. Follow David Holmes on twitter @Dholmesf5.