Network Changes – Digging Out from the Deep Freeze

A Network Freeze Doesn’t Mean There Aren’t Risks On Your Network That Need To Be Addressed…

The holidays are officially upon us (and have been apparently since the minute after Halloween was over). While consumers are racing to get all of their shopping done and businesses spend inordinate amounts of time and money trying to keep their stores busy, many IT operations teams are either preparing or in the midst of putting their network on ice.

Retail organizations earn a big chunk of their annual business during the holidays. With this in mind, it is fairly common practice for organizations to put a freeze on network changes leading up to and during these times to guard against errant changes which could cause outages – and potentially lost business – during valuable high traffic days. One doesn’t have to look very hard to find stories of outages and lost business due to a mismanaged change (also see The Most Dangerous Device on Your Network: Your Admin’s Keyboard) made to the production environment. In fact, in a survey from earlier this year, poor change management was one of the top reasons listed for network outages.

I recently spoke with an information security professional who years ago worked with an organization that had routers in a high-availability cluster for redundancy. During one of the company’s busiest times of the year, a different consultant made an out-of-process change to the cluster on a project he was working on. The result was a network outage – and a significant loss of revenue since this was during a peak business time. Beyond the outage, the issue is that there were no change control tickets or review process followed – hence the change went unchecked.

This example is exactly why freezes are in place during certain times of year. The underlying complexity of making a change (i.e. what devices are impacted, how is traffic impacted, are any new risks created by the change, etc.?) must be understood before a change is processed. Every time you make a change to a production system or network you risk the possibility of having a system outage, corrupted database, faulty route, misplaced firewall rule. With all of that said, change is not something we can run away from.

Even with change management being a challenge for many, we need to rethink this “Ice Age” approach which can also impede business and put it at risk – frozen in a vulnerable, unprotected state. Just because you’re in a freeze doesn’t mean there aren’t risks on your network that need to be addressed.

I’d like to propose a more balanced network freeze – one that enables the business to quickly thaw the network while limiting unnecessary risk. Before the freeze goes into effect, make sure you are coordinated and have a solid plan with your IT operations counterparts (Read How to Cross the Divide with Your IT Operations Colleagues for more tips on working with IT ops peers). This means having a plan that is understood by all parties – whether it is to break the ice if dictated by a highly exploitable security risk or waiting until the network thaws to address a low-risk vulnerability on a non-sensitive system. Just because a freeze is in effect does not mean security stops. Vulnerabilities are still out there as are attackers waiting to exploit them.

While a network freeze may be a necessary part of your SOP during the holiday season, make sure you have a plan to dig out quickly if or when necessary. Happy holidays everyone!