Change is the norm for most security teams, as evolving business requirements as well as new threats dictate new or modified policies and tools. But security change management is handled poorly by most organizations. In some cases, it is “just” inefficient and slow, further contributing to the security department’s reputation of interfering with the business. In other cases it is error-prone, exposing the organization to risks from changes that were executed incorrectly or even causing network outages that can damage the company’s revenue and reputation. You need to look no further than the recent GoDaddy outage – originally suspected as a result of a DDoS attack, but later attributed by GoDaddy’s CEO to “internal network events”.
Several key factors contribute to change management challenges including:
• Complexity – today’s “security policy” is spread out across multiple technologies and vendors and often spans multiple geographies and teams.
• Lack of a formal change control process that clearly defines and enforces the necessary steps, including documenting the “who, what, when, why and how” of making a change, and more importantly, the discipline to enforce this process even during “emergency” changes (even if it is after the fact). Questions such as “did anyone change anything on the firewall yesterday because XYZ just stopped working” are asked way too often by organizations.
• Silos that separate security and operations teams which can lead to out-of-band changes, where one team makes a change without the proper checks and balances. In a network security survey from earlier this year, 55% of those surveyed said that an out-of-band or out-of-process change resulted in a system outage.
• The “If it ain’t broke, don’t fix it” approach. Just because your network hasn’t gone offline doesn’t mean network security changes are causing issues such as slowing down the network or opening up gaps for attackers to exploit.
If we focus on firewall changes (since firewalls by far introduce the largest number of changes), many organizations have a helpdesk ticketing system that is used to track the status through completion. However these solutions were never designed to provide the necessary insight into everything that goes into processing a firewall change.
So as more changes come down the pipe, what needs to change in the change management process? In no way is my list below exhaustive, but it gives you a few things to think about in bringing your change management process up to par with industry leading organizations:
• Documentation. One of most important tasks that is least liked is documenting firewall changes that were made, such as what rules were added or changed, by whom, when, etc. so that other administrators understand the purpose of each rule. Good documentation can simplify troubleshooting and reduce the risk of service outages.
• Get Network Security and Operations teams on the Same Page. This is more of a C-level initiative, but one that is critical in the day-to-day effectiveness of the change management process. Security and operations teams both should have ownership of different aspects of this process and if they are aligned properly, both security and agility will win out.
• Reconciliation. Let’s face it, every organization has its “cowboys” that may introduce out-of- band change. Make sure that every change request is processed as approved but just as important, make sure that every change can be mapped out to a proper request.
• Automation. Manually discovering all of the devices and rules impacted by a potential change, as well as understanding any potential change in risk or compliance levels, is time-consuming, tedious and prone to mistakes. Automation can be used to ensure accuracy, reduce risk and significantly reduce the time to process changes, which enables the organization to be able to more quickly respond to changing business needs.
You have two choices – you can continue to slowly chug along with manual change management processes that drain your IT resources, most likely introduce risk and ultimately impede agility. Or you can improve the process and communication and better enable these improvements through automation that helps align the different stakeholders involved in the change process (i.e. network operations, network security, compliance, business owners, etc.) and helps the business run more smoothly.
