Industrial control systems (ICS) and operational technology (OT) environments are often described as quiet, highly controlled worlds. In reality, they contain a range of risks, unexpected configurations, and operational complexities that are difficult to fully uncover through standard penetration testing or conventional risk assessments.
SecurityWeek spoke with several ICS security experts and companies about their most memorable experiences in the field. These are not theoretical scenarios or lab simulations — they are real situations they encountered while working directly with organizations.
Their stories highlight the gap that often exists between written security policies and what actually happens on the plant floor.
Here are some of the most interesting and cautionary tales shared by ICS security experts, straight from the trenches:
John Simmons, FortiGuard Incident Response, Americas, Fortinet:
“During an incident response engagement in the Middle East, the FortiGuard Incident Response team identified what appeared to be an Iranian-linked APT threat actor attempting to laterally move from the customer’s IT environment into their OT systems. Each time the customer tried to contain the activity, the threat actor adapted, deploying new malware, standing up fresh infrastructure, and reestablishing access almost immediately.
We paused reactive containment of individual issues and our team shifted to a full-scope investigation to understand the attacker’s foothold, movement patterns, and objectives. As we worked through the environment, we uncovered a persistent mechanism we hadn’t expected, an “n-day” vulnerability that hadn’t been publicly documented as exploited in the wild. It gave the threat actor a reliable path to re-enter the network, even after apparent clean up.
Ultimately, the attacker’s goal was clear: establish sustained access to the OT network. We observed repeated attempts to move laterally from the IT network using jump boxes and malicious tunnels, but they never fully compromised the OT network. By coordinating closely with the customer and executing a comprehensive remediation plan, we were able to cut off access, remove persistence, and stabilize the environment before the OT network was impacted.”
Brian Proctor, Founder and CEO, Frenos:
“Working at a combined-cycle power generation plant, a cybersecurity compliance member walked in the room and said, “I’m gonna conduct a vulnerability scan on the turbine networks.” I replied, “Oh no, you aren’t; this isn’t like an enterprise IT environment, you can’t use those tools”.
He escalated the issue and showed me emails from directors mandating the scan. Well he did, and two minutes and 11 seconds after he pushed that scan button both turbines completely stopped, creating a sound heard from over a mile away.
They immediately escorted us off the site and our cybersecurity team wasn’t allowed back in for years. Lesson learned, don’t use IT tools in OT.”
Morey Haber, Chief Security Advisor, BeyondTrust:
“Shortly after 9-11, my presence was requested at a secure facility in South Florida. With security being the most paramount concern in everyone’s mind, I hightailed it down the Florida Turnpike to my destination.
Needless to say, the police officer that pulled me over for doing 90 did not share my enthusiasm for security. After a nice hefty fine, I arrived onsite and began my work. I was delegated to work with a contractor that was completely unfamiliar with my tools and had an open source product he preferred better. He loaded it and showed me all the capabilities he liked and wanted. Needless to say, he was trying to convince me what he wanted was better than what the client had paid for and wanted installed.
After his brief demo, we began installation and usage of the solution the client wanted. Now this facility was so secure, I was not allowed to touch the keyboard. The contractor needed to do all the typing, and simple things like passwords just did not seem to work. He blamed my tool and again reinforced why his was better. Did I say this was a secure government facility? In either case, after troubleshooting for more than a day, I finally had him re-enter credentials, and things started to work. We had barely enough time to finish the setup before we did a demo for the officers in charge of the facility. The demo went fine, and we were fully installed, operational, and the contractor was trained.
Before I left, I had a debriefing with a senior official at the facility. We discussed the delays, and I mentioned the open source software the contractor loaded. He was completely lost in the conversation and stated that this was a secure facility, and no one should ever load unapproved software on the network. He promptly called the contractor in and confronted him with the accusation. He attempted to do the installation and pleaded his case. I was asked to leave and not to worry about anything else.
I gathered my things, turned in my bathroom hall pass, and saw the contractor leaving too. I said goodbye and found out through a rather awkward conversation that he was terminated for installing unauthorized software. I have never heard from either of them again.”
Kevin Paige, Field CISO, C1:
“I was hired to assess a federal engineering agency before an upcoming audit. The staff was largely new and wanted a clean picture of their gaps before the auditors arrived. A few days in, during network discovery, I came across a cluster of servers I couldn’t account for. They didn’t appear on any list I’d been given. I traced them to a different floor, behind a door very few people had reason to open. The same room held network cameras, also unaccounted for, also using factory defaults.
The Solaris servers hadn’t been patched in years, and local accounts, like the cameras, were on default credentials. I was told they were mission-critical: they ran the agency’s industrial field control systems. The doctrine had been to leave them alone; they were Solaris, they were physically isolated, and only a handful of engineers knew how to operate them. There was no path in, I was told, except through that door.
From a corporate workstation a few floors away, and later demonstrated from anywhere on the public internet, I could reach the servers and cameras, log in with the default credentials, escalate to root, and issue commands directly into the field control systems. The failure modes of those systems were physical and public; misuse of that access wouldn’t have been a conventional data breach but a real-world event with consequences felt by people who never knew the agency ran a single server. Nothing about the environment was isolated. The original developer had retired; the maintenance contract had lapsed while the agency searched for a successor. In that gap, “locked away” had become a story the organization told itself, and “not on the internet” had quietly become another one.
The lesson I keep coming back to is that physical isolation does real work only if the network reflects it, and “we’re not exposed” only counts when measured rather than assumed. When knowledge of a critical system walks out of an organization because the developer retires, the contract lapses, or the right question goes unasked for long enough, those systems don’t become safer for being undisturbed. They become more dangerous because the organization slowly forgets they’re reachable at all.”
Agnidipta Sarkar, Chief Evangelist, ColorTokens:
“Throughout my time as a CISO, my biggest challenge was building breach readiness during digital transformation in OT. Rapid digital transformation meant OT could now get connected to IT. But it was not only IT but also shadow IT and shadow SaaS.
About that time, one of the largest pharma companies in the country was breached, and they notified the financial stock exchange regulator that they would incur a revenue loss due to a cyberattack originating in a foreign nation that traveled all the way to the HQ. I knew the CISO, and talking to him was what saved my day.
One element stood out for me. And that was something unique to the pharma industry. Computer Systems Validation, or CSV, is a documented process for ensuring that a computer system performs exactly as designed to maintain data integrity, patient safety, and product quality, and to reduce the risk of product recalls and regulatory action. And CSV can take anywhere between 4 weeks and 18 months, depending on the scope.
I realized that it would be a nightmare if our GMP systems were hacked. I had to make “corrections”. That led me back to the shopfloor. And we went on a hunt to find devices and connections that were “shadow”. And we found a treasure trove. We found disconnected networks full of old, vulnerable systems. We found open USB ports on an old labeling machine that was active. We found Windows XP consoles that were not connected to the IT network but had a nearby LAN port. We found old printers connected to the network. The next step was not easy. I approved connections, scanning, and disconnections.
I had to sit down with all stakeholders and quickly draft a plan to ensure we could work within GMP constraints and avoid a CSV. And, most importantly, human safety was ensured. The OT leadership led the charge, and after 48 hours of continuous effort, we had reduced exposure to acceptable levels. We removed all Shadow SaaS and streamlined many Shadow IT instances, bringing them under IT management. And yet, every time I go back, I think we got lucky. We were not breached.”
Vivek Ponnada, SVP Growth and Strategy, Frenos:
“Walked into an oil and gas refinery to discuss a risk assessment. They chuckled and said, look around you – we’ve workstations all on Windows XP, out of support Cisco Switches etc., so we know it’s risky. Questioned what exactly a risk assessment will show.
The reality is that a risk assessment would help them move on from assuming the worst to actually documenting and quantifying the risk, then help figure out mitigations that make sense.
Every OT organization has to start somewhere e.g., identify whether their perimeter protection is in place/valid, start segmenting non-patchable devices, ensuring any remote access is monitored (and access restricted with MFA) etc.”
Dan Hewitt, staff product manager, OT Security, Tenable:
Story 1
“A large manufacturing company that we work with had just finished making a multimillion-dollar investment in firewall technology across its entire footprint. The engineering team told us they finished implementing the firewalls successfully, and their goal was simple. Ideally, if we stood up our solution in an environment central to the business, we shouldn’t be able to point it outward to the various sites and discover control systems and sensitive OT devices.
However, when we deployed Tenable OT and directed it to perform remote discovery of their manufacturing environment, it came back with thousands of devices. Not only this, but we were able to get really deep information about the devices because we were able to speak to the proprietary protocols of the devices, clear across the network to the remote facility.
It was obvious from that point that the firewalls were not blocking or hindering traffic in any way within the company network. It was helpful to have a complete inventory before applying the firewall rules. It let us confirm that as we added and adjusted rules, we maintained the same level of visibility and control over the systems.
Although some remote manufacturing locations were large enough to warrant a full deployment of OT security, we had a baseline to compare against. From that point onward, we knew what good visibility looked like, so that when we made firewall changes, we understood if we were locking out the good guys or just the bad actors.”
Story 2
“We have a large grocery retailer that needed deeper visibility over their stores and distribution centers than they were able to get from their passive threat detection solution or from their typical IT VM scans. The customer knew that they had numerous devices per distribution center, such as chillers, conveyors, and other smart process gear that were completely missed by these other solutions. Right away after deploying our solution, the customer noticed that we were able to detect passively nested devices that were essentially an isolated process control network behind other devices. Their passive-only solution missed this entirely because it didn’t have the context or the understanding of the proprietary industrial protocols that our OT security supports.
After monitoring for just a couple of days, the customer found over 1000 additional assets across the two sites that they hadn’t previously known about. These assets were indirectly networked and behind other control systems. A number of devices were uncovered that were end of life or end of support and couldn’t be updated to resolve the vulnerabilities that were affecting them. This fostered a much better conversation between teams around how they can work to reduce risk together by replacing obsolete equipment while investigating misconfiguration on their still-active supported equipment.”
Jose Bonilla, Lead Professional Services Engineer, Nozomi Networks:
“After installing a Nozomi Networks sensor at a manufacturing facility, the customer noticed a constant malformed traffic alert originating from a Windows computer within a manufacturing cell. Initially, the user was inclined to rule out the alert as a false positive, assuming it was simply noise due to the age and configuration of the systems (these systems had been running for years without major updates). However, given the recurring nature of the alert, they decided to review the event with us.
We discovered that the sensor was triggering on DNS traffic originating from this computer because the DNS queries contained extremely long labels. Furthermore, we observed that the subdomains appeared to be encoded, and that the frequency of the queries was abnormal and suspicious for a computer in a manufacturing cell. We informed the customer that this behavior displayed all the characteristics of malware communicating with a command-and-control server and performing data exfiltration via DNS tunneling.
The customer initiated their incident response procedures with the support of their IT SOC and analyzed the system, ultimately identifying a well-known malware. The subsequent post-incident analysis highlighted key gaps in visibility and detection within areas of the OT environment, reinforcing the value of continuous monitoring. These insights provided momentum for other projects to strengthen their OT security posture.”
Nicholas DiCola, VP of Customers, Zero Networks:
“At a global manufacturing company, a routine risk assessment started with a simple question: if an attacker got in, could they spread? The assumption was that operational systems were effectively isolated. But as the team mapped real-world connectivity, they found the opposite. IT and OT environments were more interconnected than expected, and lateral movement paths were wide open. If those production systems were impacted, uptime would be at risk and business operations would effectively come to a full stop.
The challenge was how to fix it without disrupting always-on operations. In an environment where downtime wasn’t an option and many devices couldn’t support traditional controls, the team instead focused on understanding actual communication patterns across the network. Within weeks, we were able to define and enforce policies based on real behavior—allowing legitimate traffic to continue uninterrupted while quietly eliminating unnecessary pathways between systems, particularly those bridging IT and OT.
The takeaway is one we see repeatedly in industrial environments: Many organizations believe they’re segmented, but in practice, their networks are far more permissive than intended. It only takes a single foothold for an issue in IT to reach the systems that keep production running, which is why controlling lateral movement has become central to business resilience.”
