Still need to get your cybersecurity strategy in place? Start with good asset management.

Designed for water and wastewater utilities seeking to secure their systems and acquire funding for further protection, Secure Systems: Fortifying Cybersecurity for Utilities is a series of articles addressing the unique challenges utilities face when establishing robust cybersecurity.

Control of six large, 250 horsepower, finished water pumps suddenly went dark. The pumps just stopped, and no one knew why.

Called to the water utility to investigate, my team and I examined each network switch and the programmable logic controller (PLC) controlling the pumps. We examined drawings. All network hardware seemed to be accounted for.

After some investigation, it turns out the original systems integrator had installed an additional network switch in the system. However, this additional switch wasn’t documented anywhere! Once located, the bad switch was replaced, the communication problem was solved, and the pumps were up and running again.

The utility client had no way to know the switch even existed; it wasn’t shown on the design or as-built drawings. Had the switch been tracked during construction as part of an asset management program, the power loss to the pumps would have likely been a 15-minute problem instead of a days-long problem with the potential to turn into a cybersecurity risk.

Good asset management is the foundation of a good cybersecurity strategy

I got my start in the water industry as a treatment plant operator and later as an operations manager. I’m familiar with the many responsibilities that come with the position — managing budgets, coordinating maintenance and repair, overseeing construction and startup, supervising staff. And I understand how asset management for cybersecurity can be seen as a lower priority. Safety concerns are minimal if there’s an unmanaged switch that goes bad, right? That may be true, but there are other consequences that are well worth avoiding.

That story about the downed pumps and the mystery hidden switch could have had a much different ending. Not only was that switch unknown, but it also wasn’t configured properly. Worst case scenario if it had been hacked? The plant could have been completely shut down. The pumps and variable frequency drives (VFDs) could have been damaged by phase switching or other nefarious digital tampering. Or there could have been damage to a PLC. Replacing just one PLC can cost from around $8,000 to as much as tens of thousands of dollars. Even replacing only parts for legacy PLCs can be costly, as those parts can be hard to get, leading to more downtime. In addition to financial impact and downtime, permit violation is also a potential consequence of a bad actor hacking into a network through an unknown, unmanaged asset.

And that story isn’t an anomaly when it comes to the occurrence of unknown assets. It’s just one example. Here’s another: packaged equipment suppliers frequently place cellular remote access modems in their cabinets for troubleshooting and maintenance purposes. Their intent isn’t nefarious, but they often place these modems without permission, creating an unknown and unmanaged access point into the network.

Reducing risk and improving resilience are everyday, commonsense goals when it comes to safeguarding critical infrastructure. Asset management as part of an overall cybersecurity strategy shouldn’t be dismissed as nice-to-have or relegated to being a reminder on a sticky note, waiting for the time when everything else is done. It should be a priority.

The reason why is simple and obvious: you can’t protect what you don’t know exists. And as technology advances, there are more assets than ever before to track and protect. This is about way more than pumps, motors, and valves.

Fifteen to twenty years ago, most network and communication assets were network switches, firewalls, and modems — that was the bulk of it. Now, many pump motors, VFDs and motor control centers (MCCs) come with networked capabilities and managed web servers. Today, there are just more components integrated into the network overall.

Each of those networked assets represents a potential entry point or target for cyber threats. That’s not hyperbole; it’s just the reality.

Asset management provides protection and other benefits

Good asset management practices include a comprehensive inventory of all hardware, software, and data assets, including their location, status, age, software versioning, and ownership. It also includes a comprehensive history of asset maintenance, repairs, replacement, and even decommissioning.

Having these practices in place helps utilities avoid the negative consequences of cyber attacks and provides other significant benefits.

When you know what all your existing assets are, you can accurately assess and prioritize risks based on criticality and vulnerability. Asset management, in short, facilitates effective risk assessment of operational technology (OT) and information technology (IT) assets. It prevents unauthorized network components from proliferating and introducing hidden vulnerabilities; ensures that all assets are current with the latest firmware updates and patches, reducing the risk of exploitation; and enables better implementation of privilege access controls, limiting exposure.

Asset management also creates operation continuity. When staff turnover happens, legacy information isn’t lost. Good asset management is a tracking system and up-to-date record that endures through personnel changes. Good asset tracking helps utilities meet compliance requirements, which emphasize visibility and control over critical systems and public health.

And comprehensive asset management helps lay the groundwork for identifying data flows and implementing stricter traffic controls as cybersecurity practices mature, though additional network visibility tools may be needed to fully map communication patterns and enforce more control.

Another benefit? Clarity about how to respond to incidents. Knowing what systems and devices are affected and any history of incidents with these systems or devices enables faster and more precise responses.

Tools and tips for starting asset management for cybersecurity

Inventory is step one. Utilities should document every asset in their communications network, along with how devices communicate with each other. There are good, automated tools to help get this done.

A widely used approach involves vulnerability scanning software that can discover hosts, identify installed software, and assess potential risks. While these software tools’ primary function is vulnerability assessment, many also include features for asset identification, helping organizations gain visibility into their SCADA network components.

Many tools are built for IT systems — they use aggressive scanning techniques, such as sending large volumes of connection requests, to identify devices and services. IT environments can typically handle this, but OT networks are often more sensitive and may experience disruptions or outages. Takeaway? For OT environments, use tools designed with non-intrusive scanning methods or run scans in a controlled manner.

A good rule of thumb, whichever tool a utility decides is most appropriate, is to have knowledgeable personnel who understand the network environment and the tool’s limitations overseeing its use.

Step two is to categorize assets by criticality and sensitivity, focusing on those assets that are essential to your operations and public health and safety. Step three is to put processes in place for continuous monitoring to detect changes or additions to the asset inventory and for regular audits of the inventory to ensure accuracy and relevance as systems evolve.

And finally, just like a utility needs knowledgeable personnel to run any asset management tool effectively, it needs to make sure employees understand the importance of asset management and know to report unauthorized devices or applications. Good employee training is a must.

Join the asset management movement and set the foundation for a good cybersecurity program

The adoption of asset management best practices continues to increase across the water industry. In the 2020 Level of Progress in Utility Asset Management Survey Results from the American Water Works Association (AWWA), a follow-up to the 2015 survey, results showed, among other things, an increase in “investment in asset management, more asset management planning and support, and the advancement of asset management practices across the sector.” That’s progress.

To utilities still weighing the cost and benefits of asset management — add “foundational for a good cybersecurity strategy” to the asset management benefits pile.