Timehri Networks OT Resilience Series

By Franklyn C. Adams, Timehri Networks

An unexpected controller change, suspicious remote session, or frozen SCADA display can put a utility under immediate pressure. An incident response plan gives personnel a shared way to assess what is happening, protect the process, and coordinate the next decision.

For water and wastewater utilities, cyber incident response must account for physical consequences. Disconnecting a device or restarting a server may affect visibility, alarms, or control. The plan must bring operations, OT, IT, and management together before uncertainty becomes conflicting action.

This article complements Part 1 — Continuity of Service: Keep Essential Services Running – Creating and Exercising a Continuity of Service Plan. Continuity planning addresses how essential service will continue. Incident response establishes how the utility will detect, assess, contain, and manage an incident. Disaster recovery provides the detailed restoration procedures. These activities can run concurrently under coordinated leadership.

Figure 1 — How the Three Plans Work Together

Build a plan that people can activate

A useful plan begins with a short activation checklist and leads staff to the procedures they need. EPA’s cybersecurity planning resources include a customizable water and wastewater incident response template and the joint CISA, EPA, and FBI sector incident response guide.

NIST SP 800-61 Revision 3 places incident response within ongoing cybersecurity risk management. For a utility, that means preparation and improvement should continue between incidents.

Figure 2 — OT Incident Response Lifecycle

1. Define scope, triggers, and severity

Identify the facilities and systems covered, including remote access, SCADA servers, engineering workstations, controllers, communications, and business systems whose disruption could affect operations.

Define what personnel should report and who assesses it. Examples include unexplained logic changes, unauthorized access, loss of trustworthy process data, and suspected ransomware. A symptom alone does not establish malicious activity. Staff should record observations, assess operational consequences, and escalate uncertainty without waiting for proof of an attack.

Classify incidents by potential harm to people, water quality, the environment, and service availability, alongside technical scope. A single affected device can warrant urgent action if its process role is critical.

2. Assign decision authority and alternates

Name an incident lead and alternates. Define who assesses process safety, investigates technical evidence, approves containment, authorizes emergency spending, coordinates external support, and releases public information. Include after-hours coverage and a clear handover process.

Operations should have explicit authority over process changes. Technical responders should understand which decisions require operational approval and which emergency protective actions are already authorized by site procedures. A small utility may combine roles, but responsibilities must remain clear.

3. Establish immediate assessment and communications

Specify how the team checks immediate safety and service conditions, validates critical readings through approved independent means, and identifies affected sites. Maintain an incident log with timestamps, observations, decisions, approvals, and actions. Record time zones or known clock discrepancies when correlating events.

Provide an alternative communication method if normal email or collaboration systems are unavailable or suspected compromised. Keep verified contact details for operators, OT and IT support, integrators, management, emergency partners, and specialist responders in accessible offline copies.

4. Prepare containment options with process safeguards

Develop scenario-specific playbooks for incidents such as compromised vendor access, ransomware on an engineering workstation, or an unauthorized controller change. For each option, document its purpose, expected process effect, approval authority, verification steps, and fallback.

Containment may involve restricting a remote access path or isolating an affected workstation. Its suitability depends on the architecture and operating condition. Avoid generic instructions to disconnect all OT systems, reboot controllers, or switch to manual control. Such actions can remove monitoring or control that operators still need.

Link containment decisions to the continuity plan. If safe degraded operation cannot be sustained, identify the approved escalation, service restriction, alternative supply, or controlled shutdown procedure.

Figure 3 — OT Containment Decision

5. Preserve evidence and coordinate notifications

Assign trained personnel to preserve relevant logs, access records, configurations, and other evidence using methods suitable for the equipment. Document collection times, handling, and storage. Protect original evidence where practicable, and record unavoidable changes made to protect the process.

Evidence collection must not delay urgent protection of people or the environment. Avoid unapproved investigative software or intrusive scans on live control equipment.

Figure 4 — Evidence Collection, Protect People First

Maintain a notification matrix identifying recipients, decision owners, applicable reporting triggers and deadlines, and approved communication channels. Validate obligations for the utility’s jurisdiction, contracts, and insurance arrangements. Separate required reporting from requests for technical assistance. Public updates should distinguish confirmed facts, unresolved questions, and protective actions.

6. Define the transition to recovery and closure

Identify the conditions for beginning restoration, including the status of containment, affected access paths, and the trustworthiness of recovery materials. Invoke the disaster recovery plan when systems need rebuilding or replacement. Preserve needed evidence before destructive recovery actions where operationally feasible.

Require technical and operational acceptance before reconnecting restored systems or returning them to control. Establish heightened monitoring and a clear route back to incident response if suspicious activity recurs.

Exercise decisions before an incident

Timehri Networks recommends quarterly readiness checks, focused drills every six months, and a coordinated exercise at least annually. Treat this as a planning baseline, adjusted for risk and applicable obligations, rather than a universal regulatory requirement.

Table 1 — Timehri Networks Recommended Incident Readiness Checks

Frequency	Recommended activity
Quarterly	Verify contacts, alternates, escalation routes, offline access, and alternative communications.
Every six months	Exercise a selected playbook, such as suspected vendor account compromise or loss of trustworthy telemetry.
At least annually	Run a coordinated scenario with operations, OT, IT, management, and relevant partners; review and approve the plan.
After significant changes or incidents	Update affected procedures and retest material weaknesses.

EPA’s tabletop exercise resources support scenario development and evaluation. Discussion exercises should be complemented by practical demonstrations of selected capabilities, with safeguards for live processes.

A scenario worth practicing

An operator notices an unexpected setpoint change during an unfamiliar vendor session. The vendor cannot immediately confirm whether the session is legitimate.

The exercise should establish who validates the process condition, checks change authorization, assesses the remote session, and approves containment. Introduce a second complication, such as unavailable email or a missing shift supervisor, to test alternates and communications.

Measure time to operational assessment, activation, and contact with decision makers. Check whether containment decisions considered process consequences and whether evidence and notifications were handled as planned. Do not simulate the scenario by making unauthorized changes to a live controller.

Make each exercise improve the next response

Debrief promptly and assign every significant finding an owner, due date, and closure test. Correct unclear authority, missing contacts, inaccessible procedures, and unsafe assumptions. Brief affected personnel and retest the changes.

A strong incident response plan enables a utility to act with shared understanding while facts are still developing. Its value is demonstrated when the team can protect the process, coordinate decisions, and support safe service continuity and recovery under pressure.

Sources & Further Reading

Timehri Networks
OT Cybersecurity • Critical Infrastructure