Timehri Networks OT Resilience Series

By Franklyn C. Adams, Timehri Networks

A backup can help restore a server. A disaster recovery plan must explain how the utility will restore the systems, dependencies, and verified control functions needed for safe operations.

Water and wastewater utilities may need recovery after ransomware, equipment failure, fire, flooding, power damage, or a failed system change. In each case, restoration depends on more than having files available. Personnel need compatible equipment, trusted configurations, usable procedures, and authority to return systems to service.

This article complements Part 1 — Continuity of Service: Keep Essential Services Running – Creating and Exercising a Continuity of Service Plan and Part 2 — Incident Response: Control and Manage the Incident – Creating and Exercising an OT Incident Response Plan. Continuity sustains essential service during disruption. Incident response coordinates assessment and control of the event. Disaster recovery restores the affected technology and infrastructure. Recovery can begin while response continues, provided the teams coordinate their decisions.

Figure 1 — How the Three Plans Work Together

Start with the service and work back to the systems

Define the plan’s scope and activation criteria. Include the physical and technical dependencies required to recover critical treatment, pumping, collection, and distribution functions. Establish who declares a recovery operation, sets priorities, authorizes resources, and approves completion.

Build a recovery inventory that connects each critical function to its controllers, operator interfaces, servers, communications, power, field instrumentation, and supporting services. Include system owners, vendors, replacement lead times, and approved alternatives. Identify dependencies such as licenses, certificates, software versions, and specialist engineering tools.

NIST SP 800-82 Revision 3 addresses OT security in the context of its performance, reliability, and safety requirements. Recovery planning must respect those same constraints.

Set recovery objectives that operations can defend

Work with operations and engineering to determine the maximum tolerable interruption for each essential function and the limits of the continuity arrangements. Then establish achievable recovery objectives.

Table 1 — Recovery Plan Meaning

Objective	Meaning for the recovery plan
Recovery time objective (RTO)	The target elapsed time to restore an agreed capability. Define when the clock starts and what verified condition counts as restored.
Recovery point objective (RPO)	The acceptable extent of data loss, expressed as a point in time to which data must be recovered. Define it separately for relevant data sets and systems.
Operational acceptance	The specific evidence and approvals required before a restored system resumes its intended process role.

RPO is useful for historian and other time-based records. For controller programs and configurations, also require traceability to the approved operating baseline. A recent copy may still contain an unauthorized change, while an older copy may omit a necessary engineering modification.

For example, if an approved continuity arrangement can sustain a function for eight hours but demonstrated recovery takes twelve, the plan has a service gap. Address that gap through faster recovery, stronger continuity capability, additional resources, or a revised service strategy approved by responsible personnel.

Build a complete recovery package

Identify what is needed to rebuild each system, not just what the backup software captures. Depending on the environment, recovery materials may include:

Controller and remote terminal unit programs, hardware configurations, and approved setpoints.
SCADA and HMI projects, alarm configurations, tag databases, and application dependencies.
Server images, databases, historian data, and installation media.
Firewall, switch, router, and remote communications configurations.
Supported firmware and software versions, licenses, certificates, and vendor instructions.
Network drawings, equipment inventories, commissioning records, and recovery runbooks.

Provide protected emergency access to required credentials and keys without placing secrets in broadly distributed documents. Ensure recovery access does not depend entirely on the identity system or network that may be unavailable.

NIST’s OT Backup Quick Start Guide emphasizes regular backups, integration with change management, testing, and review during recovery exercises. Apply those principles to approved OT changes so recovery materials remain aligned with the installed environment.

Protect copies from the same failure or compromise that could affect production. Use suitable offline or immutable copies, separate administrative access, and geographic separation where needed. Retain appropriate historical versions and verify that authorized staff can retrieve and decrypt them. Replication can also reproduce corruption or malicious changes; it does not by itself establish recoverability.

Write a dependency-based restoration sequence

Document a recovery sequence for the actual architecture. Before installing equipment, confirm that the site, electrical supply, environmental conditions, and communications are suitable. Determine which trusted network, identity, name resolution, time, or virtualization services must be available before dependent applications can function.

For each recovery step, specify prerequisites, responsible personnel, approved materials, expected results, verification, and rollback or stop conditions. Include vendor and integrator support, replacement hardware compatibility, and how to proceed if a primary supplier is unavailable.

After a cyber incident, coordinate with the incident response team to assess backup integrity, address the entry path and affected credentials, and avoid reconnecting restored systems to a compromised environment. The newest backup is not automatically the best recovery source.

Figure 2 — Backup Is Not Recovery

Verify the process before returning control

A system that starts successfully may still have incorrect settings, stale data, missing alarms, or mismatched software. Define acceptance checks appropriate to its role, including configuration comparison, timestamp and data-quality checks, communications, access controls, alarms, and approved process safeguards.

Use qualified personnel and approved commissioning procedures to verify critical instrumentation and control behavior. Tests affecting live equipment require operational authorization, safe conditions, abort criteria, and a recovery path. Do not bypass protective functions simply to accelerate acceptance.

Require technical and operational sign-off, a coordinated handover from continuity arrangements, and enhanced monitoring after restoration. Document remaining restrictions and the conditions that would trigger rollback or renewed incident response.

Figure 3 — Trusted OT Recovery Workflow

How often should recovery be tested?

Timehri Networks recommends quarterly recovery-readiness reviews, practical restoration tests at least every six months, and an integrated recovery exercise at least annually. These are planning recommendations, not a universal regulatory schedule. Increase frequency where risk, change rate, or applicable obligations require it.

Table 2 — Timehri Networks Recommended Disaster Recovery Readiness Review

Frequency	Recommended activity
Ongoing and after approved changes	Monitor backup results, investigate failures, and update affected recovery materials through change control.
Quarterly	Check recovery access, inventories, contacts, spares, licenses, and coverage of critical assets.
At least every six months	Restore representative critical systems in an isolated environment and measure recovery against objectives.
At least annually	Exercise the coordinated restoration sequence, including operational acceptance; review and approve the plan.
After major changes, incidents, or failed tests	Retest affected procedures and verify corrective actions.

Rotate test coverage so all critical recovery paths are demonstrated over a defined period. A tabletop checks decisions; a successful backup job checks that a job completed. Neither proves that applications, configurations, and dependencies can be restored within the required time.

Test an imperfect recovery situation

Consider a failed SCADA host whose replacement has different hardware. The latest backup is available, but the license needs reactivation and the normal identity service is offline.

Can the recovery team access the backup, assemble the compatible software, contact the vendor, restore dependencies, and verify the application? Does the elapsed recovery time fit within the continuity plan’s operating limits? Can an alternate perform the work using the runbook?

Record actual times, missing prerequisites, failed steps, and acceptance results. Assign findings an owner and deadline, revise the plan, and retest significant gaps.

A disaster recovery plan provides confidence when the utility has demonstrated that it can restore trusted systems and safely resume their operational role. Tested backups, clear procedures, and verified process acceptance make that confidence defensible.

Sources & Further Reading

Timehri Networks
OT Cybersecurity • Critical Infrastructure