Ot Incident Response Plan Template

Standard Operating Procedure: Operational Technology (OT) Incident Response Plan

1. Document Control Block

• Document ID: SOP-SEC-OT-042
• Effective Date: October 24, 2023
• Version: 2.1.0
• Review Cadence: Semi-Annual (Next Review: April 24, 2024)
• Owner: Julian Vance, Chief Architect, Template Registry
• Classification: Restricted - Internal Operations Only

---

2. Executive Summary & Purpose

This Standard Operating Procedure (SOP) defines the institutional-grade framework for detecting, containing, eradicating, and recovering from cyber-physical security incidents within Operational Technology (OT) and Industrial Control Systems (ICS) environments. The primary objective is to ensure human safety, protect environmental integrity, maintain physical process continuity, and preserve forensic data integrity without violating real-time industrial availability requirements.

---

3. Scope & Prerequisites

3.1 Scope

• Applicability: All manufacturing plants, energy distribution nodes, remote terminal units (RTUs), programmable logic controllers (PLCs), distributed control systems (DCS), and associated industrial DMZs (IDMZ) managed by Template Registry.
• Exclusions: Enterprise IT infrastructure (laptops, corporate email, enterprise resource planning systems) unless directly leveraged as an attack vector into the OT domain.

3.2 Prerequisites & Required Tools

• Hardware/Software: Out-of-band management interfaces, air-gapped forensic workstations, write-blockers, packet capture (PCAP) appliances, certified OT asset discovery tools (e.g., Claroty, Nozomi, Dragos).
• Documentation: Up-to-date Purdue Model network diagrams, physical piping and instrumentation diagrams (P&IDs), master asset inventory registries.
• Personal Protective Equipment (PPE): NFPA 70E compliant arc flash gear, safety boots, and eye protection for physical asset access in hazardous industrial zones.

---

4. Roles & Responsibilities (RACI Matrix)

---

5. Step-by-Step Procedure

Phase 1: Preparation & Triage (T+00:00)

• 1.1 Acknowledge incoming industrial SIEM or physical process anomaly alerts and log initial event metadata into the secure ticketing system.
• 1.2 Convene the OT Incident Response Team (OT-IRT) via secure, out-of-band communication channels (e.g., encrypted radio or secondary cellular network).
• 1.3 Verify physical site safety and confirm with the Plant Operations Manager that no immediate threat to human life or environmental containment exists.

Phase 2: Identification & Scoping (T+00:30)

• 2.1 Isolate suspicious segments by disabling compromised network links at the IDMZ boundary, prioritizing logical segmentation over physical power-down to prevent unmanaged process state drops.
• 2.2 Execute passive network traffic analysis using certified PCAP tools to map lateral movement and command-and-control (C2) channels.
• 2.3 Query engineering workstations and HMI panels for unauthorized firmware modifications, unexpected logic changes, or unauthorized user accounts.

Phase 3: Containment & Safety Preservation (T+01:00)

• 3.1 Engage manual override controls on critical loops if automated safety instrumented systems (SIS) are suspected of compromise.
• 3.2 Implement emergency firewall rule drops to sever external vendor remote access paths without disconnecting local plant-floor safety networks.
• 3.3 Crucial Rule: Do not reboot or power off PLCs or RTUs during containment unless directed by the Forensic Investigator, as volatile memory (RAM) containing active malware payloads will be permanently lost.

Phase 4: Eradication & Remediation (T+04:00)

• 4.1 Isolate and remove compromised controller logic; verify integrity by comparing running firmware hashes against known-good baseline repositories stored in the Template Registry.
• 4.2 Reimage compromised engineering workstations and HMIs from verified golden images.
• 4.3 Patch exploited vulnerabilities, update default industrial protocol credentials (e.g., Modbus, DNP3, BACnet authentication keys), and enforce strict multi-factor authentication (MFA) on all jump hosts.

Phase 5: Recovery & Normalization (T+08:00)

• 5.1 Execute step-by-step process restoration in coordination with Plant Operations, bringing sub-systems online sequentially while monitoring telemetry for anomalous pressure, temperature, or flow variations.
• 5.2 Validate sensor calibration and control loop stability before declaring the process loop fully operational.
• 5.3 Transition monitoring from incident posture back to continuous baseline surveillance.

Phase 6: Post-Incident Review & Hardening (T+24:00)

• 6.1 Conduct a formal blameless Post-Mortem with the OT-IRT, plant engineers, and executive leadership within 48 hours of incident closure.
• 6.2 Compile all forensic artifacts, network logs, and timeline reconstructions into a centralized incident ledger.
• 6.3 Update the Template Registry OT architecture templates to reflect newly discovered attack paths and hardening requirements.

---

6. Quality Assurance & Pro-Tips

6.1 Best Practices

• Safety First: Human life and physical asset integrity supersede forensic data collection. If physical danger arises, immediate plant shutdown procedures override standard containment steps.
• Out-of-Band Communications: Assume corporate IT infrastructure (email, Slack, VoIP) is compromised during an advanced persistent threat (APT) campaign targeting OT. Always default to analog or dedicated secure out-of-band channels.

6.2 Common Pitfalls

• The "Reboot" Trap: Power-cycling a PLC wipes volatile RAM containing critical attack telemetry. Always capture forensic memory states prior to device remediation.
• IT/OT Convergence Blindness: Applying standard IT endpoint detection and response (EDR) agents directly to legacy controllers without vendor validation can cause kernel panics and process crashes.

6.3 Metric Thresholds

• Mean Time to Detect (MTTD): < 15 minutes for critical process anomalies.
• Mean Time to Contain (MTTC): < 60 minutes from verification of malicious activity.
• Forensic Integrity Compliance: 100% chain-of-custody documentation for all pulled artifacts.

---

7. Frequently Asked Questions (FAQ)

Q1: Can we shut down the master control network immediately upon suspecting ransomware?

A: Absolutely not. Sudden, uncoordinated shutdowns of continuous industrial processes (such as chemical refineries or power generation turbines) can cause catastrophic mechanical damage, structural explosions, or severe environmental releases. Containment must be surgical (isolating logical paths) rather than brute-force (pulling power), executed in lockstep with the Plant Operations Manager.

Q2: How do we handle third-party vendors who require remote access during an active incident?

A: All third-party remote access tunnels must be immediately terminated upon incident declaration. If vendor assistance is required for proprietary controller remediation, access must be re-established exclusively through a heavily monitored, temporary jump host equipped with session recording, operating under the direct supervision of an OT Systems Engineer.