It Disaster Recovery Runbook

Standard Operating Procedure: IT Disaster Recovery Runbook

1. Document Control & Revision History

2. Executive Summary & Scope

This document delineates the institutional protocol for restoring mission-critical IT infrastructure following a catastrophic system failure, cyber-attack, or physical site disaster. This SOP aligns with NIST SP 800-34 and ISO/IEC 27031 standards.

Scope: Applies to all primary data centers, cloud environments (AWS/Azure/GCP), and inter-dependent network architectures. It excludes individual endpoint remediation unless explicitly directed by the Incident Response (IR) lead.

3. Prerequisites & Tool Requirements

• Access: Active Directory Administrative Privileges, Multi-Factor Authentication (MFA) bypass tokens (Offline/Hardware), and Cloud IAM Global Admin.
• Infrastructure: Verified "Air-Gapped" backups (Immutable S3 buckets or Offsite LTO Tapes).
• Documentation: Hard-copy (physical) versions of the Network Topology Map and Asset Inventory.
• Communication: Out-of-band communication channel (e.g., Signal or Satellite uplink).
• PPE: Antistatic wrist straps and ESD-safe floor mats for physical rack hardware restoration.

4. Roles & Responsibilities (RACI Matrix)

R: Responsible, A: Accountable, C: Consulted, I: Informed

5. Step-by-Step Operational Procedure

Phase 1: Assessment and Declaration

• Verify the catastrophic event criteria against current metrics.
• Officially trigger the "Disaster Declaration" via the C-Suite emergency signal.
• Establish the War Room (Physical/Virtual) and log the timestamp of inception.

Phase 2: Isolation and Containment

• Sever public-facing edge gateways to prevent data exfiltration.
• Isolate compromised segments to prevent "lateral movement" of malware.
• Snapshot current states of impacted virtual machines for post-mortem forensics.

Phase 3: Restoration and Recovery

• Initialize "Clean-Room" environment on secondary failover site.
• Verify integrity of the latest Immutable Backup set (Hash Validation).
• Restore Core Infrastructure (Identity Provider, DNS, Database Services) in strict sequence.
• Restore Application Layers utilizing IaC (Terraform/Ansible) templates.

Phase 4: Validation and Cutover

• Perform smoke tests on core business applications.
• Execute vulnerability scan on restored environment before exposing to production.
• Route traffic back to the production environment using traffic management software.

6. Quality Assurance & Verification Criteria

• Recovery Time Objective (RTO): < 4 Hours for Tier-1 Systems.
• Recovery Point Objective (RPO): < 15 Minutes (Data loss threshold).
• Data Integrity Check: 100% parity verification between current restored state and offsite immutable snapshot.
• Performance Benchmarking: Post-restoration latency must remain within ±10% of pre-disaster baseline.

7. Practical Troubleshooting Scenarios

Scenario A: Backup Corruption (Checksum Mismatch)

• Identification: Verification script fails to validate the hash of the backup archive.
• Action: Immediately move to the "N-1" generation backup. Do not attempt to repair the corrupted volume. Escalate to the storage lead for forensic analysis of the failed storage block.

Scenario B: Identity Provider (IdP) Loop

• Identification: On-premises AD and Cloud-based SSO (Okta/Azure AD) fail to authenticate due to corrupted Kerberos tokens.
• Action: Execute the "Break-Glass" protocol using pre-staged local administrative service accounts stored in the physical safe. Re-initialize IdP synchronization once local access is secured.

Scenario C: Network Partitioning (Split-Brain)

• Identification: Failover site and Primary site both attempt to act as primary authorities, causing database collision.
• Action: Execute immediate FORCE_SHUTDOWN on the primary site's network gateway. Implement a 5-minute freeze on write operations while the synchronization engine re-establishes cluster consensus.