Network Disaster Recovery Plan Template

Standard Operating Procedure: Network Disaster Recovery Plan (NDRP) Execution

Document ID: SOP-TR-NET-042
Effective Date: October 24, 2023
Version: 3.2.0
Review Cadence: Semi-Annual

---

1. Executive Summary & Purpose

This Standard Operating Procedure (SOP) defines the institutional-grade framework for executing the Network Disaster Recovery Plan (NDRP) at Template Registry. The objective is to establish a deterministic, repeatable, and audited workflow to restore core network infrastructure, edge routing, and interconnectivity to an operational state following a catastrophic failure, cyber incident, or physical disaster. Adherence to this protocol minimizes Mean Time to Recovery (MTTR) and enforces compliance with ISO/IEC 27001 business continuity mandates.

---

2. Scope & Prerequisites

Scope

• Encompasses all production local area networks (LAN), wide area networks (WAN), data center spine-leaf fabrics, and cloud-interconnect edge gateways managed by Template Registry.
• Applies to Primary Data Center (DC-1) and Secondary Disaster Recovery Facility (DR-2).

Prerequisites & Required Tools

• Hardware/Access: Out-of-Band (OOB) management console access via dedicated cellular/satellite uplinks, crypto-tokens (YubiKey) for multi-factor authentication, enterprise laptop with pre-configured serial/rollover cables.
• Software: Ansible Core v2.14+, Git (access to tr-infra-net-config repository), Terraform v1.5+, SecureCRT/Termius with SSH key-pair authentication.
• Data: Read-only access to the cold-storage offsite backup repository containing encrypted network state snapshots (Configuration backups, BGP route filters, SSL/TLS certificates).
• Physical/Environmental: Level 3 access credentials to DR-2 Meet-Me-Room (MMR) and secure server enclaves.

---

3. Roles & Responsibilities

---

4. Step-by-Step Procedure

Phase 1: Incident Assessment & Activation

• 1.1 Verify trigger conditions via automated monitoring alerts (PagerDuty SEV-1) or manual declaration by the Incident Commander.
• 1.2 Convene the Emergency Bridge via secure communication channels (Signal/Matrix bridge).
• 1.3 Isolate the scope of failure: Determine whether the event is localized to DC-1 or a systemic network partition.
• 1.4 Authorize the execution of the NDRP by signing off in the Incident Management portal.

Phase 2: Out-of-Band (OOB) Isolation & Triage

• 2.1 Establish secure Out-of-Band (OOB) management sessions to critical core routers and firewalls via cellular gateways.
• 2.2 Verify status of core environmental controls, power distribution units (PDUs), and primary interface links at DR-2.
• 2.3 Execute status verification script to check control-plane responsiveness:

  ansible-playbook -i inventories/dr2/hosts playbooks/net_health_check.yml --extra-vars "target=core"
  

• 2.4 Document baseline hardware failure metrics and preserve corrupted system logs for forensic analysis.

Phase 3: Infrastructure Provisioning & Configuration Restoration

• 3.1 Pull the latest immutable network configurations from the verified Git repository (tr-infra-net-config/releases/stable-latest).
• 3.2 Provision replacement physical spine-leaf infrastructure at DR-2 utilizing automated Zero-Touch Provisioning (ZTP) profiles.
• 3.3 Apply base golden-image configurations to core switching fabrics:

  terraform apply -target=module.dr2_core_network -auto-approve
  

• 3.4 Validate interface operational states, MTU settings, and link aggregation groups (LAG/LACP).

Phase 4: Routing, BGP, & Security Policy Re-Establishment

• 4.1 Re-establish external BGP peering sessions with upstream Internet Service Providers (ISPs) and Cloud Interconnects.
• 4.2 Verify Autonomous System (AS) path advertisements and route propagation tables via looking glass utilities.
• 4.3 Apply and verify next-generation firewall (NGFW) security policies, Access Control Lists (ACLs), and micro-segmentation rules.
• 4.4 Force synchronization of Virtual Private Network (VPN) tunnels and remote access gateways.

Phase 5: Validation & Return to Service (RTS)

• 5.1 Execute end-to-end synthetic transaction probes (ICMP, TCP handshake latency, DNS resolution time) across all network tiers.
• 5.2 Confirm application cluster connectivity to restored database and storage area networks (SAN).
• 5.3 Obtain formal sign-off from the Information Security Officer regarding compliance verification of restored firewall rules.
• 5.4 Transition traffic routing from fallback paths to primary operational routes via progressive load-shedding.
• 5.5 Declare formal termination of the network disaster recovery phase and publish the incident ticket status update.

---

5. Quality Assurance & Pro-Tips

Best Practices

• Idempotency is King: Always ensure network automation scripts are completely idempotent to prevent configuration drift during high-stress restorations.
• Atomic Commits: Never apply monolithic configuration blobs; push changes modularly (Interfaces -> Routing -> Security Policies).

Common Pitfalls to Avoid

• Split-Brain Scenarios: Do not bring up BGP routing announcements at DR-2 until explicit confirmation is received that DC-1 interfaces are physically or logically severed.
• Credential Lockout: Ensure OOB SSH keys are tested semi-annually; do not rely solely on radius/TACACS+ authentication servers if directory services are down.

Metric Thresholds

• Recovery Time Objective (RTO): Core network infrastructure must be fully operational within < 2 hours of incident activation.
• Recovery Point Objective (RPO): Configuration state data loss must not exceed < 1 hour (enforced by automated hourly Git backups).

---

6. Frequently Asked Questions (FAQ)

Q1: What is the protocol if the primary automated configuration repository (tr-infra-net-config) is inaccessible during the disaster?
A: Operators must immediately drop to local operational continuity storage. Every critical engineer maintains an encrypted, air-gapped local USB drive containing the previous quarter's gold-standard configurations. Deploy these via manual console serial connections, followed by an immediate out-of-band sync once storage APIs recover.

Q2: How do we handle asymmetric routing issues when traffic starts flowing back into the DR site?
A: Asymmetric routing typically manifests when stateful firewalls drop unexpected return packets. Force an immediate flush of the connection state tables on the secondary firewall cluster using the command clear firewall state-table all, and verify that BGP local preference metrics are heavily weighted toward the active DR ingress paths.