Disaster Recovery Plan Example

STANDARD OPERATING PROCEDURE: Enterprise Disaster Recovery Execution

Document ID: SOP-TR-DR-042
Effective Date: October 24, 2023
Version: 4.2.0
Review Cadence: Semi-Annual
Author: Julian Vance, Chief Architect, Template Registry

---

1. Executive Summary & Purpose

This Standard Operating Procedure (SOP) defines the mandatory protocol for executing a complete disaster recovery (DR) failover and restoration of the Template Registry core infrastructure. The purpose of this document is to ensure business continuity, minimize Recovery Point Objective (RPO < 15 minutes), and achieve Recovery Time Objective (RTO < 60 minutes) during a catastrophic infrastructure failure or localized region outage.

---

2. Scope & Prerequisites

2.1 Scope

This procedure applies to all production Kubernetes clusters, managed database instances (PostgreSQL primary/replica topologies), object storage buckets, and API gateway routes residing within the primary cloud region (us-east-1) and targeting failover to the secondary disaster recovery region (us-west-2).

2.2 Prerequisites & Tooling

• Access Requirements: Root administrative access to AWS/GCP IAM, HashiCorp Vault, and Cloudflare DNS management plane.
• Local Tooling Required:
  • kubectl (v1.28+)
  • terraform (v1.5+)
  • helm (v3.12+)
  • aws-cli (v2.13+)
• Physical/Logical Security: Multi-factor authentication (MFA) hardware token and authorized break-glass PGP key.

---

3. Roles & Responsibilities (RACI Matrix)

• Definitions: Responsible (does the work), Accountable (has final approval), Consulted (provides input), Informed (kept updated).

---

4. Step-by-Step Procedure

Phase 1: Incident Verification & Declaration

• 1.1 Convene the emergency bridge via PagerDuty priority-1 escalation channel.
• 1.2 Validate primary region outage utilizing automated synthetic monitoring telemetry (Datadog/Prometheus metrics).
• 1.3 Obtain formal sign-off from the Incident Commander to execute Disaster Recovery Protocol SOP-TR-DR-042.
• 1.4 Broadcast operational status update via internal Slack #incident-command and external status page (status.templateregistry.internal).

Phase 2: Infrastructure Provisioning in DR Region (us-west-2)

• 2.1 Authenticate against the cloud provider control plane using break-glass IAM credentials.
• 2.2 Initialize Terraform workspace for the secondary region:

  terraform workspace select dr-us-west-2
  terraform init
  

• 2.3 Execute infrastructure deployment plan to provision foundational networking, compute nodes, and managed storage:

  terraform apply -auto-approve -target=module.foundation
  

• 2.4 Verify cluster readiness and node health:

  kubectl get nodes --context=dr-us-west-2-cluster
  

Phase 3: Database Failover & Point-in-Time Recovery

• 3.1 Promote the asynchronous read replica in the secondary region to primary status:

  aws rds promote-read-replica --db-instance-identifier template-registry-db-dr
  

• 3.2 Poll the database status until the instance state transitions to available:

  aws rds describe-db-instances --db-instance-identifier template-registry-db-dr --query "DBInstances[0].DBInstanceStatus"
  

• 3.3 Execute the database migration validation script to ensure schema integrity and index synchronization:

  python3 scripts/validate_db_sync.py --target=dr-primary
  

Phase 4: Application State & Secrets Hydration

• 4.1 Unseal HashiCorp Vault in the DR region using distributed Shamir's Secret Sharing keys:

  vault operator unseal <key-share-1>
  vault operator unseal <key-share-2>
  vault operator unseal <key-share-3>
  

• 4.2 Deploy core application helm charts with production overrides:

  helm upgrade --install template-registry-core ./charts/core \
    --namespace production \
    --values ./charts/core/values-dr.yaml \
    --wait
  

• 4.3 Verify application pod health and liveness/readiness probes:

  kubectl get pods -n production --context=dr-us-west-2-cluster
  

Phase 5: Traffic Routing & DNS Cutover

• 5.1 Update Cloudflare DNS records to route external traffic from the failed primary region load balancer to the secondary region entry point:

  cloudflare-cli dns update --zone=templateregistry.io --record=api --content=dr-lb.templateregistry.io --proxied=true
  

• 5.2 Purge Cloudflare edge cache to prevent stale payload delivery:

  cloudflare-cli cache purge --zone=templateregistry.io --everything
  

• 5.3 Verify global routing propagation using dig and ensure edge latency is within nominal bounds (< 150ms globally).

---

5. Quality Assurance & Pro-Tips

5.1 Best Practices

• Immutable Infrastructure: Never patch running DR instances manually; always rely on Terraform state definitions and version-controlled Helm charts.
• Regular Simulation: Conduct dry-run failovers in a staging environment bi-annually to validate RTO/RPO metrics against real-world drift.

5.2 Common Pitfalls

• Split-Brain Scenarios: Ensure the primary region is completely fenced/isolated via network security groups before promoting the database replica to prevent write conflicts.
• Secret Drift: Ensure Vault auto-unseal mechanisms or cloud KMS policies are regularly tested in the DR region; expired TLS certificates will halt application bootstrap.

5.3 Metric Thresholds

• Maximum Allowable RPO: 15 minutes of data loss.
• Maximum Allowable RTO: 60 minutes from incident declaration to DNS routing cutover.
• Error Rate Threshold: < 0.01% HTTP 5xx responses post-cutover during the 30-minute stabilization window.

---

6. Frequently Asked Questions (FAQ)

Q1: What happens if the database read replica lag exceeds the 15-minute RPO threshold during promotion?
A: If replication lag exceeds 15 minutes, abort the standard promotion workflow. Execute point-in-time recovery (PITR) from the latest immutable S3 WAL archive snapshot taken in the primary region. Notify the Incident Commander immediately, as RTO will extend by approximately 20 minutes.

Q2: How do we handle webhook callbacks and external API integrations pointing to the old region during DNS propagation?
A: External webhooks originating from partners are buffered via our AWS SQS dead-letter and holding queues at the edge routing layer. Once DNS propagation completes (typically 2–5 minutes), the queuing layer automatically drains payloads to the new cluster endpoints without data loss.