Template of Disaster Recovery Plan

STANDARD OPERATING PROCEDURE: Enterprise Disaster Recovery Execution

Document ID: SOP-TR-DR-042
Effective Date: October 24, 2023
Version: 3.2.0
Review Cadence: Semi-Annual
Author: Julian Vance, Chief Architect, Template Registry

---

1. Document Control & Metadata

---

2. Executive Summary & Purpose

This Standard Operating Procedure (SOP) defines the mandatory, deterministic workflow for executing disaster recovery (DR) protocols across Template Registry infrastructure. The objective is to ensure minimal Service Interruption, strict adherence to Recovery Point Objectives (RPO $\le$ 15 minutes), and Recovery Time Objectives (RTO $\le$ 60 minutes) during a catastrophic system failure, regional cloud outage, or targeted cyber incident. All operations must strictly follow the immutable sequence outlined herein.

---

3. Scope & Prerequisites

3.1 Scope

This SOP applies to all production environments, containerized orchestration layers, core databases, and object-storage repositories managed by Template Registry.

3.2 Prerequisites & Required Access

• Identity & Access Management: Multi-Factor Authentication (MFA) enabled hardware token with global root or break-glass administrative privileges.
• Tooling Suite:
  • Terraform v1.5+ (Infrastructure as Code)
  • AWS CLI v2.13+ configured with secondary region failover profiles
  • Kubernetes CLI (kubectl) with cluster-admin access
  • HashiCorp Vault CLI (for secret unsealing)
• Communications: Out-of-band PagerDuty responder channel, dedicated Bridge Line Alpha, and Statuspage.io administrative access.

---

4. Roles & Responsibilities (RACI Matrix)

(Legend: R = Responsible, A = Accountable, C = Consulted, I = Informed)

---

5. Step-by-Step Procedure

Phase 1: Triage, Assessment, and Disaster Declaration

• 1.1 Verify telemetry alerts indicating an unrecoverable primary region failure or systemic data corruption via the primary Grafana dashboard (https://monitoring.templateregistry.internal/d/dr-triage).
• 1.2 Convene the Emergency Response Team on Bridge Line Alpha.
• 1.3 Conduct quorum validation with the Incident Commander and Lead Systems Engineer to officially declare a Severity-1 Disaster Event.
• 1.4 Update the public/internal status page to reflect "Major Outage - Failover Protocol Initiated" via automated script:

  tr-status-cli set-incident --severity=SEV-1 --template=dr-failover-init
  

• 1.5 Isolate and revoke compromised IAM credentials if the disaster is suspected to be a security breach.

Phase 2: Secondary Infrastructure Provisioning

• 2.1 Authenticate to the designated secondary disaster recovery region (e.g., us-west-2) using administrative credentials.
• 2.2 Initialize the immutable infrastructure state using Terraform:

  cd /opt/template-registry/infrastructure/dr-failover/
  terraform init -backend-config="region=us-west-2"
  

• 2.3 Execute the infrastructure deployment plan to provision core compute, networking, and storage layers:

  terraform apply -target=module.core_infrastructure -auto-approve
  

• 2.4 Verify VPC peering, security group rules, and Kubernetes control plane availability:

  kubectl get nodes --context=dr-secondary-context
  

Phase 3: Data Layer Restoration

• 3.1 Locate the latest cryptographically verified snapshot in the cross-region backup bucket (s3://tr-production-backups-replica/).
• 3.2 Execute the database restoration script to spin up the primary PostgreSQL RDS instance from the point-in-time recovery (PITR) manifest:

  python3 /opt/tools/restore_db.py --region=us-west-2 --snapshot-id=latest-verified
  

• 3.3 Validate database schema integrity and transaction logs up to the failure threshold:

  SELECT pg_is_in_recovery(), pg_last_wal_receive_lsn();
  

• 3.4 Mount and synchronize the persistent object store buckets using the incremental replication tool.
• 3.5 Unseal and inject production secrets into the secondary HashiCorp Vault cluster using emergency recovery keys.

Phase 4: Traffic Cutover & Synthetic Validation

• 4.1 Update Route53 DNS weighted routing policies or Global Accelerator endpoints to shift 100% of ingress traffic from the primary region to the secondary DR region:

  aws route53 change-resource-record-sets --hosted-zone-id Z123456789 --change-batch file://dns-failover.json
  

• 4.2 Execute the automated post-failover synthetic test suite to validate API endpoints, template rendering engines, and user authentication flows:

  pytest /opt/template-registry/tests/dr/failover_validation.py --env=production-dr
  

• 4.3 Confirm application error rates (HTTP 5xx) drop below the threshold of 0.01% for a sustained 5-minute window.
• 4.4 Update the status page to "System Operational (Running in DR Region)."

---

6. Quality Assurance, Pro-Tips, and Thresholds

6.1 Critical Metrics & Thresholds

• RPO Threshold: $\le$ 15 minutes of data loss maximum.
• RTO Threshold: $\le$ 60 minutes from declaration to traffic restoration.
• Error Rate Threshold: Must remain below 0.1% post-cutover.

6.2 Pro-Tips & Operational Best Practices

• Immutable Backups: Ensure cross-region backup buckets enforce S3 Object Lock in compliance mode to prevent ransomware propagation during a DR event.
• Terraform State Locking: Always verify that stale state locks from the failed primary region are manually cleared using terraform force-unlock before applying configurations in the secondary region.

6.3 Common Pitfalls to Avoid

• Pitfall: Forgetting to unseal HashiCorp Vault in the secondary region prior to application boot.
  • Mitigation: Integrate Vault unsealing into Phase 3 initialization scripts using cloud-KMS auto-unseal configurations.
• Pitfall: TTL caching issues causing clients to query the dead primary IP addresses.
  • Mitigation: Pre-emptively lower DNS TTL values to 60 seconds 24 hours prior to scheduled maintenance, or rely on AWS Global Accelerator health-check-based routing.

---

7. Frequently Asked Questions (FAQ)

Q1: What happens if the secondary region experiences a partial capacity shortage during failover?
A: The Terraform auto-scaling configurations in module.core_infrastructure are pre-configured with spot-fallback and multi-AZ instance diversification. If capacity is constrained, immediately execute the high-priority compute expansion script (/opt/tools/scale_compute_emergency.sh) which bypasses non-critical batch processing nodes to reserve resources strictly for API and Database tiers.

Q2: How do we handle split-brain scenarios if the primary region suddenly comes back online mid-failover?
A: Never attempt to blindly sync a revived primary region back into the active pool without running the isolation routine. The revived primary region must be fenced off using security group overrides, mounted as a read-only historical replica, and audited for state divergence before any reverse-replication or failback procedure is initiated.

Q3: Who authorizes the public communication updates during a P1 Disaster Recovery event?
A: The Incident Commander (IC) holds sole authority to authorize external communications. However, pre-approved template statements located in the /docs/incident-templates/ directory may be populated and queued by the Communications Lead while awaiting final IC sign-off.