It Dr Runbook

Document Control Block

• Document ID: SOP-OPS-TR-DR-4091
• Effective Date: March 30, 2026
• Version: 4.2.0
• Review Cadence: Semi-Annual
• Target Audience: Site Reliability Engineers (SRE), Disaster Recovery (DR) Incident Commanders, Database Administrators, Infrastructure Operations Architects

---

1. Executive Summary & Scope

1.1 Clinical Statement of Operational Boundaries

This Standard Operating Procedure (SOP) defines the mandatory, deterministic execution path for recovering the Template Registry core infrastructure, database clusters, and stateless container workloads during a catastrophic regional outage, infrastructure compromise, or site-level hardware failure. This document enforces operational compliance with ISO/IEC 27001, SOC 2 Type II, and NIST SP 800-34 Rev. 1 (Contingency Planning Guide for Federal Information Systems).

1.2 Operational Metrics & Tolerances

• Recovery Point Objective (RPO): $\le 15\text{ minutes}$ (Maximum tolerable data loss measured in time).
• Recovery Time Objective (RTO): $\le 60\text{ minutes}$ (Maximum tolerable time to restore full operational capability).
• High Availability (HA) Boundary: Cross-region active-passive topology with automated DNS failover utilizing AWS Route 53 health checks and continuous replication to the designated cold-standby disaster recovery region (us-west-2).

---

2. Prerequisites & Tool Requirements

Execution of this SOP requires explicit authorization via the Template Registry Emergency Incident Management System (SEIMS) and fulfillment of the following environmental dependencies:

• Access & Authentication:
  • Root access to HashiCorp Vault production cluster (vault.prod.templateregistry.internal).
  • AWS Administrator Access or cross-account Disaster Recovery Role (arn:aws:iam::998877665544:role/TemplateRegistry-DR-Commander).
  • Kubernetes administrative context for the DR cluster (kubectl config use-context dr-us-west-2-prod).
• Software & Tooling:
  • Terraform CLI $\ge 1.6.0$
  • kubectl matching the EKS control plane version ($1.28+$)
  • aws-cli/2.15.0+
  • PGP Private Key for decryption of operational vault bootstrap keys (Shamir's Secret Sharing threshold: 3 of 5).
• Hardware & Connectivity:
  • Dedicated hardware token for multi-factor authentication (YubiKey 5 Series).
  • Out-of-band (OOB) terminal access via secure SSH bastion (bastion.dr.templateregistry.internal).

---

3. Roles & Responsibilities (RACI Matrix Table)

---

4. Step-by-Step Operational Procedure

Phase 1: Incident Declaration & Initial Assessment

• 1.1 Verify primary region (us-east-1) complete telemetry loss via Datadog and PagerDuty automated alerts.
• 1.2 Convene the Emergency Incident Response Bridge via internal secure conference channel.
• 1.3 Issue formal Disaster Recovery Declaration in SEIMS, logging incident ID DR-INC-[YYYYMMDD]-[SEQ].
• 1.4 Notify Executive Leadership and compliance stakeholders via automated webhook dispatch.

Phase 2: Infrastructure Provisioning & Network Routing

• 2.1 Authenticate AWS CLI using the DR Commander cross-account IAM profile: aws sts assume-role --role-arn arn:aws:iam::998877665544:role/TemplateRegistry-DR-Commander --role-session-name DR-Execution
• 2.2 Initialize and apply the Terraform disaster recovery workspace to spin up base VPC, routing, and transit gateways in us-west-2: cd /ops/terraform/dr && terraform init && terraform apply -auto-approve
• 2.3 Verify transit gateway routing tables and inter-region peering connections are reporting healthy status codes.
• 2.4 Execute DNS failover sequence via Route 53 to divert incoming traffic from primary load balancer to the DR cluster: aws route53 change-resource-record-sets --hosted-zone-id Z123456789 --change-batch file://ops/dns/failover-to-dr.json

Phase 3: Database Restoration & State Verification

• 3.1 Locate the latest valid snapshot point-in-time recovery (PITR) identifier for the primary PostgreSQL RDS cluster in us-east-1.
• 3.2 Promote the asynchronous Read Replica in us-west-2 to a standalone primary database instance: aws rds promote-read-replica --db-instance-identifier tempreg-prod-db-dr
• 3.3 Monitor RDS promotion logs until DBInstanceStatus transitions to available.
• 3.4 Execute database integrity check script to confirm zero sequence drift: psql -h tempreg-prod-db-dr.c123456.us-west-2.rds.amazonaws.com -U admin -d registry -f /ops/sql/integrity_check.sql

Phase 4: Application Deployment & Secrets Injection

• 4.1 Update current Kubernetes context to the disaster recovery cluster: kubectl config use-context arn:aws:eks:us-west-2:998877665544:cluster/templateregistry-dr-cluster
• 4.2 Initialize HashiCorp Vault in the DR environment and unseal using the designated Shamir's Secret Sharing keys: vault operator unseal [KEY_1], vault operator unseal [KEY_2], vault operator unseal [KEY_3]
• 4.3 Apply Kubernetes manifests for core application microservices via ArgoCD automated synchronization: argocd app sync template-registry-core-dr --prune
• 4.4 Verify all Kubernetes pods reach Running and Ready status states: kubectl get pods -n template-registry-prod --watch

Phase 5: Post-Recovery Validation & Sign-off

• 5.1 Execute synthetic canary transactions against the staging API endpoint: newman run /ops/postman/dr-smoke-tests.json --environment /ops/postman/dr.env.json
• 5.2 Confirm HTTP status 200 OK responses across 100% of core API routes (/healthz, /api/v2/templates, /auth/verify).
• 5.3 Review error rate and latency percentiles via Datadog operational dashboards (Threshold: P99 latency $< 250\text{ms}$).
• 5.4 Secure formal sign-off from the Chief Operations Architect and Lead SRE to transition system status from "Disaster Recovery" to "Normal Operations - Secondary Primary".

---

5. Quality Assurance & Verification Criteria

5.1 Audit Checklist

• Database replication lag verified at $\le 0\text{ seconds}$ prior to traffic acceptance.
• TLS certificates successfully validated via AWS Certificate Manager (ACM) for all edge endpoints.
• CloudTrail logging and GuardDuty threat detection verified active in the DR region.

5.2 Tolerance Thresholds & Acceptance Benchmarks

• API Availability Benchmark: $\ge 99.95%$ successful request completion over a continuous 15-minute observation window post-failover.
• Data Consistency Threshold: Zero unresolved foreign key constraint violations reported by the post-restoration SQL verification harness.
• Network Latency Threshold: Round-trip time (RTT) from edge CDN to origin nodes $\le 45\text{ms}$.

---

6. Practical Troubleshooting Scenarios

Scenario A: Database Replica Promotion Failure / Lag Timeout

• Symptom: Step 3.2 times out, and the RDS promote command returns an error indicating replication lag exceeds the safety threshold or the replica is in a storage-optimization state.
• Immediate Corrective Action Protocol:
  1. Abort standard promotion sequence. Execute emergency manual snapshot restoration from the most recent immutable cross-region automated backup: aws rds restore-db-instance-from-db-snapshot --db-instance-identifier tempreg-prod-db-dr --db-snapshot-identifier arn:aws:rds:us-east-1:998877665544:snapshot:tempreg-prod-backup-latest
  2. Notify Lead DBA and Incident Commander immediately of potential RPO extension ($\le 30\text{ minutes}$).
  3. Once the instance transitions to available, execute a forced transaction log truncation bypass script (/ops/scripts/bypass_lag.sh) and re-run Step 3.4.

Scenario B: HashiCorp Vault Unseal Deadlock

• Symptom: Step 4.2 fails because unseal keys are rejected, or the KMS auto-unseal configuration cannot reach the AWS KMS key in us-west-2 due to IAM policy propagation delay.
• Immediate Corrective Action Protocol:
  1. Verify local AWS credentials and IAM policy attachments on the operational bastion host.
  2. If KMS auto-unseal fails, fallback to manual unseal mode. Retrieve the PGP-encrypted master keys from the secure physical offline hardware token.
  3. Execute manual unsealing protocol interactively: vault operator unseal (Repeat sequence for threshold requirement of 3 distinct key custodians).
  4. If key corruption is suspected, initiate emergency Vault disaster recovery token generation via the secondary quorum procedure documented in SOP-SEC-092.

---

End of Standard Operating Procedure — Template Registry Operations Architecture.