SAAS Disaster Recovery Plan Template

STANDARD OPERATING PROCEDURE: SaaS Disaster Recovery Plan Execution

Document ID: SOP-TR-DR-402
Effective Date: October 24, 2023
Version: 3.4.0
Review Cadence: Semi-Annual (Every 6 Months)
Author: Julian Vance, Chief Architect, Template Registry

---

1. Executive Summary & Purpose

This Standard Operating Procedure (SOP) defines the mandatory protocols for declaring, executing, and recovering from a critical infrastructure outage or catastrophic data loss event within the Template Registry SaaS ecosystem. The purpose of this document is to establish a deterministic, repeatable framework that minimizes Recovery Point Objective (RPO) and Recovery Time Objective (RTO) deviations during a Disaster Recovery (DR) event. Adherence to this SOP ensures platform availability, data integrity restoration, and systematic operational resumption in alignment with enterprise Service Level Agreements (SLAs).

---

2. Scope & Prerequisites

2.1 Scope

This procedure applies to all production environments, multi-tenant databases, Kubernetes control planes, object storage buckets, and edge routing layers managed by Template Registry infrastructure teams across primary and secondary regions.

2.2 Prerequisites & Tooling Access

Operators executing this SOP must possess active, authenticated access to the following systems:

• Infrastructure Access: AWS IAM / GCP IAM with AdministratorAccess or pre-approved DisasterRecovery-Exec custom role.
• Version Control & Pipeline: GitHub Enterprise (Template Registry org) with PAT scopes for deployment triggers.
• Observability Stack: Datadog Enterprise, PagerDuty (Global Admin), and Grafana Enterprise dashboards.
• Secrets Management: HashiCorp Vault (Production cluster) with unseal key shares or AWS Secrets Manager.
• Communication: PagerDuty Incident Command Bridge and the #sec-incident-command Slack channel.

---

3. Roles & Responsibilities (RACI Matrix)

Legend: R = Responsible, A = Accountable, C = Consulted, I = Informed

---

4. Step-by-Step Procedure

Phase 1: Incident Assessment & DR Declaration

• 1.1 Verify PagerDuty automated alerts or manual severity 1 (Sev-1) escalation triggers indicating primary region unavailability or unrecoverable state corruption.
• 1.2 Convene the Emergency Response Team (ERT) via the PagerDuty Bridge within 5 minutes of alert generation.
• 1.3 Assess telemetry data via Grafana primary dashboards to confirm threshold breaches: Error rate > 5% for 3 consecutive minutes, or complete ingress loss.
• 1.4 Formally declare a Disaster Recovery event by executing the declaration script:

  ./scripts/dr-declare.sh --region us-east-1 --reason "infrastructure-total-failure" --auth-token $VAULT_TOKEN
  

• 1.5 Post status update to the internal #sec-incident-command channel and transition status page to "Major Outage - Failover Initiated".

Phase 2: Traffic Redirection & Edge Isolation

• 2.1 Isolate the compromised primary region by updating Cloudflare/Route53 DNS weight configurations to 0.
• 2.2 Verify global DNS propagation metrics to ensure edge caching nodes reject incoming requests to the primary cluster.
• 2.3 Route 100% of incoming global ingress traffic to the secondary (warm standby) region (us-west-2).
• 2.4 Validate edge termination rules, SSL/TLS certificates, and Web Application Firewall (WAF) rule sets on the secondary gateway.

Phase 3: Database & State Restoration

• 3.1 Verify the replication lag status of the target Read Replica in the secondary region prior to promotion.
• 3.2 Execute database promotion script to convert the secondary read replica into the primary write instance:

  aws rds promote-read-replica --db-instance-identifier template-registry-db-replica-west
  

• 3.3 Confirm database write availability by querying the health check endpoint:

  curl -s https://api.secondary.templateregistry.io/health/db | jq .status
  

• 3.4 Mount secondary regional S3 bucket replicas for user-uploaded assets and verify object versioning parity.
• 3.5 Inject required production environment secrets from HashiCorp Vault into the secondary Kubernetes cluster namespaces:

  vault kv get -format=json secret/prod/config | kubectl apply -f -
  

Phase 4: Compute Cluster & Application Spin-Up

• 4.1 Scale up the secondary regional Kubernetes (EKS) cluster node groups to handle full production load:

  eksctl scale nodegroup --cluster=tr-prod-west --name=app-nodes --nodes=25 --min=10 --max=50
  

• 4.2 Apply Helm charts to deploy core microservices (Auth, Template Engine, Registry API, Billing):

  helm upgrade --install tr-services ./charts/production --namespace production --wait
  

• 4.3 Verify pod health status and ensure zero CrashLoopBackOff states:

  kubectl get pods -n production --field-selector=status.phase!=Running
  

Phase 5: Verification & Post-Recovery Validation

• 5.1 Execute synthetic monitoring scripts against the secondary environment to validate end-to-end user workflows (Auth -> Read Template -> Write Template -> Checkout):

  npm run test:synthetic --env=dr-failover
  

• 5.2 Validate RPO/RTO metrics against established SLAs (Target RPO < 15 minutes, Target RTO < 60 minutes).
• 5.3 Issue an all-clear notification to the Executive Leadership Team and update the public status page to "Operational".
• 5.4 Schedule the Post-Incident Review (PIR) meeting for 48 hours post-resolution.

---

6. Quality Assurance & Pro-Tips

Best Practices

• Immutable Infrastructure: Never hot-patch configurations during a failover; always rely on declared state in Terraform/Helm.
• Readiness Probes: Maintain automated daily testing of database replication pipelines to ensure sub-minute recovery windows.

Common Pitfalls

• Split-Brain Scenarios: Failing to isolate the primary region before promoting the secondary database can lead to data corruption. Always execute DNS/Ingress isolation first (Phase 2).
• Secret Drift: Forgetting to sync regional Vault instances results in immediate application failure during cluster spin-up.

Metric Thresholds

• Recovery Point Objective (RPO): $\le 15 \text{ minutes}$ maximum data loss window.
• Recovery Time Objective (RTO): $\le 60 \text{ minutes}$ elapsed time from declaration to full traffic restoration.

---

7. Frequently Asked Questions

Q: What happens if the secondary database replica is out of sync beyond the RPO threshold?
A: If replication lag exceeds 15 minutes and the primary region is unrecoverable, the DBA must perform a Point-in-Time Recovery (PITR) using the latest valid transaction log backup stored in S3, accepting data loss up to the snapshot timestamp while notifying legal/compliance teams.

Q: Can we perform a partial failover (e.g., API services only, excluding billing)?
A: No. Partial failovers violate transactional consistency across the microservice boundary. The Template Registry architecture requires an all-or-nothing regional failover to maintain data integrity.

Q: Who possesses the authority to abort a DR drill or live failover?
A: Only the Incident Commander (IC) in direct consensus with the Chief Architect (Julian Vance) or CISO may abort a failover execution sequence.