Disaster Recovery Plan Template Azure

Standard Operating Procedure: Azure Disaster Recovery Plan (DRP) Execution and Governance

================================================================================
DOCUMENT CONTROL BLOCK
--------------------------------------------------------------------------------
Document ID:     SOP-TR-ARC-AZ-042
Effective Date:  October 24, 2023
Version:         3.1.0
Review Cadence:  Semi-Annually (Next Review: April 2024)
Owner:           Julian Vance, Chief Architect, Template Registry
Classification:  Confidential - Internal Operations Only
================================================================================

---

1. Executive Summary & Purpose

This Standard Operating Procedure (SOP) defines the institutional framework and technical execution pathway for Disaster Recovery (DR) operations within Template Registry's Microsoft Azure infrastructure. The purpose of this document is to establish a deterministic, verifiable protocol for recovering mission-critical workloads, stateful data stores, and global routing mechanisms in the event of a regional Azure datacenter failure, catastrophic data corruption, or severe security breach.

Adherence to this SOP ensures compliance with Enterprise Recovery Point Objective (RPO) thresholds of $\le 15$ minutes and Recovery Time Objective (RTO) thresholds of $\le 60$ minutes across Tier-0 assets.

---

2. Scope & Prerequisites

Scope

This procedure applies to all cloud infrastructure, Platform-as-a-Service (PaaS) deployments, and Infrastructure-as-a-Service (IaaS) virtual machines hosted within Template Registry Azure subscriptions across Primary (e.g., East US 2) and Secondary/Recovery (e.g., Central US) paired regions.

Prerequisites & Required Access

• Identity & Access Management: Azure Active Directory (Entra ID) Global Administrator or Privileged Role Administrator role.
• Role-Based Access Control (RBAC): Contributor or User Access Administrator on target Resource Groups and Recovery Services Vaults.
• Tooling:
  • Azure CLI (v2.50.0+)
  • Azure PowerShell Module (Az v10.0+)
  • Terraform v1.5+ (for infrastructure state reconstruction)
• Physical/Network Prerequisites: Out-of-band operational workstation with zero dependency on the impaired primary Azure region; active MFA token registered via hardware token or alternate secure enclave.

---

3. Roles & Responsibilities

The execution of this DRP follows a strict RACI governance model.

---

4. Step-by-Step Procedure

Phase 1: Incident Assessment & Declaration

• 1.1 Confirm primary region outage via Azure Service Health API, internal synthetic transaction monitors, and external monitoring telemetry (Datadog/PagerDuty).
• 1.2 Convene emergency bridge via out-of-band communication channel (Signal/Teams Out-of-Band).
• 1.3 Incident Commander formally declares a Disaster Recovery event, shifting posture to Recovery Operations.
• 1.4 Notify Executive Stakeholders via automated status-page webhook escalation.

Phase 2: Global Traffic Manager / Front Door Failover

• 2.1 Access Azure Portal or execute CLI to inspect Azure Front Door / Traffic Manager routing profile status.
• 2.2 Force manual failover of endpoint priorities if automatic health probes have failed to reroute traffic:

  az network frontdoor endpoint disable --resource-group rg-networking-prod \
    --front-door-name tr-global-fd --endpoint-name primary-endpoint
  

• 2.3 Verify global DNS propagation and ensure incoming ingress traffic is successfully draining toward the secondary paired region (Central US).

Phase 3: Secondary Region Infrastructure Bring-Up (IaC)

• 3.1 Authenticate to Azure CLI using service principal with deployment rights to the secondary region:

  az login --service-principal -u $AZURE_SP_CLIENT_ID -p $AZURE_SP_SECRET --tenant $AZURE_TENANT_ID
  

• 3.2 Initialize and apply Terraform state targeting the secondary region deployment workspace:

  terraform workspace select disaster-recovery-centralus
  terraform apply -auto-approve -var-file="config/prod.centralus.tfvars"
  

• 3.3 Validate core networking topology (Virtual Networks, Subnets, User-Defined Routes, and Network Security Groups) are provisioned and operational in the recovery region.

Phase 4: Stateful Data Restoration & Failover

• 4.1 Azure SQL / Cosmos DB Failover: Initiate forced geo-failover for primary transactional databases:

  az sql db replica failover --resource-group rg-data-prod \
    --server tr-sql-secondary --name TemplateRegistryDB --allow-data-loss false
  

• 4.2 Azure Storage Accounts: Promote Read-Access Geo-Redundant Storage (RA-GRS) accounts to primary status:

  az storage account failover --name trstorageprod --resource-group rg-storage-prod --yes
  

• 4.3 Verify data integrity and consistency by executing read-write validation queries against the promoted endpoints.

Phase 5: Compute Workload & PaaS Synchronization

• 5.1 Trigger Azure Site Recovery (ASR) orchestration to power on replicated Virtual Machines in the recovery region:

  az site-recovery fabric list --resource-group rg-recovery-vault
  az site-recovery protection-container wakeup --name primary-container \
    --fabric-name primary-fabric --resource-group rg-recovery-vault
  

• 5.2 Validate AKS (Azure Kubernetes Service) cluster state in the secondary region; apply deployment manifests via GitOps (ArgoCD/Flux):

  az aks get-credentials --resource-group rg-aks-dr --name tr-aks-centralus-cluster
  kubectl apply -k ./deploy/overlays/dr-centralus
  

Phase 6: Validation, Smoke Testing, & Sign-Off

• 6.1 Execute automated synthetic test suite against the recovery endpoints to validate HTTP 200 responses and API latency thresholds ($\le 250\text{ms}$).
• 6.2 Verify database connection pools, external webhook integrations, and authentication token issuance (OAuth2/OIDC).
• 6.3 Chief Architect (Julian Vance) or designated Incident Commander reviews operational metrics and signs off on full operational resumption.

---

5. Quality Assurance & Pro-Tips

Best Practices

• Immutable State: Keep Terraform state files and deployment secrets strictly version-controlled in an independent, geo-replicated backing store outside the primary execution path.
• Continuous Validation: Perform automated, non-disruptive regional failover drills quarterly during scheduled maintenance windows.

Common Pitfalls to Avoid

• Hardcoded Endpoints: Never hardcode regional IP addresses or specific regional DNS records in application configurations; always utilize Azure Traffic Manager or Front Door DNS aliases.
• Ignoring Quota Limits: Ensure regional vCPU and IP address quotas are pre-allocated in the secondary region via Azure Support requests to prevent deployment throttling during an active incident.

Metric Thresholds

• Recovery Point Objective (RPO): $\le 15\text{ minutes}$
• Recovery Time Objective (RTO): $\le 60\text{ minutes}$
• Synthetic Test Success Rate: $100%$ pass rate across Tier-0 health probes before user traffic restoration.

---

6. Frequently Asked Questions (FAQ)

Q1: What happens if the secondary paired region also experiences degradation during a primary outage?
A: In the unlikely event of a dual-region outage, the Incident Commander must escalate immediately to Tier-3 support via the Microsoft Enterprise Agreement (EA) critical incident bridge. Concurrently, evaluate the deployment of fallback static degradation pages via edge-CDN layers (e.g., Cloudflare/Akamai) to preserve user-facing status transparency while failing over to an alternate non-paired region (e.g., West US 3) via pre-staged Terraform configurations.

Q2: How do we handle database split-brain scenarios post-failover?
A: Azure platform-managed geo-replication natively prevents split-brain scenarios by revoking write access on the original primary instance prior to promoting the secondary instance. If a manual override was executed using --allow-data-loss true, the original primary database must be completely re-seeded via point-in-time restore from the new primary instance before reintegrating it back into the replication topology.

Q3: Who authorizes the reversion back to the primary region once it is recovered?
A: Reversion (failback) operations require explicit joint authorization from the Chief Architect (Julian Vance) and the CISO. Failbacks are strictly scheduled during designated low-traffic maintenance windows to mitigate the risk of cascading transactional conflicts or downstream cache corruption.