Critical Incident Response Plan Template

Standard Operating Procedure: Critical Incident Response Plan (CIRP)

---

1. Executive Summary & Purpose

This Standard Operating Procedure (SOP) defines the mandatory protocol for the identification, containment, eradication, and post-incident review of critical infrastructure events at Template Registry. The purpose of this document is to establish a deterministic, repeatable framework that minimizes Mean Time to Detection (MTTD) and Mean Time to Resolution (MTTR), preserves forensic integrity, and ensures systematic communication across engineering, executive leadership, and external stakeholders during Sev-1 (Critical) incidents.

---

2. Scope & Prerequisites

2.1 Scope

This SOP applies to all production environments, staging clusters, identity providers, core databases, and supporting continuous integration/continuous deployment (CI/CD) pipelines managed by Template Registry engineering teams.

2.2 Prerequisites & Required Tooling

• Access Level: PagerDuty Admin, AWS/GCP Root/Organization Administrator, GitHub Enterprise Owner, Kubernetes Cluster Admin (cluster-admin).
• Communication Channels: Slack (#sec-incident-room, #eng-ops-command), PagerDuty Mobile App, Enterprise Bridge Line (Zoom/Webex active 24/7).
• Forensic Tooling: Datadog APM, Prometheus/Grafana, Wireshark, AWS CloudTrail, Falco runtime security.

---

3. Roles & Responsibilities (RACI Matrix)

Legend: R = Responsible, A = Accountable, C = Consulted, I = Informed.

---

4. Step-by-Step Procedure

Phase 1: Detection & Triage

• Acknowledge incoming PagerDuty Sev-1 alert within 3 minutes of emission.
• Spin up the designated incident bridge via PagerDuty automated conference bridge.
• Declare the incident severity level (Sev-1: Complete service outage; Sev-2: Degradation of core features).
• Assign the role of Incident Commander (IC) to the most senior engineer available if not pre-assigned.
• Open the #inc-[YYYYMMDD]-[short-name] Slack war room and pin the PagerDuty and Zoom links.

Phase 2: Containment & Isolation

• Execute automated circuit breakers or traffic-shedding scripts if DDoS or cascading failures are detected.
• Isolate compromised Kubernetes nodes or AWS EC2 instances using security group quarantine rules:

  aws ec2 modify-instance-attribute --instance-id i-0123456789abcdef0 --groups sg-quarantine-lockdown
  

• Revoke compromised IAM credentials, API keys, and service account tokens immediately.
• Verify that external edge routing (Cloudflare/AWS CloudFront) is redirecting traffic to maintenance landing pages if total isolation is required.

Phase 3: Eradication & Remediation

• Identify root cause via Datadog APM tracing, log analysis (Elasticsearch/CloudWatch), and memory dump analysis.
• Apply hotfixes, patch vulnerabilities, or execute rollback procedures to the last known good state (LKG):

  kubectl rollout undo deployment/template-registry-core --namespace=production
  

• Run integrity checks on primary relational databases and object stores (S3/GCS buckets).
• Validate system metrics (CPU utilization, error rates < 0.01%, p99 latency) for 15 consecutive minutes post-patch.

Phase 4: Recovery & Closure

• Scale production replica sets back to normal operating capacity.
• Officially declare the incident resolved via the Incident Commander on the bridge and Slack.
• Update the public/internal status page to reflect 100% operational status.
• Schedule the mandatory Blameless Post-Mortem meeting within 48 hours of resolution.

---

5. Quality Assurance & Pro-Tips

5.1 Best Practices

• Single Source of Truth: The Incident Commander is the sole authority on public communication and technical prioritization during the active phase.
• Log Preservation: Never terminate compromised instances without first snapshotting EBS volumes and capturing memory dumps for forensic analysis.

5.2 Common Pitfalls to Avoid

• The "Hero" Anti-Pattern: Engineers making undocumented manual configuration changes directly on production servers without peer review.
• Communication Silos: Failing to update the Communications Lead every 30 minutes during prolonged outages.

5.3 Metric Thresholds

• MTTD (Mean Time to Detection): $\le 3$ minutes.
• MTTR (Mean Time to Resolution): $\le 45$ minutes for Sev-1 infrastructure events.

---

6. Frequently Asked Questions

Q: What constitutes an automatic escalation from a Sev-2 to a Sev-1 incident?
A: Any event resulting in complete data loss, unauthorized access to Personally Identifiable Information (PII), or an unmitigated core API downtime exceeding 15 minutes affecting greater than 20% of enterprise tenants automatically triggers Sev-1 protocols.

Q: Who is authorized to speak with external media or customers during an active incident?
A: Only the designated Communications Lead or Executive Sponsor may release statements externally. Engineering resources must direct all external inquiries to the #pr-incident-response channel.