Standard Operating Procedure: Incident Action Plan Execution for Engineering

Standard Operating Procedure: Incident Action Plan (IAP) Execution

Template Registry Engineering Operations

---

1. Document Control Block

• Document ID: SOP-OPS-IAP-042
• Effective Date: October 24, 2023
• Version: 2.1.0
• Review Cadence: Semi-Annual / Post-Major-Incident
• Classification: Internal Operations / Systems Engineering

---

2. Executive Summary & Purpose

2.1 Purpose

This Standard Operating Procedure (SOP) defines the mandatory engineering protocol for creating, authorizing, executing, and closing an Incident Action Plan (IAP) during high-severity (Sev-1/Sev-2) production disruptions within Template Registry infrastructure.

2.2 Objective

To eliminate cognitive load during critical incidents by enforcing a standardized, deterministic framework that unifies communication, delineates operational boundaries, tracks mitigation vectors, and mandates post-incident validation. Compliance with this SOP is compulsory for all engineering personnel participating in active Incident Response rotations.

---

3. Scope & Prerequisites

3.1 Scope

• In-Scope: All production environments, staging clusters replicating production states, data pipelines, and core API gateways managed by Template Registry.
• Out-Scope: Local developer environments, third-party SaaS outages outside internal dependency chains (unless cascading).

3.2 Prerequisites & Tooling Access

• Command Line Access: Production-tier bastion host access via SSH with hardware-token Multi-Factor Authentication (MFA).
• Observability Suites: Grafana Enterprise, Prometheus, Datadog APM, and AWS CloudWatch / GCP Operations Suite.
• Incident Management Platform: PagerDuty Enterprise and Jira Service Management (JSM).
• Communication Channels: Designated Slack bridge (#inc-YYYYMMDD-[slug]) and PagerDuty Conference Bridge.
• Hardware/Software PPE: Not applicable (Digital Infrastructure Operations).

---

4. Roles & Responsibilities

---

5. Step-by-Step Procedure

Phase 1: Triage and IAP Initialization

Trigger: Sev-1 or Sev-2 incident declared via PagerDuty.

• 1.1 Establish the official incident bridge and secure communication channels (Slack #inc-[slug]).
• 1.2 Formally assign the Incident Commander (IC) role; all operational directives must route through the IC.
• 1.3 Initialize the master Incident Action Plan (IAP) document using the template registry link: tpl.reg/iap-master-v2.
• 1.4 Define the Operational Period (standard default: 2 hours per tactical cycle).
• 1.5 Establish primary, secondary, and tertiary incident objectives (e.g., 1. Stop data loss, 2. Restore core API routing, 3. Verify downstream consumer recovery).

Phase 2: Tactical Assessment & Strategy Formulation

Trigger: IAP initialized and team synchronized.

• 2.1 Direct the SRE / Metrics lead to capture baseline telemetry snapshots (CPU, memory, IOPS, error rates, p99 latency).
• 2.2 Convene a 3-minute structural sync with Sub-Team Leads to identify root hypotheses.
• 2.3 Formulate the Primary Mitigation Vector (Strategy A) and Fallback Vector (Strategy B).
• 2.4 Verify blast radius metrics: determine total impacted user percentage, SLA threshold breach status, and regional boundaries.
• 2.5 Document known constraints (e.g., database locks, rate-limiting upstream API dependencies).

Phase 3: Authorization and Execution

Trigger: Mitigation strategy selected and documented in the IAP.

• 3.1 Perform peer review of the proposed execution runbook (code patch, configuration change, feature flag toggle, or infrastructure scaling).
• 3.2 Secure explicit sign-off from the Incident Commander and (if destructive data actions are required) the Engineering Director.
• 3.3 Execute mitigation via CI/CD pipeline override or direct infrastructure management interface, logging command execution strings in the incident timeline.
• 3.4 Monitor real-time telemetry dashboards for system response during and immediately following execution.
• 3.5 If mitigation fails: Immediately halt execution, revert to baseline via Strategy B, and re-convene Phase 2.

Phase 4: Validation and Stabilization

Trigger: Technical execution completed and telemetry showing recovery.

• 4.1 Run synthetic user transactions (smoke tests) against affected endpoints to verify functional integrity.
• 4.2 Monitor system metrics across two consecutive operational cycles (minimum 60 minutes) to confirm stability.
• 4.3 Formally declare system recovery to the Communications Lead for stakeholder distribution.
• 4.4 Lock the IAP document state; append final system telemetry graphs to the incident record.

Phase 5: Handoff and De-escalation

Trigger: Stabilization verified.

• 5.1 Transition incident status from "Active Mitigation" to "Monitoring / Post-Incident Review (PIR)".
• 5.2 Schedule the mandatory Blameless Post-Mortem within 48 business hours.
• 5.3 Archive the IAP document into the Template Registry Historical Audit Store.

---

6. Quality Assurance & Pro-Tips

6.1 Best Practices

• Single Source of Truth: Never deviate from the official IAP document. Side-channel decisions communicated verbally without Scribe logging are invalid.
• Timeboxing Operations: If an operational period expires without reaching the stated objective, force a strategic pivot rather than grinding through failing execution paths.
• Asynchronous Comms: Keep status updates punchy, metric-driven, and devoid of technical jargon when communicating with non-engineering stakeholders.

6.2 Common Pitfalls

• Hero Syndrome: Operating in silos without updating the IC or Scribe. Correction: Enforce strict check-ins every 15 minutes.
• Premature Declaration of Victory: Ending the incident before stability metrics hold flat for at least 30 minutes. Correction: Adhere strictly to Phase 4 timing gates.

6.3 Metric Thresholds

• Time to Initialize IAP: $\le 5\text{ minutes}$ from Sev-1 declaration.
• Operational Period Duration: $\le 120\text{ minutes}$ per tactical cycle.
• Post-Mortem Scheduling Window: $\le 48\text{ hours}$ post-resolution.

---

7. Frequently Asked Questions (FAQ)

Q1: What happens if the designated Incident Commander loses connectivity during an active incident?
A: The IC role must feature a pre-designated shadow (Deputy IC). Upon communication loss exceeding 90 seconds, the Deputy IC automatically assumes the primary IC role, announces the transition on the bridge, and logs the handoff time.

Q2: Can we bypass the IAP authorization step if we are experiencing a catastrophic security breach?
A: No. While speed is paramount during security incidents (such as data exfiltration or active compromise), executing un-reviewed isolation actions can trigger cascading system failures. Use the "Emergency Fast-Track" subsection of the IAP, which allows concurrent peer-review and execution within a 60-second window.

Q3: Who is authorized to modify an active IAP once it has been signed off?
A: Only the Incident Commander holds write permissions to alter operational objectives mid-cycle. Sub-team leads may submit change requests verbally on the bridge, which the Scribe will log as "Pending IC Approval."