Incident Response Plan Document Example

Standard Operating Procedure: Enterprise Incident Response Plan (IRP)

Document ID: SOP-SEC-IRP-042
Effective Date: October 24, 2023
Version: 3.1.0
Review Cadence: Semi-Annual

---

1. Document Control Block

---

2. Executive Summary & Purpose

This Standard Operating Procedure (SOP) defines the mandatory, institutional-grade lifecycle management protocol for identifying, containing, eradicating, and recovering from high-severity security and infrastructure incidents within the Template Registry production environments.

The primary objective is to minimize Mean Time to Detection (MTTD) and Mean Time to Resolution (MTTR), preserve evidentiary chain-of-custody for forensic analysis, and ensure absolute operational continuity while maintaining regulatory compliance (SOC2, ISO 27001, GDPR).

---

3. Scope & Prerequisites

3.1 Scope

• Applies to all cloud-native infrastructure (AWS/GCP), containerized clusters (Kubernetes/EKS), internal application microservices, database tiers, and CI/CD pipelines managed by Template Registry.
• Covers Severity 1 (Critical) through Severity 3 (Low) operational and security events.

3.2 Prerequisites & Required Tooling

• Access Credentials: Privileged AWS/GCP IAM roles, multi-factor authentication (MFA) enabled, HashiCorp Vault administrative tokens.
• Toolchain:
  • SIEM / Log Aggregation: Datadog / Splunk Enterprise.
  • Incident Orchestration: PagerDuty & Jira Service Management (JSM).
  • Forensics & Containment: AWS Systems Manager (SSM), Kubectl CLI, Wireshark, Volatility Framework.
• Communication Channels: Out-of-band encrypted comms (Signal / Matrix) and dedicated PagerDuty bridge lines.

---

4. Roles & Responsibilities (RACI Matrix)

Legend: R = Responsible, A = Accountable, C = Consulted, I = Informed.

---

5. Step-by-Step Procedure

Phase 1: Identification & Triage (Sev-1 / Sev-2 Declaration)

• Monitor incoming alerts via automated PagerDuty triggers or manual SecOps escalations.
• Classify the incident severity using the matrix below:
  • Sev-1 (Critical): Active data exfiltration, root compromise, ransomware, total system outage.
  • Sev-2 (High): Isolated service degradation, unexploited vulnerability with active exploit in the wild.
  • Sev-3 (Moderate): Non-critical anomaly, suspicious login attempts without lateral movement.
• Initialize the dedicated incident war room (Zoom/Bridge) and secondary out-of-band Slack/Signal channel (#inc-YYYYMMDD-[name]).
• Designate the Incident Commander (IC) to lead the response lifecycle.

Phase 2: Containment (Isolation & Preservation)

• Execute immediate short-term containment to halt lateral movement or active exploitation without destroying forensic artifacts.
• Network Isolation: Revoke compromised IAM keys, rotate Kubernetes service account tokens, and update AWS Security Groups to isolate affected EC2 instances/pods:

  aws ec2 modify-instance-attribute --instance-id i-0123456789abcdef0 --groups sg-isolated
  

• Memory & Disk Preservation: Capture full memory dumps and snapshot EBS volumes of affected nodes before applying remediation patches:

  aws ec2 create-snapshot --volume-id vol-0123456789abcdef0 --description "Forensic snapshot for Incident INC-9921"
  

• Verify containment effectiveness by reviewing Datadog network telemetry and SIEM anomaly alerts.

Phase 3: Eradication (Root Cause Elimination)

• Identify the attack vector, entry point, and persistence mechanisms utilized (e.g., backdoors, cron jobs, rogue IAM policies).
• Purge malicious artifacts, scripts, and compromised user credentials across all identity providers (Okta/Active Directory).
• Rebuild affected immutable infrastructure components from clean, cryptographically verified golden AMIs or Kubernetes manifests.
• Apply emergency infrastructure patches or update Web Application Firewall (WAF) rules to block known exploit vectors.

Phase 4: Recovery & Validation

• Restore services systematically, prioritizing core Template Registry APIs and database persistence layers.
• Execute automated integration and smoke test suites to validate data integrity and operational performance:

  pytest tests/integration/ --env=production --smoke-test
  

• Monitor system error rates, latency metrics, and CPU/Memory utilization profiles for a mandatory 2-hour stabilization window.
• Formally declare the incident resolved and notify internal stakeholders via status page updates.

Phase 5: Post-Incident Review (PIR)

• Schedule the mandatory PIR meeting within 48 business hours of incident closure.
• Compile the timeline of events, system telemetry, and remediation actions into the Jira PIR ticket.
• Assign action items with strict deadlines to prevent recurrence of the root vulnerability.

---

6. Quality Assurance & Pro-Tips

Best Practices (Pro-Tips)

• Preserve Evidence First: Never power down a live compromised host; always capture volatile RAM and disk states first to aid forensic investigators.
• Single Source of Truth: Keep the Jira incident ticket and PagerDuty notes explicitly updated; avoid fragmented updates in ad-hoc direct messages.
• Blameless Culture: Focus engineering reviews strictly on procedural, architectural, and tooling weaknesses rather than human error.

Common Pitfalls to Avoid

• Premature Remediation: Deleting logs or terminating instances too early, which permanently destroys forensic chains of custody.
• Communication Silos: Failing to notify Legal/Compliance early when PII or financial data exposure is suspected.

Quantitative Thresholds & SLAs

• MTTD (Mean Time to Detection): < 5 minutes for automated Sev-1 alerts.
• Initial Response SLA: < 15 minutes acknowledgment by the on-call SRE/SecOps rotation.
• PIR Completion Window: < 48 hours post-resolution.

---

7. Frequently Asked Questions (FAQ)

Q: What triggers an immediate escalation to Executive Leadership and Legal?
A: Any confirmed or highly probable compromise involving customer Personally Identifiable Information (PII), intellectual property exfiltration, ransomware execution, or sustained downtime exceeding 60 minutes on core production tiers requires immediate notification.

Q: Who possesses the ultimate authority to shut down production environments during an active attack?
A: The designated Incident Commander (IC) and the Chief Information Security Officer (CISO) hold dual authority to sever production ingress/egress connections to safeguard the wider enterprise infrastructure.

Q: How are out-of-band communications handled if internal corporate systems are compromised?
A: If enterprise collaboration tools (Slack/Email) are compromised or unavailable, the team must immediately transition to the pre-authenticated, encrypted secondary Signal group managed via personal devices as detailed in the on-call runbook.