IT Incident Response Plan Template

Standard Operating Procedure: IT Incident Response Plan (IRP)

1. Document Control Block

---

2. Executive Summary & Purpose

This Standard Operating Procedure (SOP) defines the institutional-grade framework for detecting, containing, eradicating, and recovering from Information Technology security incidents across Template Registry infrastructure. The objective is to minimize downtime, preserve forensic integrity, ensure regulatory compliance, and systematically restore operational baseline functionality with zero unvetted regressions.

---

3. Scope & Prerequisites

Scope

Applies to all production environments, staging environments, corporate networks, cloud resources (AWS/GCP), and endpoints managed by Template Registry.

Prerequisites & Required Tooling

• SIEM / Observability: Datadog, AWS CloudTrail, OpenSearch.
• Forensic Tooling: Volatility, Wireshark, Autopsy, OSQuery.
• Communication Channels: PagerDuty (Primary Alerting), Secure Slack Enterprise (War Room), Out-of-Band Bridge (Backup).
• Access Control: Break-glass AWS IAM administrative roles, Vault secret management tokens.

---

4. Roles & Responsibilities (RACI Matrix)

---

5. Step-by-Step Procedure

Phase 1: Identification & Triage

• 1.1 Acknowledge incoming alerts within 5 minutes via PagerDuty.
• 1.2 Verify false positive metrics utilizing the primary SIEM dashboard filters.
• 1.3 Open an emergency incident channel in Slack (#incident-YYYYMMDD-[name]).
• 1.4 Designate the Incident Commander (IC) and assign log-keeping responsibilities.
• 1.5 Assign initial severity rating (Sev-1 through Sev-4) based on data exposure and operational impact:
  • Sev-1: Active data exfiltration, total core service outage.
  • Sev-2: Partial service degradation, isolated endpoint compromise.
  • Sev-3: Non-critical application error, anomalous internal traffic.

Phase 2: Containment

• 2.1 Execute network isolation protocols for compromised EC2/GCE instances via Security Group overrides (sg-quarantine).
• 2.2 Revoke compromised IAM credentials, active session tokens, and OAuth grants instantly via HashiCorp Vault / AWS IAM CLI.
• 2.3 Capture volatile system memory (RAM) and disk snapshots for forensic analysis prior to rebooting or terminating assets.
• 2.4 Implement temporary ingress/egress filtering at the WAF (Web Application Firewall) layer to block malicious IP ranges.

Phase 3: Eradication

• 3.1 Identify root cause vulnerability vector (e.g., zero-day exploit, stolen credential, dependency compromise).
• 3.2 Remove malicious payloads, backdoors, web shells, and unauthorized cron jobs from affected hosts.
• 3.3 Patch underlying vulnerabilities or rollback affected container deployments to known-good Git hashes.
• 3.4 Rotate all system-level shared secrets, API keys, and database passwords touched during the compromise window.

Phase 4: Recovery

• 4.1 Restore system state from immutable, cryptographically verified backups if file integrity is unrecoverable.
• 4.2 Gradually reintroduce isolated services back into the production load balancer pool.
• 4.3 Execute synthetic transactions and automated smoke tests to validate end-to-end functionality.
• 4.4 Monitor CPU, memory, and network throughput baselines for 4 hours post-restoration.

Phase 5: Post-Incident Review (PIR)

• 5.1 Schedule mandatory PIR meeting within 48 hours of incident closure.
• 5.2 Compile complete timeline of events, artifacts, and communication logs.
• 5.3 Draft corrective action tickets in Jira with assigned owners and hard deadlines.
• 5.4 Archive all forensic evidence in the secure cold-storage compliance bucket.

---

6. Quality Assurance & Pro-Tips

Best Practices (Architectural Directives)

• Preserve State: Never power off a compromised running host before acquiring a memory dump; volatile evidence is permanently lost upon reboot.
• Out-of-Band Comm: Assume internal corporate networks are hostile during a Sev-1; rely on out-of-band cellular or secondary isolated tenants for coordination.

Common Pitfalls

• Premature Notification: Do not notify external stakeholders or regulatory bodies before the IC and Legal have verified scope and containment.
• Log Contamination: Avoid logging into compromised systems using administrative accounts that share credentials across other infrastructure nodes.

Metric Thresholds

• MTTD (Mean Time to Detect): $\le 15 \text{ minutes}$
• MTTR (Mean Time to Respond/Contain): $\le 30 \text{ minutes}$ (Sev-1)

---

7. Frequently Asked Questions

Q: At what point must an incident be escalated to external legal counsel and regulatory bodies (e.g., GDPR/CCPA)?
A: Escalation occurs immediately upon verified confirmation of Personally Identifiable Information (PII) exfiltration, or if system unavailability breaches enterprise SLA thresholds. The Legal Counsel must be tagged in the war room within 60 minutes of a confirmed Sev-1 data breach.

Q: Can developers independently modify production security groups during an active incident if the DevOps lead is unavailable?
A: No. In the absence of the DevOps lead, access must be routed through the designated on-call Incident Commander using break-glass administrative paths (AssumeRole: EmergencyOps), which automatically record immutable audit trails.