Incident Response Policy and Procedures Template

STANDARD OPERATING PROCEDURE: Enterprise Incident Response Policy & Procedures

Document ID: SOP-SEC-TR-042
Effective Date: October 24, 2023
Version: 4.2.0
Review Cadence: Semi-Annual

---

1. Executive Summary & Purpose

This Standard Operating Procedure (SOP) defines the mandatory lifecycle, operational workflows, and governance structures for identifying, containing, eradicating, and recovering from security incidents at Template Registry. The purpose of this document is to ensure rapid, auditable, and resilient mitigation of threats to protect proprietary architecture, customer tenant data, and infrastructure availability. Compliance with this policy is mandatory for all engineering, operations, and security personnel.

---

2. Scope & Prerequisites

2.1 Scope

This policy applies to all physical and virtual assets, cloud environments (AWS/GCP/Azure), container registries, internal corporate networks, and software supply chain components managed or operated by Template Registry.

2.2 Prerequisites & Tooling

Execution of this SOP requires pre-configured access and authorization to the following systems:

• SIEM/Observability: Datadog / Splunk Enterprise Security
• Incident Command Platform: PagerDuty & Jira Service Management (Incident Management module)
• Container Security & Orchestration: Kubernetes CLI (kubectl), AWS CLI, Terraform
• Forensic & Isolation Tooling: AWS Systems Manager (SSM), Automated Snapshot Scripts, Wireshark, Volatility
• Communication Channels: Slack (#sec-incident-command), PagerDuty Bridge, Out-of-Band (OOB) Zoom bridge

---

3. Roles & Responsibilities (RACI Matrix)

Legend: R = Responsible, A = Accountable, C = Consulted, I = Informed

---

4. Step-by-Step Procedure

Phase 1: Detection & Triage

• 1.1 Monitor automated alerts generated by SIEM, IDS/IPS, or user-submitted tickets via #sec-report.
• 1.2 Acknowledge high/critical PagerDuty alerts within 5 minutes of receipt.
• 1.3 Convene the triage channel (#sec-incident-command) and spin up the designated Zoom bridge.
• 1.4 Assign the Incident Commander (IC) role to establish operational cadence and freeze log mutations.
• 1.5 Classify the incident severity using the Template Registry Severity Matrix:
  • SEV-1 (Critical): Active data exfiltration, root compromise, ransomware, or core service outage impacting $>50%$ of tenants.
  • SEV-2 (Major): Single-tenant isolation breach, unauthorized privilege escalation, non-destructive denial of service.
  • SEV-3 (Moderate): Scanned vulnerability with exploit proof-of-concept, policy violation without data loss.

Phase 2: Containment

• 2.1 Execute immediate network isolation of compromised hosts using automated security group modifications (aws ec2 modify-instance-attribute or equivalent cloud controls).
• 2.2 Revoke active IAM sessions, API tokens, and OAuth credentials associated with compromised service accounts or user principals.
• 2.3 Scale down compromised Kubernetes deployments to prevent lateral movement (kubectl scale deployment --replicas=0 -n <namespace> <deployment-name>).
• 2.4 Implement temporary egress/ingress blocking via WAF or Network Security Groups (NSGs) for suspected command-and-control (C2) IP ranges.
• 2.5 Capture volatile memory and snapshot persistent storage volumes (EBS/PersistentDisks) for forensic chain of custody before executing any destructive fixes.

Phase 3: Eradication

• 3.1 Identify the root vector (CVE, misconfiguration, compromised credential, insider threat).
• 3.2 Terminate malicious processes, persistence mechanisms (cron jobs, systemd services, registry keys), and unauthorized binaries.
• 3.3 Patch underlying software vulnerabilities or roll back infrastructure state using verified, clean Terraform configurations.
• 3.4 Rotate all secrets, database credentials, and cryptographic certificates touched during the blast radius.

Phase 4: Recovery

• 4.1 Restore systems from known-good, immutable backups verified clean of malware.
• 4.2 Gradually re-introduce traffic to isolated services using Canary deployments or phased load balancer weight shifts.
• 4.3 Execute synthetic transactions and automated integration test suites to validate systemic integrity.
• 4.4 Monitor telemetry dashboards (Error rates, CPU/Memory profiles, Latency percentiles) for 60 continuous minutes post-restoration.
• 4.5 Formally declare the incident closed in Jira and notify stakeholders via the public status page.

Phase 5: Post-Incident Review (PIR)

• 5.1 Schedule the Blameless PIR meeting within 48 business hours of incident closure.
• 5.2 Compile the chronological timeline of events, system logs, and communication artifacts.
• 5.3 Draft Action Items (Corrective & Preventive Actions - CAPA) with explicit engineering owners and JIRA tracking IDs.
• 5.4 Archive the incident dossier in the secure compliance repository.

---

5. Quality Assurance & Pro-Tips

5.1 Pro-Tips & Best Practices

• Preserve Evidence First: Never reboot a live, compromised container or virtual machine without dumping memory first; volatile state is critical for root cause analysis.
• Maintain Out-of-Band Comms: If internal corporate Slack is compromised or unavailable, immediately transition the incident command team to the encrypted Signal group or PagerDuty incident bridge.
• Single Source of Truth: Update the Jira incident ticket every 30 minutes during SEV-1 events to prevent redundant investigative steps across global engineering pods.

5.2 Common Pitfalls to Avoid

• Premature Remediation: Deleting the attacker's access before identifying the entry vector often destroys forensic evidence and leaves secondary backdoors active.
• Scope Creep: Do not attempt architecture refactoring during an active SEV-1; focus exclusively on containment, eradication, and restoration.

5.3 Metric Thresholds

• Mean Time to Acknowledge (MTTA): $< 5$ minutes for SEV-1.
• Mean Time to Contain (MTTC): $< 30$ minutes for SEV-1.
• Mean Time to Resolve (MTTR): $< 4$ hours for SEV-1.

---

6. Frequently Asked Questions (FAQ)

Q1: What is the exact protocol if an executive or PR representative asks for details during an active SEV-1?
A: Direct all internal and external inquiries to the designated Communications Lead assigned by the Incident Commander. Engineering personnel must never speculate, confirm breaches, or release timelines on unverified communication channels.

Q2: How do we handle forensic isolation when dealing with a heavily utilized multi-tenant Kubernetes cluster?
A: Do not delete the cluster node. Instead, use Kubernetes NetworkPolicies to isolate the offending pod namespace (pod-isolation: true), cordon and drain the specific worker node (kubectl cordon), and attach isolated security groups to capture forensic snapshots without impacting neighboring tenants.