Ddos Incident Response Plan Template

STANDARD OPERATING PROCEDURE: Distributed Denial of Service (DDoS) Incident Response Plan

Document ID: SOP-SEC-042
Effective Date: October 24, 2023
Version: 3.2.0
Review Cadence: Semi-Annual
Classification: Internal Restricted / Tier-1 Operational

---

1. Executive Summary & Purpose

This Standard Operating Procedure (SOP) defines the institutional-grade protocol for detecting, mitigating, and recovering from Distributed Denial of Service (DDoS) attacks targeting Template Registry infrastructure. The primary objective is to maintain service availability, protect core data registries, minimize latency, and execute rapid failovers without human error during high-stress operational events.

---

2. Scope & Prerequisites

2.1 Scope

This protocol applies to all public-facing API gateways, edge proxies, authoritative DNS servers, and internal microservices residing within Template Registry's cloud and on-premise datacenters.

2.2 Prerequisites & Required Toolset

Operators executing this SOP must have pre-configured, authenticated access to the following systems:

• Edge & CDN: Cloudflare Enterprise / AWS Shield Advanced console.
• Observability: Datadog / Grafana dashboards (Metrics: ingress_requests_per_sec, p99_latency, http_5xx_error_rate).
• Network Instrumentation: Wireshark, tcpdump, iftop.
• Infrastructure Access: Multi-Factor Authentication (MFA) enabled SSH keys to jump hosts, Kubernetes (kubectl) cluster administrative contexts.
• Communications: PagerDuty, dedicated Slack incident bridge (#inc-sec-network).

---

3. Roles & Responsibilities (RACI Matrix)

Legend: R = Responsible, A = Accountable, Consulted = C, Informed = I.

---

4. Step-by-Step Procedure

Phase 1: Detection & Triage

• 1.1 Acknowledge automated PagerDuty alert triggered by anomaly detection (Threshold: ingress_requests_per_sec > 300% of baseline, or http_5xx_error_rate > 5%).
• 1.2 Declare an active security incident by executing /incident declare ddos-sev1 in the #inc-sec-network Slack channel, automatically spinning up the bridge.
• 1.3 Open the master monitoring dashboard (https://dash.templateregistry.internal/d/edge-security) to isolate the attack vector:
  • Layer 3/4 (SYN Flood, UDP Reflection): Inspect bandwidth saturation and packet-per-second (pps) metrics.
  • Layer 7 (HTTP Flood, Slowloris): Inspect URI distribution, unique IP signatures, and User-Agent strings.

Phase 2: Containment & Mitigation

• 2.1 (Layer 3/4 Attack) Log into AWS Shield / upstream ISP provider console and activate manual scrubbers or trigger BGP flowspec scrubbing rules.
• 2.2 (Layer 7 Attack) Access the Cloudflare/WAF dashboard and elevate the security level to "Under Attack Mode" (JS Challenge enforcement).
• 2.3 Apply emergency rate-limiting rules at the API gateway ingress controllers via kubectl:

  kubectl apply -f /infra/security/emergency-rate-limit-patch.yaml
  

• 2.4 Review dropped traffic metrics for false positives. If legitimate client IPs are blocked, whitelist CIDR blocks or specific TLS fingerprints immediately.
• 2.5 Scale internal auto-scaling groups (ASGs) and Kubernetes Horizontal Pod Autoscalers (HPA) to absorb residual legitimate surge:

  kubectl scale deployment registry-api --replicas=150
  

Phase 3: Eradication & Verification

• 3.1 Capture traffic samples using tcpdump on the edge proxy for forensic analysis:

  sudo tcpdump -i eth0 -nn -s0 -c 10000 'tcp[tcpflags] & (tcp-syn) != 0' -w /var/log/forensics/ddos_suspects.pcap
  

• 3.2 Verify service recovery by checking synthetic transaction monitors (/healthz endpoints returning HTTP 200).
• 3.3 Confirm latency metrics (p99) have returned to standard operational SLAs (< 250ms).

Phase 4: Post-Incident Recovery & Closure

• 4.1 De-escalate security configurations (disable "Under Attack Mode", normalize rate limits) 30 minutes after attack traffic ceases.
• 4.2 Archive forensic PCAP files and WAF logs to secure S3 bucket (s3://tr-security-forensics-vault/2023/).
• 4.3 Convene the Post-Incident Review (PIR) meeting within 48 hours to complete the Root Cause Analysis (RCA).

---

5. Quality Assurance & Pro-Tips

Best Practices

• Assume Compromise of Edge: Always maintain a secondary, out-of-band management network path to core routers in case BGP routing tables are poisoned or unstable.
• Automate Where Possible: Rely on upstream automated scrubbing (Cloudflare/AWS) as the first line of defense; manual IP blacklisting during a 100Gbps attack is operationally infeasible.

Common Pitfalls

• Pitfall: Blanket-blocking geographic regions. Correction: Attackers frequently spoof or leverage legitimate global botnets; use behavioral and rate-based fingerprinting instead of geo-blocks.
• Pitfall: Forgetting to scale backend databases when scaling API gateways. Correction: Ensure connection poolers (e.g., PgBouncer) are adjusted concurrently with API replicas.

Metric Thresholds

• Normal Baseline: < 10,000 req/sec globally; p99 latency < 150ms.
• Sev-1 Threshold: > 50,000 req/sec globally, or p99 latency > 1,000ms coupled with 5xx spikes.

---

6. Frequently Asked Questions (FAQ)

Q: What should I do if the WAF control plane is inaccessible due to API saturation?
A: Immediately pivot to the cloud provider's emergency CLI or contact the upstream provider's (Cloudflare/AWS) enterprise support hotline using the priority PIN stored in the vault (vault://sec/enterprise-support-pins). Execute baseline infrastructure drops at the provider level via direct-connect bypass.

Q: How do we distinguish between a flash-crowd (legitimate traffic spike) and a Layer 7 DDoS attack?
A: Inspect the request distribution and entropy. A flash-crowd typically exhibits natural user behavior (diverse user agents, valid session cookies, distributed geographic origin, and standard browsing paths). A Layer 7 DDoS often shows uniform User-Agents, requests targeting non-cacheable expensive database endpoints (/api/v1/search), low cookie entropy, or exact mathematical request intervals.