Production Deployment Plan Example and Protocol

Standard Operating Procedure: Production Deployment Lifecycle & Execution Protocol

1. Document Control Block

• Document ID: SOP-ENG-TR-042
• Effective Date: October 24, 2023
• Version: 4.1.0
• Review Cadence: Semi-Annually
• Owner: Julian Vance, Chief Architect, Template Registry

---

2. Executive Summary & Purpose

This Standard Operating Procedure (SOP) defines the mandatory, deterministic workflow for promoting software releases to production environments within the Template Registry infrastructure. The primary objective is to eliminate deployment-induced regressions, enforce zero-downtime availability targets (SLAs $\ge 99.99%$), and establish a verifiable audit trail for compliance frameworks (SOC 2, ISO 27001). Adherence to this protocol is compulsory for all engineering operations personnel.

---

3. Scope & Prerequisites

3.1 Scope

• Applies to all microservices, stateful data stores, infrastructure-as-code (IaC) configurations, and edge routing rules deployed to the prod-us-east-1 and prod-eu-west-1 regions.

3.2 Required Tools & Access

• CLI Utilities: kubectl (v1.28+), helm (v3.12+), terraform (v1.5+), aws-cli (v2.13+).
• Identity & Access: Active PagerDuty on-call rotation, AWS Administrator Access (via temporary STS assume-role), GitHub Actions deployment runner privileges.
• Observability: Datadog APM, Prometheus/Grafana operational dashboards, PagerDuty integration hook.

3.3 Safety & Environmental Constraints

• Maintenance Windows: Standard deployments must occur during low-traffic windows (Tuesday–Thursday, 0200–0400 UTC). Emergency hotfixes require explicit written authorization from the Chief Architect or VP of Engineering.

---

4. Roles & Responsibilities

---

5. Step-by-Step Procedure

Phase 1: Pre-Deployment Verification

• 1.1 Confirm all unit, integration, and security scans (SAST/DAST) passed successfully in the CI pipeline for release tag v4.1.0.
• 1.2 Verify that the staging environment (stage-us-east-1) has operated stably on the target build for a minimum of 24 continuous hours.
• 1.3 Execute the dependency audit script to ensure no unresolved CVEs exist:

  make security-audit RELEASE_TAG=v4.1.0
  

• 1.4 Broadcast a pre-deployment notification via the #ops-announcements Slack channel and PagerDuty change event interface T-minus 60 minutes prior to execution.

Phase 2: Environment Preparation & Backup

• 2.1 Initiate an on-demand snapshot of all production PostgreSQL databases and persistent volumes:

  aws rds create-db-snapshot --db-instance-identifier template-registry-prod \
      --db-snapshot-identifier template-registry-prod-pre-deploy-v4-1-0
  

• 2.2 Verify snapshot completion status and ensure automated retention locks are applied.
• 2.3 Drain and verify active connection pools on the legacy application nodes to prepare for traffic shifting.

Phase 3: Execution (Canary & Progressive Delivery)

• 3.1 Apply infrastructure modifications via Terraform, ensuring state lock acquisition succeeds:

  terraform init && terraform apply -target=module.production_infra
  

• 3.2 Deploy the canary release representing 10% of total cluster weight using ArgoCD/Helm:

  helm upgrade --install template-registry ./charts/template-registry \
      --namespace production \
      --set image.tag=v4.1.0 \
      --set canary.weight=10
  

• 3.3 Monitor the canary phase metrics for a mandatory observation window of 15 minutes. Check error rates, latency percentiles ($p_{99}$), and CPU throttling.
• 3.4 Incrementally scale traffic weights to 50%, then 100%, pausing 10 minutes between increments while continuously evaluating Datadog APM monitors.

Phase 4: Post-Deployment Verification & Sign-off

• 4.1 Run the automated synthetic smoke test suite against the live production endpoints:

  npx playwright test --config=playwright.prod.config.ts
  

• 4.2 Verify that application logs show zero unhandled exceptions or connection pool exhaustion errors:

  kubectl logs -l app.kubernetes.io/name=template-registry --tail=500 --prefix=true
  

• 4.3 Update the internal deployment registry and close the corresponding Jira change ticket.
• 4.4 Send deployment success confirmation to #ops-announcements.

---

6. Quality Assurance & Pro-Tips

6.1 Best Practices

• Immutable Artifacts: Never rebuild containers or rewrite tags in production. Use the exact binary artifact promoted from staging.
• Feature Flags: Decouple code deployment from feature release. Wrap high-risk UI or transactional workflows in dynamic feature flags.

6.2 Common Pitfalls

• Skipping Smoke Tests: Assuming an API health check (/healthz) indicates functional correctness. Always execute transactional synthetic user tests.
• Ignoring Database Migrations: Running destructive schema changes (e.g., dropping columns) in the same release cycle as application updates. Use the expand-contract pattern.

6.3 Metric Thresholds (Rollback Triggers)

• Error Rate ($5xx$): $> 0.1%$ over a rolling 3-minute window.
• Latency ($p_{99}$): $> 450\text{ms}$ sustained for 5 minutes.
• CPU Saturation: $> 85%$ node utilization across $> 3$ worker nodes.

---

7. Frequently Asked Questions

Q: What is the exact protocol if the canary phase trips a rollback threshold?
A: Immediately abort the promotion by executing the automated rollback script: ./scripts/rollback.sh --version v4.0.9. This reverts the ingress weight to 0% on the new build, routes 100% of traffic back to the stable release, and pages the on-call engineer. Do not attempt manual debugging of live production containers until traffic is fully reverted.

Q: Can database schema migrations be executed automatically inside the Helm chart lifecycle hooks?
A: No. Helm pre-install or pre-upgrade hooks can cause cluster deadlocks if pods fail or time out. All migrations must be executed out-of-band via a validated Kubernetes Job object prior to shifting application traffic to the new version.