Cloud Infrastructure Setup and Maintenance

Standard Operating Procedure: Cloud Infrastructure Setup and Maintenance

1. Document Control Block

• Document ID: SOP-TR-OPS-4091
• Effective Date: October 24, 2023
• Version: 3.2.0
• Review Cadence: Semi-Annual (Every 6 Months)
• Target Audience: Cloud Operations Engineers, DevOps Specialists, Security Compliance Officers, Lead Systems Architects

---

2. Executive Summary & Scope

This Standard Operating Procedure (SOP) defines the institutional requirements for provisioning, hardening, scaling, and maintaining cloud infrastructure across all Template Registry operational environments (Development, Staging, Production). This protocol ensures strict adherence to SOC 2 Type II, ISO/IEC 27001, and HIPAA data governance frameworks.

The scope encompasses all Infrastructure-as-Code (IaC) repositories, cloud service provider (CSP) tenant accounts, container orchestration clusters, and automated continuous integration/continuous deployment (CI/CD) pipelines managed by Template Registry operations teams.

---

3. Prerequisites & Tool Requirements

Personnel executing this SOP must possess verified administrative access and specialized tooling configurations:

• Certifications Required: AWS Certified Solutions Architect – Professional, Certified Kubernetes Administrator (CKA), or equivalent institutional equivalent.
• Hardware/Software Access:
  • FIPS 140-2 validated Hardware Security Module (HSM) or corporate-issued YubiKey for Multi-Factor Authentication (MFA).
  • Terraform v1.5+ CLI installed locally.
  • Kubectl (matching cluster version $\pm 1$).
  • AWS CLI v2 / Azure CLI authenticated via short-lived assumed roles.
• Security Protocols: Zero-Trust Network Access (ZTNA) active session routed through the corporate secure gateway.

---

4. Roles & Responsibilities (RACI Matrix Table)

---

5. Step-by-Step Operational Procedure

Phase I: Architecture Provisioning via Infrastructure-as-Code (IaC)

• 1.1 Clone the master infrastructure repository and checkout a dedicated feature branch following naming convention feature/infra-[ticket-id].
• 1.2 Validate the local Terraform variable files (*.tfvars) to ensure environment-specific parameters (CIDR blocks, instance sizing, replica counts) match the deployment tier.
• 1.3 Initialize the Terraform workspace and lock the remote state backend using the command: terraform init -backend-config=config.hcl.
• 1.4 Execute terraform plan -out=tfplan.binary and perform a line-by-line review of the resource modification delta (creates, updates, destroys).
• 1.5 Submit the generated execution plan for peer review via pull request; obtain at least one formal approval from a designated peer and one automated security scan clearance (e.g., Checkov or tfsec).
• 1.6 Apply the execution plan in the target environment using terraform apply tfplan.binary.

Phase II: Network Hardening and Identity Access Management (IAM)

• 2.1 Verify Virtual Private Cloud (VPC) topology isolation: confirm that database and worker subnets lack direct Internet Gateways (IGW) and route exclusively through NAT Gateways.
• 2.2 Audit Security Groups and Network Access Control Lists (NACLs) to ensure zero open ingress rules on ports outside of port 443 (HTTPS) and port 22 (restricted via Bastion Host/Session Manager).
• 2.3 Enforce Least Privilege IAM policies by executing automated validation scripts that scan for wildcards (*) in action and resource attributes.
• 2.4 Confirm that all storage buckets and block storage volumes have default-at-rest encryption enabled utilizing customer-managed KMS keys (CMK).

Phase III: Continuous Maintenance and Patch Management

• 3.1 Schedule maintenance windows during off-peak hours (Sunday 02:00–04:00 UTC) and broadcast maintenance notifications to internal telemetry channels.
• 3.2 Update base container images and underlying node AMI instances using immutable infrastructure replacement strategies (blue/green or rolling updates).
• 3.3 Apply security patches to cluster control planes and worker nodes without inducing service degradation; drain nodes gracefully via kubectl drain [node-name] --ignore-daemonsets --delete-emptydir-data.
• 3.4 Validate cluster operational health post-patching through automated smoke tests and canary metric observation.

---

6. Quality Assurance & Verification Criteria

Deployments and maintenance cycles are considered successful only when meeting the following strict parameters:

• Audit Checklist:
  • Zero high or critical vulnerabilities identified via daily container and infrastructure vulnerability scanners (e.g., Prisma Cloud / Trivy).
  • All resource tags (Environment, Owner, CostCenter, DataClassification) verified and compliant with tagging policies.
• Tolerance Thresholds:
  • API Gateway and Ingress latency degradation must not exceed $\le 5%$ baseline during rolling updates.
  • Zero data loss during storage volume resizing or database failover simulations.
• Acceptance Benchmarks:
  • Automated monitoring agents (Datadog/Prometheus) report 100% node reachability and cluster health within 10 minutes of operation completion.

---

7. Practical Troubleshooting Scenarios

Scenario A: Terraform State Lock Contention / Orphaned Lock

• Symptom: terraform apply fails with an error stating the state file is locked by another process, preventing legitimate pipeline execution.
• Immediate Corrective Action Protocol:
  1. Verify across the team communication channels that no other engineer is actively running an apply against the target workspace.
  2. Query the DynamoDB state-locking table (or equivalent remote lock backend) to identify the conflicting LockID.
  3. Force-unlock the state explicitly via CLI: terraform force-unlock [LOCK_ID]. Warning: Ensure no concurrent execution is occurring to prevent state corruption.
  4. Re-run terraform plan to confirm state integrity.

Scenario B: Kubernetes Pod CrashLoopBackOff Post-Node Upgrade

• Symptom: Core microservices enter a CrashLoopBackOff state following an underlying worker node AMI update and OS kernel patch.
• Immediate Corrective Action Protocol:
  1. Inspect container logs immediately: kubectl logs [pod-name] --previous.
  2. Check for kernel-level incompatibilities or missing CSI (Container Storage Interface) drivers on the new node image.
  3. If unresolvable within 15 minutes, cordon and taint the faulty node: kubectl cordon [node-name].
  4. Roll back the deployment revision: kubectl rollout undo deployment/[deployment-name] -n [namespace].
  5. Escalate to the Lead Systems Architect and file an incident ticket with the cloud provider support desk if hypervisor-level metrics indicate hardware degradation.