Kubernetes Production Engineering Roadmap

Kubernetes (often abbreviated as K8s, referring to the 8 characters between 'K' and 's') is an open-source container orchestration system originally created by Google based on 15 years of experience running production workloads with Borg and Omega.

1.1 Why Use Kubernetes?

In modern microservices architectures, running raw containers across bare VMs leads to operational failure: manual host placement, hardcoded IP configurations, complex secret management, and manual failovers. Kubernetes solves this by acting as a distributed operating system with core guarantees:

• Declarative State Management: You tell the cluster what you want (e.g. "3 replicas of payment service, 500m CPU limit"), and continuous reconciliation loops ensure reality matches your desired state.
• Automated Self-Healing: Kubelet automatically restarts failed containers; the scheduler reschedules pods when a worker node crashes; unready endpoints are removed from traffic routing instantly.
• Horizontal Autoscaling & Bin Packing: Automatically scales pod replicas based on CPU, RAM, or custom Prometheus metrics, and dynamically packs workloads onto right-sized compute nodes to minimize cloud cost.
• Native Service Discovery & Traffic Routing: Assigns stable virtual IPs (VIPs) and internal DNS names to dynamic pods, load-balancing traffic across healthy endpoints without external service registries.

1.2 Key Concepts and Terminologies

1.3 Kubernetes Alternatives (When NOT to Use K8s)

Kubernetes is not always the best choice for small teams or simple architectures. Consider these alternatives based on complexity:

• Docker Swarm: Native clustering built into Docker. Super fast setup (1 command), zero external dependencies, but lacks advanced CNI routing, fine-grained RBAC, and modern CRD operators. Great for simple setups.
• HashiCorp Nomad: Single lightweight binary that schedules non-containerized binaries, Java JARs, and VMs in addition to containers. Much simpler architecture than K8s, popular in hybrid cloud.
• AWS ECS (Elastic Container Service): Fully managed AWS proprietary container orchestrator. Zero control plane management, deep AWS IAM and CloudWatch integration, but locks you into AWS.

1.4 Container Internals: Kernel Primitives Under the Hood

A container is not a virtual machine. There is no hypervisor or guest kernel. A container is a regular Linux process restricted by two core kernel subsystems:

1.5 Control Plane vs Worker Node Architecture

Every production cluster is split into control plane master components and worker node execution daemons:

Choosing the right Kubernetes environment depends on whether you are running rapid local unit tests, CI/CD pipelines, or multi-region production workloads.

2.1 Choosing a Managed Cloud Provider (EKS vs GKE vs AKS)

2.2 Installing a Local Multi-Node Sandbox with Kind

Kind (Kubernetes in Docker) spins up Docker containers that act as virtual Kubernetes nodes, perfectly simulating multi-node network traffic on your laptop:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
  - containerPort: 443
    hostPort: 443
- role: worker
  labels:
    topology.kubernetes.io/zone: us-east-1a
- role: worker
  labels:
    topology.kubernetes.io/zone: us-east-1b

# Create cluster from configuration
kind create cluster --config kind-3node.yaml --name dev-cluster

# Verify nodes and their readiness
kubectl get nodes -o wide

# Verify system pods
kubectl get pods -n kube-system

2.3 Deploying Your First Application: Imperative vs Declarative

While imperative commands (kubectl run) are handy for quick debugging, production infrastructure should always be declarative:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-api
  namespace: default
  labels:
    app: demo-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demo-api
  template:
    metadata:
      labels:
        app: demo-api
    spec:
      containers:
      - name: web
        image: nginx:1.27-alpine
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
          limits:
            cpu: 250m
            memory: 128Mi
---
apiVersion: v1
kind: Service
metadata:
  name: demo-api-service
spec:
  type: ClusterIP
  selector:
    app: demo-api
  ports:
  - port: 80
    targetPort: 80

Kubernetes provides different workload controllers tailored to distinct application architectural patterns: stateless microservices, stateful databases, node-level daemons, and batch jobs.

3.1 Pod Lifecycle & State Machine

A Pod transitions through a strict state machine from admission to termination:

3.2 Workload Controllers Breakdown

• ReplicaSets: The low-level controller that ensures a specified number of pod replicas are running at all times using label selectors. You rarely create ReplicaSets directly; Deployments manage them for you.
• Deployments: Declarative management of stateless applications. Handles rolling updates, canary revisions, pause/resume, and instant rollbacks via kubectl rollout undo.
• StatefulSets: Manages stateful workloads (databases, message queues). Guarantees:
  • Deterministic network hostnames: db-0, db-1, db-2.
  • Ordered creation, updates, and graceful termination (from N-1 down to 0).
  • Dedicated PersistentVolume per pod created dynamically via volumeClaimTemplates.
• DaemonSets: Ensures all (or specific) worker nodes run a copy of a pod. Used for log forwarders (FluentBit), node monitors (node-exporter), and CNI agents (Cilium, Calico).
• Jobs & CronJobs: Runs batch workloads to completion. Unlike Deployments (which restart terminated pods), Jobs ensure pods run until they exit with status 0. CronJobs trigger Jobs based on cron expressions (e.g. 0 2 * * * for nightly backups).

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-database-backup
  namespace: database
spec:
  schedule: "0 2 * * *" # Every night at 02:00 UTC
  concurrencyPolicy: Forbid # Never start a new job if previous backup is still running
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 2 # Maximum 2 retries before marking Job as failed
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: pg-dump
            image: postgres:16-alpine
            command: ["/bin/sh", "-c", "pg_dump -h postgres-headless -U postgres prod_db | gzip > /backups/db-$(date +%F).sql.gz"]
            volumeMounts:
            - name: s3-backup-mount
              mountPath: /backups
          volumes:
          - name: s3-backup-mount
            persistentVolumeClaim:
              claimName: backup-pvc

The core Kubernetes networking model dictates: Every Pod gets its own IP address, and any Pod can communicate with any other Pod without NAT.

4.1 CNI Evolution: iptables vs Modern eBPF (Cilium)

How packets are routed inside the cluster has evolved dramatically:

4.2 Service Types & Load Balancing

• ClusterIP (Default): Allocates an internal Virtual IP accessible only inside the cluster. CoreDNS creates an A record pointing svc-name.namespace.svc.cluster.local to this VIP.
• NodePort: Opens a high static port (30000–32767) on every node in the cluster, forwarding traffic to the backing pods.
• LoadBalancer: Integrates with cloud controllers to provision external Layer 4 load balancers (e.g. AWS Network Load Balancer).
• Headless (clusterIP: None): Allocates no VIP; DNS resolution returns the list of all individual Pod IPs directly. Essential for StatefulSet master/replica discovery.
• ExternalName: Internal CNAME alias redirecting cluster requests to an external FQDN (e.g. database.rds.amazonaws.com).

4.3 The Next Generation: Gateway API vs Legacy Ingress

The legacy Ingress API combined route definitions, TLS termination, and infrastructure provisioning into a single brittle object littered with custom annotations. The modern Kubernetes Gateway API decouples these responsibilities into clear role-oriented primitives:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: checkout-route
  namespace: ecommerce
spec:
  parentRefs:
  - name: public-gateway
    namespace: gateway-system
  hostnames:
  - "shop.naveedkumbhar.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /checkout
    backendRefs:
    - name: checkout-service-v1
      port: 8080
      weight: 90 # 90% Production Traffic
    - name: checkout-service-v2
      port: 8080
      weight: 10 # 10% Canary Trial

Following 12-factor application methodology, configuration must strictly be separated from code. However, Kubernetes Secrets are stored base64-encoded in plaintext unless KMS envelope encryption is explicitly configured.

5.1 External Secrets Operator (ESO) Architecture

Modern GitOps teams synchronize secrets directly from cloud secret vaults into Kubernetes Secrets automatically:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-irsa-sa
            namespace: external-secrets
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payment-credentials
  namespace: payments
spec:
  refreshInterval: 1h # Automatically sync rotated credentials every hour
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: payment-secret-live # The Kubernetes Secret generated in-cluster
    creationPolicy: Owner
  data:
    - secretKey: STRIPE_API_KEY
      remoteRef:
        key: prod/payments/stripe
        property: api_key

Resource misconfiguration is the leading cause of cluster node instability, latency spikes, and unexpected OOM (Out of Memory) kills.

6.1 Requests vs Limits: The Critical Mechanics

• CPU Requests: Guarantees a minimum slice of CPU bandwidth. Used by kube-scheduler to decide node placement.
• CPU Limits: Enforced by Linux Completely Fair Scheduler (CFS). If a container exceeds its CPU limit, its execution is throttled (slowed down), causing latency spikes, but the process is NOT killed.
• Memory Requests: Used by the scheduler to ensure the node has sufficient memory capacity.
• Memory Limits: Hard boundary. If a container exceeds its memory limit, the Linux kernel OOM killer immediately terminates it with Exit Code 137.

6.2 Quality of Service (QoS) Eviction Hierarchy

6.3 Assigning Quotas to Namespaces: ResourceQuota & LimitRange

Protect your cluster from rogue pods exhausting all compute or storage capacity:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-compute-quota
  namespace: staging
spec:
  hard:
    requests.cpu: "16"
    requests.memory: 32Gi
    limits.cpu: "32"
    limits.memory: 64Gi
    pods: "50"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: default-container-limits
  namespace: staging
spec:
  limits:
  - default: # Default limit if developer omits limits in pod spec
      cpu: 500m
      memory: 512Mi
    defaultRequest: # Default request if developer omits requests
      cpu: 100m
      memory: 128Mi
    type: Container

The kube-scheduler assigns pods to healthy nodes using a 2-stage cycle: Filtering (Predicates) to discard ineligible nodes, followed by Scoring (Priorities) to rank the remaining nodes.

7.1 Node Selection, Affinity & Anti-Affinity

• nodeSelector: Simple key-value label matching (e.g. disktype: ssd).
• nodeAffinity: Expressive matching rules with Boolean operators (In, NotIn, Exists) supporting hard constraints (requiredDuringSchedulingIgnoredDuringExecution) and soft preferences (preferred...).
• podAntiAffinity: Prevents multiple replicas of the same service from running on the same node or in the same availability zone.

7.2 Taints and Tolerations

Taints repel pods from nodes unless the pod has a matching toleration. Used for dedicated GPU nodes or control plane isolation:

# Node Taint applied via CLI:
# kubectl taint nodes gpu-node-01 dedicated=gpu:NoSchedule

# Pod with matching toleration:
spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"
  containers:
  - name: ml-inference
    image: pytorch/pytorch:latest
    resources:
      limits:
        nvidia.com/gpu: 1

7.3 Topology Spread Constraints

Evenly distribute pods across cloud Availability Zones (AZs) to survive a datacenter outage:

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: payment-api

7.4 Pod Priorities, Preemption & Node Evictions

When a cluster runs out of resources, pods with a higher PriorityClass preempt (kill) lower-priority pods. Meanwhile, if a node suffers disk or memory exhaustion, kubelet initiates Node Pressure Evictions (e.g. when memory.available < 100Mi).

Stateful workloads (PostgreSQL, Kafka, Elasticsearch) require persistent storage governed by the Container Storage Interface (CSI).

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer # Delays volume creation until pod lands on an AZ node
allowVolumeExpansion: true
reclaimPolicy: Retain
parameters:
  type: gp3
  encrypted: "true"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-cluster
  namespace: database
spec:
  serviceName: postgres-headless
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgresql
        image: postgres:16-alpine
        ports:
        - containerPort: 5432
        volumeMounts:
        - name: pgdata
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: pgdata
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: ebs-gp3-sc
      resources:
        requests:
          storage: 100Gi

Kubernetes security follows a Defense-in-Depth model across 4 layers: Cloud/Hardware, Cluster (API & RBAC), Container, and Code.

9.1 Role-Based Access Control (RBAC): Least Privilege

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: staging
  name: app-developer
rules:
# Can view and restart pods/deployments, but CANNOT read secrets or delete PVCs
- apiGroups: ["", "apps"]
  resources: ["pods", "deployments", "replicasets", "services"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
  resources: ["pods/log", "pods/portforward"]
  verbs: ["get", "list", "create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: bind-app-developer
  namespace: staging
subjects:
- kind: Group
  name: "oidc:engineering-team"
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: app-developer
  apiGroup: rbac.authorization.k8s.io

9.2 Container Hardening with SecurityContext

Never run containers as root (UID 0), never allow privilege escalation, and drop all default Linux capabilities:

spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    runAsGroup: 10001
    fsGroup: 10001
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: hardened-api
    image: my-company/api:v3.1
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop:
        - ALL # Drops CAP_SYS_ADMIN, CAP_NET_RAW, etc.
    volumeMounts:
    - name: tmp-dir
      mountPath: /tmp
  volumes:
  - name: tmp-dir
    emptyDir: {}

In dynamic, distributed microservices, you cannot SSH into pods to check log files. You need an automated observability pipeline across Metrics, Logs, and Traces.

10.1 Core Observability Stack

• Metrics (Prometheus & Grafana): Time-series metrics scraped via HTTP endpoints formatted in Prometheus OpenMetrics standards. Monitored via kube-prometheus-stack and kube-state-metrics.
• Logs (Vector / FluentBit & Loki): Standard container runtime logs (stdout/stderr) shipped to centralized stores with namespace, pod, and container labels attached.
• Traces (OpenTelemetry & Tempo / Jaeger): Distributed request contexts traced end-to-end across multiple microservices with trace and span IDs.

10.2 Essential SRE PromQL Queries for On-Call Engineers

# 1. Pods with High CPU Throttling (% of runtime throttled by CFS quota)
sum(rate(container_cpu_cfs_throttled_periods_total[5m])) by (pod, namespace)
  /
sum(rate(container_cpu_cfs_periods_total[5m])) by (pod, namespace) * 100 > 25

# 2. Containers within 10% of their hard Memory Limit (Imminent OOMKill warning)
(container_memory_working_set_bytes{container!=""}
  /
container_spec_memory_limit_bytes{container!=""}) * 100 > 90

# 3. Microservice 5xx HTTP Error Rate over 5 minutes
sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
sum(rate(http_requests_total[5m])) * 100 > 1.5

# 4. CrashLooping Pods (Restarts > 5 in last 15 minutes)
sum(increase(kube_pod_container_status_restarts_total[15m])) by (pod, namespace) > 5

Autoscaling in Kubernetes operates across two complementary tiers: Pod Autoscaling (Horizontal Pod Autoscaler / KEDA) and Node Autoscaling (Karpenter / Cluster Autoscaler).

11.1 Horizontal Pod Autoscaler (HPA v2) with Multi-Metrics

The HPA reconciler calculates target replicas via the formula: desiredReplicas = ceil[ currentReplicas * ( currentMetricValue / desiredMetricValue ) ]

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-processor-hpa
  namespace: ecommerce
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-processor
  minReplicas: 3
  maxReplicas: 50
  metrics:
  # Metric 1: Target 75% Average CPU Utilization
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 75
  # Metric 2: Custom Metric from Prometheus (HTTP Requests per second)
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: 1500m # 1.5k requests/sec per pod
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300 # Prevent flapping (wait 5 mins before scaling down)
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60

11.2 Karpenter: Just-in-Time Node Provisioning

Traditional Cluster Autoscaler relies on AWS Auto Scaling Groups (ASGs), taking 3–5 minutes to scale nodes. Karpenter bypasses ASGs completely, calling EC2 Fleet APIs directly to launch right-sized, spot-mixed instances in under 40 seconds.

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: general-compute
spec:
  template:
    spec:
      requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["spot", "on-demand"]
      - key: kubernetes.io/arch
        operator: In
        values: ["arm64", "amd64"]
      - key: karpenter.k8s.aws/instance-category
        operator: In
        values: ["c", "m", "r"]
      nodeClassRef:
        name: default-ec2-node-class
  disruption:
    consolidationPolicy: WhenUnderutilized # Automatically bin-packs and scales down underutilized nodes
    expireAfter: 720h # 30-day node rotation for security patching

Treat infrastructure and applications as code. Never run kubectl apply from developer laptops in production. All state transitions must be versioned in Git and reconciled continuously.

12.1 Deployment Strategies: Rolling vs Blue/Green vs Canary

12.2 Declarative GitOps with ArgoCD

An Application custom resource in ArgoCD binds a Git repository branch directly to a destination cluster namespace:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payment-microservice
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'https://github.com/org/k8s-manifests.git'
    targetRevision: main
    path: environments/production/payments
    helm:
      valueFiles:
      - values.yaml
      - values-production.yaml
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: payments
  syncPolicy:
    automated:
      prune: true     # Deletes resources removed from Git
      selfHeal: true  # Reverts manual 'kubectl edit' overrides
    syncOptions:
    - CreateNamespace=true

The true power of Kubernetes is its extensibility. You can define your own APIs, write custom controllers in Go, manage multi-cluster meshes, and automate disaster recovery.

13.1 Custom Resource Definitions (CRDs) & Operators

A CRD extends the Kubernetes API by registering a new resource type (e.g. PostgresCluster). A Custom Controller / Operator runs a control loop that watches that resource and executes custom business logic:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: postgresclusters.db.naveedkumbhar.com
spec:
  group: db.naveedkumbhar.com
  versions:
  - name: v1alpha1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              replicas:
                type: integer
                minimum: 1
              storageSize:
                type: string
  scope: Namespaced
  names:
    plural: postgresclusters
    singular: postgrescluster
    kind: PostgresCluster
    shortNames:
    - pg

13.2 Cluster Administration: Installing Control Planes & Worker Nodes

For self-managed bare-metal or private cloud infrastructure, use kubeadm:

• kubeadm init: Initializes the control plane, generates TLS certificates in /etc/kubernetes/pki, writes static pod manifests for etcd and kube-apiserver, and outputs the worker join token.
• kubeadm join: Joins a new worker node to the cluster over secure mTLS.
• Node Maintenance:
  • kubectl cordon <node>: Marks node as unschedulable (no new pods placed).
  • kubectl drain <node> --ignore-daemonsets --delete-emptydir-data: Safely evicts all running pods respecting PodDisruptionBudgets before OS patching or reboot.
  • kubectl uncordon <node>: Resumes normal scheduling after maintenance.

13.3 Multi-Cluster Management & Service Mesh

Enterprises run multiple clusters for disaster recovery, regulatory data residency, and blast-radius isolation. Solutions like Cilium ClusterMesh connect pod networks across independent cloud clusters over encrypted WireGuard/IPsec tunnels with global multi-cluster service load balancing.

13.4 SRE Production Triage Runbooks

13.5 etcd Disaster Recovery: Backup & Restore

# 1. Take a live etcd snapshot on the control-plane node
ETCDCTL_API=3 etcdctl   --endpoints=https://127.0.0.1:2379   --cacert=/etc/kubernetes/pki/etcd/ca.crt   --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key   snapshot save /var/backups/etcd-snapshot-$(date +%Y%m%d_%H%M%S).db

# 2. Verify snapshot integrity
ETCDCTL_API=3 etcdctl --write-out=table snapshot status /var/backups/etcd-snapshot-*.db

# 3. Restore snapshot into a clean data directory
ETCDCTL_API=3 etcdctl   --data-dir=/var/lib/etcd-restored   snapshot restore /var/backups/etcd-snapshot-*.db

13.6 Related Learning Roadmaps

Kubernetes mastery bridges infrastructure, software engineering, and operations. Continue your path with:

• DevOps Engineer Roadmap: End-to-end CI/CD, cloud architecture, and infrastructure automation.
• Docker & Containerization Roadmap: Multi-stage builds, container security, and base image optimization.
• Linux Administration Roadmap: Shell scripting, systemd, kernel parameters, and RHCSA prep.
• Terraform & IaC Roadmap: Infrastructure as code, modules, state management, and multi-cloud delivery.
• System Design Roadmap: Distributed consensus, caching, microservices, and fault-tolerant architecture.