A Kubernetes Rollout That Failed Safely: Readiness, Deadlines, and Rollback

A deployment strategy is only credible when a failed release has been exercised. I built a local Kubernetes experiment that rolls four replicas from v1 to v2, continuously probes the service, deploys an image that never becomes ready, waits for the Deployment progress deadline, and rolls back.

The observed result was:

Stage	Outcome
Initial v1 deployment	Four available replicas
v1 → v2 rolling update	Completed
Continuous availability	264 probes, zero non-200 responses
Broken image rollout	Exceeded its progress deadline
`kubectl rollout undo`	Restored four available v2 replicas
Versions served	v1 and v2; broken never entered Service endpoints

This does not prove every Kubernetes rollout is zero-downtime. It proves that this manifest, application, and failure mode behaved as designed in Kind.

Make availability a Deployment constraint

The core manifest is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: rollout-lab
spec:
  replicas: 4
  revisionHistoryLimit: 5
  minReadySeconds: 3
  progressDeadlineSeconds: 15
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: api
          image: rollout-lab:v1
          imagePullPolicy: Never
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 1
            failureThreshold: 2
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            periodSeconds: 5

maxUnavailable: 0 tells the Deployment controller not to reduce available replicas during the rollout. maxSurge: 1 allows one extra pod while replacement proceeds. With four replicas, the controller can run up to five during transition.

minReadySeconds: 3 requires a pod to remain ready briefly before it counts as available. progressDeadlineSeconds: 15 makes a stalled rollout observable as ProgressDeadlineExceeded; it does not roll the Deployment back automatically.

imagePullPolicy: Never is deliberate only for Kind because the experiment loads local images into the node. Production should use immutable registry digests and a pull policy consistent with that release process.

Readiness and liveness answer different questions

The test application exposes:

mux.HandleFunc("/health", func(w http.ResponseWriter, _ *http.Request) {
    w.WriteHeader(http.StatusOK)
})

mux.HandleFunc("/ready", func(w http.ResponseWriter, _ *http.Request) {
    if broken == "true" {
        http.Error(w, "not ready", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
})

Liveness asks whether Kubernetes should restart the container. Readiness asks whether the pod should receive Service traffic. A dependency outage often should fail readiness without failing liveness; restarting a healthy process does not repair a remote database.

The broken image remains live but never becomes ready. Kubernetes starts it, probes it, and keeps it out of Service endpoints. Existing v2 pods continue serving while the rollout stalls.

Readiness must represent the ability to serve the endpoint safely. A hard-coded 200 response can produce a “successful” rollout that sends traffic to an unusable application.

Probe availability during the rollout

The experiment starts a background request loop after v1 is available:

while true; do
  timestamp=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  response=$(curl -sS --max-time 1 \
    -w '|%{http_code}' \
    http://localhost:18059/ \
    | tr -d '\n' || true)
  printf '%s|%s\n' "$timestamp" "$response" \
    >> artifacts/availability.log
  sleep 0.1
done

The JSON response includes the application version, so the artifact proves both v1 and v2 served traffic. It also records the status of every probe.

My first parser counted the JSON line and HTTP status line separately, reporting 244 false failures even though every request returned 200. Removing the response newline corrected the evidence. Measurement code is production code: validate its record format before trusting a deployment conclusion.

Exercise a healthy rollout

The release command changes the image and waits for the controller:

kubectl --context kind-rollout-lab \
  -n rollout-lab \
  set image deployment/api api=rollout-lab:v2

kubectl --context kind-rollout-lab \
  -n rollout-lab \
  rollout status deployment/api --timeout=90s

With maxUnavailable: 0, the controller creates a v2 pod, waits until it is available, then removes an old pod. The rollout repeats until four v2 pods are available.

The external loop recorded no non-200 response. That says more than rollout status: it observes the Service path while the controller changes pods.

Inject a readiness failure and wait for the deadline

The broken image is compiled with readiness forced false:

kubectl --context kind-rollout-lab \
  -n rollout-lab \
  set image deployment/api api=rollout-lab:broken

kubectl --context kind-rollout-lab \
  -n rollout-lab \
  rollout status deployment/api --timeout=25s

The new pod cannot become available, so the Deployment cannot continue without violating maxUnavailable: 0. After the configured progress deadline, Kubernetes reports the stalled rollout. The old ready replicas keep serving.

The response should begin with conditions and events:

kubectl -n rollout-lab describe deployment api
kubectl -n rollout-lab get pods -o wide
kubectl -n rollout-lab get events --sort-by=.lastTimestamp
kubectl -n rollout-lab logs <broken-pod>

Do not immediately restart everything. Preserve the evidence that distinguishes image pull, scheduling, startup, readiness, crash, and application failures.

Roll back deliberately

Kubernetes stores previous ReplicaSet revisions up to revisionHistoryLimit. The experiment runs:

kubectl --context kind-rollout-lab \
  -n rollout-lab \
  rollout undo deployment/api

kubectl --context kind-rollout-lab \
  -n rollout-lab \
  rollout status deployment/api --timeout=90s

The Deployment returns to v2 and reaches four available replicas. The availability loop remains active until rollback completes.

Rollback safety depends on more than the image. Database migrations, queue schemas, cache formats, and external side effects must remain backward compatible. An application rollback cannot reverse a destructive migration or undo messages already published.

Understand what the PDB does not do

The lab includes:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: rollout-api

The PDB limits voluntary disruptions such as a node drain. It is not the control for Deployment rolling-update availability; maxUnavailable and maxSurge govern that. A PDB also cannot protect against involuntary node failure.

This is why copying a PDB into a manifest does not guarantee uptime. It protects one class of disruption when the eviction path honors it.

Rolling, blue-green, and canary answer different risks

The experiment measures rolling update behavior. Other strategies change the exposure model:

Strategy	Exposure	Rollback path	Main cost
Rolling	Replace pods gradually	Reverse Deployment revision	Old and new versions coexist
Blue-green	Switch traffic between full environments	Switch traffic back	Duplicate environment capacity and state compatibility
Canary	Send a controlled traffic fraction to new version	Remove canary traffic	Requires trustworthy segmentation and automated analysis

Canary is useful only if the team can measure the canary separately and has enough representative traffic. Blue-green is safe only if data contracts work in both directions and the traffic switch is tested. Rolling is simple only when old and new versions can coexist.

Production gates I would add

Use immutable image digests and record the release revision.
Set startup, readiness, and liveness probes from measured application behavior.
Define CPU and memory requests so surge pods can actually schedule.
Verify termination handling and connection draining.
Require backward-compatible database and event-schema changes.
Probe the real ingress path during rollout, not only pod IPs.
Gate on service-level error and latency signals, not pod readiness alone.
Set a progress deadline and make stalled rollout alerts actionable.
Test rollback in a staging environment with realistic state.
Preserve rollout logs, conditions, and external probe evidence.

The Kubernetes Deployment documentation defines rolling-update and rollback behavior, while the probe documentation and PodDisruptionBudget documentation define the health and disruption boundaries used here.