The naive approach to zero-downtime deployment is seductively simple: spin up the new version behind a load balancer, wait for a health check to pass, then cut traffic over. That works 99 % of the time. The remaining 1 % is where production incidents are born — and where we spent the last three years building safeguards.
What actually causes downtime
In our data, the three most common causes of deploy-related downtime are not application crashes but infrastructure timing mismatches: a load balancer sending traffic to a container whose process has started but whose application is still warming up; a database connection pool exhausted during a rolling restart; and in-flight requests dropped because a container was terminated before all responses were flushed. Each of these is invisible to a basic TCP health check.
The Stratum deploy pipeline
Stratum's deploy engine runs four sequential gates before any pod is added to the active load-balancing pool:
- Startup probe. A dedicated HTTP endpoint that returns 200 only once the application has completed its full boot sequence — database connection established, caches warmed, background workers running. The pod is quarantined until this probe clears, regardless of the process being alive.
- Readiness probe. A continuous liveness signal polled every two seconds. The pod is pulled from the pool the moment a probe fails and returned only after three consecutive successes. This gives the application time to self-heal under transient conditions without triggering a full rollback.
- Connection drain. When a pod is being replaced, the load-balancer stops routing new connections to it immediately, but existing long-lived connections (websockets, gRPC streams, SSE channels) are given a configurable drain window — 30 seconds by default, up to 5 minutes for batch-processing workers — before the pod is terminated.
- Canary budget. The first 5 % of traffic is routed to the new version for a configurable observation window. Error rate, p99 latency, and saturation are compared against a rolling baseline. If any metric breaches the threshold, the canary is pulled and the previous version stays primary.
Automatic rollback — the last line of defence
Stratum's rollback system is event-driven, not operator-driven. It monitors the 4xx/5xx rate of the new version relative to the old version in real time. If the error rate climbs more than 0.5 percentage points above baseline within the first five minutes of a deploy, the system rolls back without any human action: it shifts traffic back to the previous artifact, pages the on-call engineer, and retains the new container image so the team can inspect it before retrying.
We have triggered automatic rollback 1,847 times in the past twelve months. In 94 % of cases the on-call team agreed afterwards that the rollback was the right call. The other 6 % were false positives we tuned away by tightening the baseline window.
Lessons from 10 billion requests
The single biggest insight from operating at this scale: downtime is almost never caused by a bad application build. It is caused by the gap between when your infrastructure thinks the new version is ready and when it actually is. Closing that gap — through startup probes, connection draining, and metric-gated canaries — is what separates a deploy that looks zero-downtime from one that genuinely is.
Key configuration defaults we recommend
- Startup probe timeout: 60 s (configurable up to 10 min for slow-booting JVM services).
- Readiness probe interval: 2 s, failure threshold: 1, success threshold: 3.
- Connection drain window: 30 s default; set to max expected request duration + 5 s buffer.
- Canary window: 5 min minimum; 15 min recommended for stateful services.
- Rollback trigger: error-rate delta > 0.5 pp or p99 latency delta > 200 ms sustained for 60 s.
Every one of these defaults is tunable per service in your stratum.yaml deploy manifest. Start with the defaults; tighten once you understand your service's boot-time and error-rate baseline.