Container scheduling sounds simple in a textbook: pick a machine with available resources, assign the container, monitor it, reschedule on failure. In practice the edge cases multiply quickly, and at 50 million deployments you have seen every one of them at least twice.
Lesson 1: CPU requests lie
CPU limits enforce a hard ceiling; CPU requests express a scheduling preference. Teams routinely set requests at 10 % of actual observed usage — either because they are guessing or because they want their pods to be scheduled on smaller nodes. The result is nodes that are nominally 60 % utilized but CPU-throttled 40 % of the time. Stratum's scheduler ingests actual 30-day p95 CPU usage metrics from the observability layer and uses those for bin-packing decisions, not the declared requests. This alone reduced scheduler-induced throttling by 34 % across the fleet.
Lesson 2: spread constraints matter more than bin-packing
Pure bin-packing maximises utilisation but concentrates replicas. When a physical host loses power, you want your service's six replicas to be on six different hosts, not four replicas on two densely-packed nodes. Stratum enforces topology spread constraints at the zone and rack level by default, and makes them visible in the deploy manifest so teams can tune the balance between utilisation and fault-domain spread.
- Zone spread: replicas are distributed across at least two availability zones whenever the replica count is ≥ 2.
- Host spread: no two replicas of the same service run on the same physical host unless the fleet is smaller than the replica count.
- Rack affinity: for GPU workloads, replicas are kept on the same rack to minimise cross-rack NVLink latency.
Lesson 3: preemption is a double-edged sword
When a high-priority workload cannot be scheduled because all nodes are full, the scheduler can evict lower-priority pods to make room. In theory this is elegant. In practice we have seen preemption cascades: evicting a batch job frees a node, but the batch job's restart triggers a spike in database writes, which causes a memory-hungry analytics service to OOM, which evicts it, which… The Stratum scheduler applies a preemption budget: no more than 15 % of pods in a namespace can be preempted within a five-minute window, and preemption is blocked entirely if the namespace's error rate is already elevated.
At 03:17 on a Tuesday we lost 40 nodes to a networking partition in our primary region. The scheduler rescheduled 12,400 containers in 94 seconds. Four services triggered their automatic rollback thresholds; the other 847 saw no user-visible impact. The post-mortem recommendation was one line: increase the preemption budget timeout from 5 minutes to 10 for stateful services.
Lesson 4: image pull latency kills your SLO
A freshly scheduled container sits in ContainerCreating state until its image is pulled from the registry. At 50 million deployments, image pull latency is one of the top three contributors to deploy time. Stratum maintains a distributed image cache across every node in the fleet: images pulled in the last 72 hours are retained on local disk, and the scheduler preferentially assigns new containers to nodes that already have the image layer cached. For a typical production deploy this reduces image-pull time from 15–45 seconds to under 2 seconds.
Summary: the five scheduling principles we now treat as non-negotiable
- Use observed CPU/memory, not declared requests, for bin-packing.
- Enforce fault-domain spread by default; never rely on teams to remember it.
- Cap preemption cascades with a time-windowed budget.
- Cache images on-node and route new containers toward warm caches.
- Expose all of the above as tunable knobs in the deploy manifest — sensible defaults, full control when needed.