Over-provisioning feels safe and quietly doubles the bill. The fix is to scale on the signal that actually predicts load.
Scale on the right signal
CPU is a lagging indicator. We autoscale on request concurrency and queue depth, so capacity arrives before latency suffers.
- Concurrency-based autoscaling instead of raw CPU
- Right-sized requests and limits informed by real percentiles
- Bin-packing workloads to reclaim stranded capacity
Prove it with percentiles
Every change is validated against p95 and p99 latency, not averages — cheaper should never mean slower for the unlucky request.