The first version of our network was a single origin region with a CDN bolted on for static assets. It worked until it didn't. A single region means a single blast radius, and when that region had a seven-minute partial outage, we lost customers. That was the forcing function for the first rewrite.
Architecture v1: CDN + single origin
Static assets cached globally; API requests routed to one region. Simple to operate, catastrophic when the origin region degrades. Lesson: a CDN is not a resilience strategy. It accelerates the happy path and makes the failure path slower to diagnose.
Architecture v2: active-passive multi-region
We added a second region as a hot standby. DNS failover would promote it if the primary became unhealthy. In practice, DNS TTLs meant failover took 90–180 seconds. During that window, traffic was split between a degraded primary and a warming secondary. The cure was almost as bad as the disease. Lesson: DNS-based failover is a last resort, not a resilience primitive.
Architecture v3: anycast + BGP
The third architecture is the one running today. Stratum announces a single anycast IP prefix from every point of presence (PoP) simultaneously via BGP. When a user's packet reaches the internet backbone, the routing protocol itself selects the nearest PoP based on AS-path length. No application-layer routing decisions; no DNS TTL lag; no warm-up period for a secondary region.
- 20 ms routing. The BGP path to the nearest PoP is established in milliseconds. We measure median routing time — from packet entering the public internet to arriving at a Stratum edge node — at 18 ms globally.
- Instant failover. If a PoP loses BGP peering, the prefix withdrawal propagates across the routing table in under 10 seconds. Traffic naturally flows to the next-nearest PoP with no application change.
- DDoS absorption. A volumetric attack against our anycast prefix is absorbed across the entire PoP fleet. No single region sees more than its fair share of attack traffic.
The health layer on top of BGP
Anycast routing optimises for network proximity, not application health. We layer a real-time health system on top: each PoP runs a local health agent that continuously probes the upstream origin fleet (compute nodes, databases, caches). If a PoP's upstream health score drops below 80 %, the edge node withdraws its BGP announcement. Traffic shifts to a healthy PoP in the same region or the next closest one, entirely transparently.
During a fibre cut in one of our European co-location facilities, the affected PoP withdrew its BGP prefix within 8 seconds. All 47,000 active connections in that PoP were re-established through the next-nearest PoP within the TCP retransmission window. Zero support tickets were filed for the incident.
What the three rewrites taught us
- Resilience must be in the routing layer, not the application layer. Application-level failover is always too slow.
- Simplicity at the data-plane level (one anycast prefix, BGP handles the rest) enables complexity at the control-plane level (health scoring, selective withdrawal) without making operations harder.
- Measure the actual user-perceived routing time, not synthetic pings. Our first two architectures looked fast in synthetic tests and were slow in practice.
- Every architecture decision that felt like a shortcut in year one became the source of an incident in year two.
The anycast network today covers 28 PoPs across 5 continents. The target for the next 18 months is 40 PoPs, with every new PoP required to demonstrate sub-15 ms routing to the nearest metro area before it enters production.