Migrating 50+ live tenants across data centers, with zero downtime

The problem

Sixty-odd dealerships depend on these sites to sell: catalog, leads, e-commerce. Downtime is not a number on a dashboard — it's a customer who opened the store's site, watched it not load, and went to buy from the competition.

All of production ran in a single data center abroad — users in Brazil, servers on a different continent, and a fleet expected to double in size in short order. The migration wasn't just a change of address: it was the chance to go from one single point of failure to two geo-separated data centers with automatic failover — lower latency, real DR, room to grow.

The obvious answer — "migrate overnight on a Sunday" — didn't work, for two reasons. First, people buy cars in the middle of the night too. Second, a platform team of one can't run on heroics: if the plan requires staying up all night and hoping, the plan is wrong. I needed a zero-downtime migration in small, reversible steps, with a cheap rollback at every one of them.

The strategy: the destination gets hot before anything flips

The principle behind everything: no traffic moves until the destination is running mirror production, with data synchronized and validated.

1. Pre-seed: the destination runs as a replica for weeks. The full stack came up in the two new DCs and spent almost two weeks "dark": PostgreSQL as a standby of the Patroni cluster (continuous streaming replication), MongoDB replica set members, a Redis replica behind Sentinel. Data flowed to the destination in real time — without a single byte written there.
2. An identical application, by definition. Services were provisioned at the destination by the same GitOps control plane — same containers, same versioned config. There is no "I forgot to copy that file" when the deploy is declarative.
3. Cutover in waves, from disposable to critical. First the internal platform services (analytics, dashboards). Then a canary: a single low-risk tenant, flipped on its own and validated end to end. Only then the revenue.
4. The cutover itself: atomic by design. Every tenant hostname points by CNAME at a single anchor record — flipping one record moves 100+ hostnames at once, and the rollback is the same record flipped back, instantly. The Patroni standby promotion was coordinated with the flip, under one hard rule: zero replication lag, or abort — and Patroni itself refuses the promotion if the lag isn't zero.
5. Gradual ramp? Considered and rejected. Serving half the traffic from each DC during the transition sounds safer — but it reopens exactly the risk the architecture had closed: two sides believing they can write. Atomic cutover with instant rollback > gradualism with split-brain risk. A conscious trade-off.
6. The legacy: traffic now follows the database. After the dust settled, the tenants were moved behind a Cloudflare Load Balancer whose health check only returns 200 in the data center where PostgreSQL is the writable leader. Since Patroni guarantees a single leader, traffic routing follows database failover automatically — no human intervention, detection in ~1–2 min, RPO measured in seconds.

"Zero downtime" is proven, not declared

Everyone says they migrated "with no downtime". Almost nobody instruments the claim. Here the validation was adversarial — the goal was to prove it broke, and fail:

• Continuous HTTP sampling across the cutover: 400+ requests, zero responses other than 200.
• Packet capture on every flip (tcpdump at the edge): SYNs landing on the right destination, no dropped connections, no orphaned RSTs.
• External health checks from ~280 Cloudflare PoPs — the view from the outside, from the whole world, not the server's opinion of itself.
• Per-tenant dashboards (error rate, p95 latency) before, during and after.
• And the final insurance: the source DC stayed alive, read-only, for weeks — rollback available through the entire soak. It was never triggered. A good rollback plan is the one you write carefully and never use.

If you didn't measure it, you didn't migrate with zero downtime — you migrated with luck.

The incident the drill revealed

A migration in production is not the end of the project. Days after the cutover, I ran a failover drill: send 100% of production traffic to the standby — on purpose, on a good day, to prove the automatic failover could take a bad one.

It couldn't. The standby was cold — empty caches — and it melted: a single site container at 138% CPU, load 168 on a 24-core host, while the warm primary served the same traffic at a load under 10. Failback via runbook in ~9 minutes, nothing lost beyond the spike. And then, root cause analysis.

The root cause wasn't in the infrastructure — it was in the behavior of Next.js SSR with a cold cache, a self-perpetuating runaway engine: the cold render blows past an internal timeout and becomes an error; the framework only caches OK responses, so the cache never warms up; every subsequent request re-renders from scratch — and the CPU feeds its own fire. A cold environment wasn't a transient state that "passes": it was a stable trap. It's the kind of root cause you only find when you understand the runtime you operate, not just the VMs underneath it.

The fix came on two tracks, documented in a postmortem:

• Infra: a per-tenant pre-warmer that heats the destination before any traffic flip — warm-up became a failover precondition in the runbook, not a detail. Plus missing indexes and query parallelization on the critical path.
• Application: graceful degradation instead of errors, a concurrency cap + circuit breaker, single-flight renders, backend caching — validated afterwards under load with a cold cache: 100% 200 responses.

Lessons

• A planned rollback is worth more than courage. Every step had a cheap way back; which is exactly why none of them needed it.
• Adversarial validation. Try to prove it failed. If you can't, then you get to claim the "zero downtime".
• Gradual is not always safer. A weighted ramp would have reopened the split-brain risk; the atomic cutover with instant rollback was the conservative option — even if it looked like the bold one.
• A "ready" standby and a "hot" standby are different things. It took a drill to reveal that — and that is exactly what drills are for. The incident became a postmortem, the postmortem became a runbook, and the failover ended up better than the migration delivered it.