The 2 AM failover we couldn't fix without a deploy
Last March our Singapore origin started serving stale trending lists to the Japanese and Korean edges. The fix was trivial — reroute jp and kr traffic to our Tokyo pull node and drop the weight on Singapore to zero. The problem was how. Our region routing table lived in a PHP config array baked into the deploy artifact. Changing one weight meant a git commit, a CI run, an FTP push to four LiteSpeed nodes, and a Cloudflare cache purge. Twenty minutes of latency on a routing decision that should take two seconds.
At TopVideoHub we aggregate trending video across thirteen Asia-Pacific markets, each with its own language edge — Japanese, Korean, Traditional and Simplified Chinese, Thai, Vietnamese, and more. A "region router" decides which origin node pulls trending data for a given market, how heavily to weight each candidate origin, and which fallback to use when an origin degrades. That decision changes constantly: an origin gets rate-limited, a CJK tokenizer index rebuild slows a node, a holiday traffic spike shifts the optimal topology. Configuration that can only change at deploy time is configuration that's always slightly wrong.
This is the story of moving that routing table into etcd, and the concrete patterns — watches, leases, transactions, and a polyglot read path — that made it safe to change live.
