breakwater is my resilience toolkit for Node.js — retry, circuit breaker, timeout, bulkhead, rate limiting, stale-while-open caching, all composable, with observability built in. It just hit 1.0.0, and the headline feature is the one no Node library did well: a circuit breaker whose state is shared across every instance of your service.
One instance sees the outage and trips the breaker. The others fail fast immediately, without each having to discover the same outage on their own users. When the cooldown elapses, exactly one of them probes the recovering dependency while the rest keep waiting.
That is the pitch. This post is about the two things I got wrong on the way there, because they were both the kind of wrong that looks right.
Part 1: a state machine where the same name means two different things
A circuit breaker is a tiny state machine. Closed, open, half-open. In one process you protect transitions with nothing at all — JavaScript is single-threaded, and the code between two awaits cannot be interrupted.
