Out-of-Order Isn't Late: What Building an IoT Ingestion Engine Taught Me About Event-Time Chaos

The scary part about a bug isn't when the numbers look wrong. It's when they look wrong in a way you can talk yourself into accepting.

That's what nearly happened building ingest-sentinel, a telemetry ingestion service exploring how to preserve state correctness when sensor events arrive out of order, duplicated, or late.

The Setup

A FastAPI ingestion endpoint accepts timestamped events from simulated IoT devices. Three invariants must hold regardless of delivery chaos:

Deduplication: The same event ID arriving twice cannot be counted twice in running aggregates.