How we turned always-on Kafka sinks into on-demand workers that shrug off nightly bombardments — by scaling on the right signal, tuning per-pod drain rate, and keeping autoscaling from sabotaging itself.
Every number in this post is measured from a local lab you can run yourself — the full code is on GitHub, and the Appendix has the commands.
The problem
We run a fleet of Kafka sinks — consumer services that read change events from Kafka, apply business logic, and write the result into a service-local database as a query-friendly materialized view. It keeps reads fast and independent from upstream systems, and it's a great pattern. But the workload has an awkward shape.
Most sinks are idle most of the day, then buried in minutes. Traffic isn't steady: changes arrive in bursts, usually from nightly imports or CDC jobs. The rest of the day the topic is quiet.
