Last year I was on a team that pushed 40 million events per day through Kafka. We had consumer lag...
Last year I was on a team that pushed 40 million events per day through Kafka. We had consumer lag alerts, rebalancing incidents, and a whole runbook for when the broker got behind. I understood how to operate Kafka. But I did not understand how Kafka works.
So I built a tiny one. No dependencies. No Zookeeper. No JVM. Just Python and the core ideas.
Here is what I learned.
The Three Things Kafka Actually Does
People say "Kafka is a message queue." That is not quite right.
Kafka's pieces are each simple — partitions, keys, consumer groups, offsets, lag — but together they...
Apache Kafka is widely recognized as the go-to way system for real-time event streaming. Modern...
Stop Treating Kafka like a queue! A story about silent lag, vanishing messages, and the...
A survival guide for when everything goes wrong in production. There's a moment every engineer...
Message Queues Explained: Why LinkedIn Built Kafka and Changed Async Communication...
If you're learning data engineering, you'll probably meet Apache Kafka very early. You'll see it in...
