The Problem We Were Actually Solving
We had built a single-node event-processing pipeline that could run 9,000 events per second with sub-10 ms latency on synthetic data—until real traffic arrived. Our Kafka consumers were dropping 20 % of messages, tail latencies jumped to 400 ms, and heap dumps grew to 2.6 GB within minutes. The observability stack screamed every metric toward Prometheus, but no one could explain why two identical JSON documents, one with a nested array of 128 elements and one without, could balloon our resident set size by 600 MB. We had tuned every JVM switch we knew: -Xmx8G, G1GC, -XX:MaxGCPauseMillis=100, even switched from Log4j to Logback async. Nothing moved the needle. At 02:33 on a Sunday morning the on-call engineers pager lit up with 18 pages for GC overhead > 98 % inside 60 seconds. That was the moment we accepted the uncomfortable truth: the runtime was the constraint, not the code.
What We Tried First (And Why It Failed)
Our first reflex was to deepen the Java tuning ritual. We spun up FlameGraphs with Async Profiler 2.9 and saw 42 % of CPU cycles spent in String.intern()—a relic of a three-year-old decision to deduplicate event IDs globally. We set interned strings aside and moved to a pooled ByteBuffer strategy for network I/O. The GC pauses dropped, but the pause jitter spiked because ByteBuffer allocations still triggered young-gen evacuation. We then tried the Azul Zulu Prime JVM with its pauseless C4 collector. The 90th percentile latency fell to 22 ms, but the 99.9th percentile climbed to 2.1 s because C4s concurrent marking phase fought for memory bandwidth with the event engine. At the same time, our Kubernetes operator was spawning new pods every 6 minutes because the Horizontal Pod Autoscaler watched CPU utilisation fan out from 65 % to 95 % in 30-second windows. The autoscaler treated that spike as a signal to double capacity, so we ended up with 24 pods for a workload that only needed 8. The infrastructure bill ballooned by 300 % and the event order guarantees started to soften. Something had to give.
