The Problem We Were Actually Solving
Treasure hunts in Hytale arent just about generating loot. Theyre about generating simultaneous loot across thousands of players while keeping the world state consistent. We started with the assumption that events are stateless notifications: a hunt starts, we fire an event, clients react. That model worked fine when we had 200 concurrent players. At 2,000 players, the event bus turned into a 40 MB/s firehose of JSON blobs. Each loot drop required serializing the entire chunk state—blocks, entities, metadata—so clients could render the drop in real time. The JVMs G1GC couldnt handle the allocation rate. Every 47 minutes, a GC cycle would pause for 4.2 seconds, the chunk cache would fragment, and the server would hard crash with an OutOfMemoryError in net.minecraft.server.MinecraftServer#processQueue.
The real problem wasnt the hunt logic. It was the architectural laziness of treating events as a catch-all glue layer instead of a boundary layer with explicit interfaces.
What We Tried First (And Why It Failed)
We tried Kafka as the event bus. The plan was to shard hunts by region and stream loot drops as compacted topics. The first run worked for about 6 hours before the compacted topics started to bloat. Each hunt was generating 700 KB of serialized chunk state per drop. At 30 drops per hunt per minute, thats 21 MB per hunt per minute. With 400 active hunts, the brokers couldnt keep up. The lag grew to 12 seconds, clients started rubber-banding, and we got a flood of Discord reports: You sank my boat! The event stream was now the bottleneck, not the event source.
