Why Hytale Treasure Hunt Servers Throttle at 100 Players (And How We Fixed It)

The Problem We Were Actually Solving

Our public alpha weekend attracted 4,200 concurrent players across three regions, but any region crossing 100 players saw latency spike to 1.4 s per treasure packet and packet loss hit 22 %. Players reported chests teleporting or duplicating; our ReplayCraft logs showed duplicate LootGenerated events exactly when the Go runtimes GC paused hit 120 ms. The Hytale client expects deterministic state updates every 50 ms; at 108 ms we breached the contract. The logs contained 43,000 lines of runtime: out of memory: cannot allocate 16384-byte stack every 47 seconds, triggered by the arena engine spawning a per-player sync.Pool of 64 KB buffers.

What We Tried First (And Why It Failed)

We rewrote the treasure tracker in Go 1.21 with two optimizations:

Replaced the mutex-protected map[int64]TreasureState with a sharded hash (256 slices), which dropped P99 latency from 1.4 s to 380 ms.