The Problem We Were Actually Solving
I was the lead systems engineer on a project to build a massive online treasure hunt platform, where thousands of users would be competing in real-time to solve puzzles and find hidden treasures. The platform had to handle a huge influx of requests, process them quickly, and scale seamlessly to avoid crashing under the load. We chose to use Rust as our programming language, due to its focus on performance and memory safety, but I was aware that it would come with a steep learning curve. Our team spent weeks designing the architecture, and we thought we had it all figured out, but it was not until we started load testing that we realized our event handling mechanism was the bottleneck. We were using a naive approach to handle events, where every incoming request would spawn a new thread, and this was leading to a huge number of context switches, slowing down the entire system.
What We Tried First (And Why It Failed)
At first, we tried to optimize the event handling mechanism by using a thread pool, where a fixed number of threads would be reused to handle incoming requests. This approach seemed to work well for small-scale testing, but when we scaled up to thousands of concurrent users, the system started to degrade. The thread pool was not able to keep up with the influx of requests, and we started to see a significant increase in latency. We used the perf tool to profile our application, and the output showed that the majority of the time was spent in the thread pool implementation, waiting for threads to become available. The allocation counts were also through the roof, with over 100,000 allocations per second, which was causing a significant amount of memory churn. I knew we had to rethink our approach to event handling.
