From a generation that spiraled out of control forever, to a morning fix that quietly halved throughput by evening
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
Spanning the weekend, today was about wrapping up things found over the past few days. They had one thing in common: all three had been going wrong quietly, with no visible sign.
Incident 1: One response ran on forever and stalled production
I noticed one of the overnight analysis jobs oddly stuck. It turned out the local LLM server had no upper limit set on response length. Normally a single response ends quickly, but if the model ever got stuck repeating itself without stopping, it could spiral out to tens of thousands of tokens and tens of minutes. What made it worse was that the monitoring setup couldn't catch this case at all — it only got noticed because someone happened to ask "why is this so slow?"
