Tracked down and fixed the real cause of a memory problem, and accidentally killed production while doing it
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
Today I finally traced a problem that had been a mystery for weeks all the way down to its root cause and fixed it. Along the way I also made two mistakes.
Weeks of mysterious overnight crashes — found the real culprit
For a while now, the dev environment had been dying at a specific hour every early morning (I'd already moved core processes into separate services independent of the dev environment because of this, but that only masked the symptom rather than fixing the cause). Today, digging deeper into the logs with another AI, I finally found the real cause.
