The most important lesson we learned is this: you need to build strong foundations of system context before you can trust any automated reasoning or analysis. If your system lacks basic visibility—like topology, deploy tracking, or incident history—it simply cannot point engineers in the right direction, no matter how smart it seems.
In our first attempt at building an incident detection system, we made some early missteps. We focused on reasoning before establishing a foundation of system context. There was no topology, deploy tracking, or incident history. As a result, the system often misidentified the cause of incidents, frequently blaming the most recently deployed service regardless of the real issue. It produced plausible-sounding explanations, but they were frequently inaccurate.
Engineers agreed with the system’s top hypothesis for only about 35% of incidents. The issue wasn’t just accuracy—it was the confidence in wrong answers, which led to costly mistakes. Within two weeks, three engineers stopped using it entirely. One person said he trusted it less than reviewing the raw trace data himself, which was disappointing since the tool was supposed to help avoid that manual work.
