Teaching an Agent to Triage Flaky Tests — and to Know When Not To
Every team with a mature Cypress suite eventually meets the same quiet adversary: the flaky test. Not the broken test—the flaky one. It fails, then passes on retry, so nobody files it, so it lingers. Over time, "just re-run it" stops being a workaround and becomes a habit, and that habit costs you the one thing an end-to-end suite is supposed to give you: confidence.
We decided to stop treating flake as background noise and start treating it as a signal worth acting on every single morning. So we built an autonomous agent to do the triage a human never quite finds time for. What we learned building it had less to do with automation than we expected, and a lot to do with judgment.
Turning a Number Nobody Reads into a Decision
The data was already there. Cypress Cloud records a flaky_test_count on every CI run and can tell you precisely which tests flaked—the error name, the message, the failing line of the stack, attempt by attempt. That is a rich signal. It was also, for us, a number that scrolled past unread.
