Originally published at nlqdb.com/blog
Our acceptance walkers pin literal UI strings on purpose. A walker asserts the homepage placeholder reads exactly what we ship, that a heading says the words we wrote, that the query composer is where we put it. Drift fails the walk loudly — and a loud failure the moment the surface changes underneath you is precisely the regression detector you want. Silent tolerance is how a broken flow ships green.
Then three things happened in one week. A homepage redesign moved the goal input to a different page. A copy edit reworded a heading. The MCP catalog additively grew two tools. None of these broke the product. All three broke the walkers — which dutifully reported 0/9, and kept reporting it for a week.
The trap isn't the literal assertions
The reflex is to blame the pinned strings and loosen them into fuzzy matches. That's the wrong lesson. The literals did their job: the surface changed, the walk went red. The actual cost is that a red which mixes "the product broke" with "the test went stale" takes a full manual triage to disentangle — and ours contained both at once. Two flows were red from pure test-drift. One flow was red from a real production wall. Same 0/9. You cannot tell which is which from the number, so every red costs you the same expensive human read regardless of whether anything is actually wrong.
