We built Pinaka to write root cause analyses automatically — read a Jira ticket, search the codebase, trace the failure, post the RCA as a comment. Before putting that in front of real teams, we needed an honest answer to one question: does it actually find the right bug, or does it just sound right?
"Sound right" is the trap with every AI tool that explains why something broke. LLMs are extremely good at producing a plausible-sounding root cause. Plausible and correct are not the same thing, and the gap between them is exactly where a team loses trust in a tool — usually after the first time it sends someone chasing the wrong file for an hour.
So we picked a real bug, in a real production codebase, with a real human-written root cause already on record — and tested two versions of Pinaka against it.
The test setup
We used BullMQ (github.com/taskforcesh/bullmq), a Redis-based job queue library for Node.js with thousands of GitHub stars and heavy production use. It's also part of our own stack, which meant we could judge the RCA quality ourselves, not just trust the AI's confidence score.
