Quality Engineering and software leaders are traditionally tasked with answering a deceptively simple question: Is this software good enough to ship? For years, the industry answered this through regression testing, automation suites, and staging environments designed to prevent bugs. However, recent insights into R&D data are supporting an important paradigm shift - one that has been challenging us to expand our definition of "quality" from merely ensuring the product is built right to ensuring we are building the right thing..
When analysing engineering throughput, an organisation's experimentation data can look alarming to some on the leadership team. We have all seen those confused faces opening THAT email. If a significant percentage of new features or ideas don't end up fully launched to production, it might look like an engineering bottleneck/waste of time. But from a modern quality perspective, that gap represents the ultimate safety net.
The Three-Tiered Quality Ledger
Velocity is a painful word to some engineers, but in Leadership to really understand the value of engineering velocity, we need that data. We need to move away from binary "pass/fail" or "deployed/abandoned" metrics. Instead, we should anchor our measurement by classifying each completed experiment into one of three outcomes: shipped, rolled back, or informed a decision. If we only measure the "Shipped" bucket, we are severely miscalculating the ROI of our engineering efforts:
