Somewhere between the press releases and the product demos, something went quietly wrong with explainable AI. What began as a serious academic and civil liberties concern about algorithmic opacity has been repackaged, polished, and slotted neatly into enterprise software brochures. The question of whether people deserve to understand why a machine denied them healthcare, flagged them as a fraud risk, or recommended a longer prison sentence has been quietly reframed. It is no longer about rights. It is about features.
The global explainable AI market was valued at approximately 7.79 billion US dollars in 2024, according to Grand View Research, and is projected to reach 21.06 billion dollars by 2030. These are not the figures of a civil liberties movement. This is a growth industry. And the distinction matters enormously, because the people building these tools and the people most harmed by opaque algorithms are almost never the same people. The explainability that corporations are selling is designed for boardrooms and compliance departments, not for the individuals whose lives hang in the balance of an algorithmic output.
When the Algorithm Decides Your Future
To understand why explainability matters, you need only look at what happens when it is absent. In Australia, the Robodebt scheme ran from 2016 to 2019, deploying an automated data-matching algorithm to calculate welfare debts by averaging annual income across fortnights. The method was mathematically crude and, as a 2019 Federal Court ruling determined, legally invalid. No warrant existed in social security law that entitled the administering agency to use income averaging as a proxy for actual income in fortnightly measurement periods. This was known internally because of legal advice received by the Department of Social Security as early as 2014. Yet the algorithm asserted 1.7 billion Australian dollars in debts against 453,000 people. A total of 746 million Australian dollars was wrongfully recovered from 381,000 individuals before the scheme was finally dismantled. The Royal Commission, established in August 2022 under Prime Minister Anthony Albanese, heard testimony from families of young people who had died by suicide after receiving algorithmically generated debt notices they could not understand or contest.
