To audit is literally to listen — the word comes from the Latin verb audire, to hear. When auditors review tax returns or business records, they listen for dissonance. Do the numbers add up? Do the records match reality? Do claims on paper hold up on closer inspection?
As AI reshapes human affairs, a similar need has emerged: to measure the outputs and understand the impacts of socially influential AI systems including large language models, algorithmic recommendations and other automated systems.
AI audits offer a way to evaluate how these systems behave in the real world, determining whether they function as designed, whether that design is flawed and whether the systems produce harms their creators did not anticipate.
Danaé Metaxa, Raj and Neera Singh Term Assistant Professor in Computer and Information Science, and co-authors recently published “Auditing AI,” a new, open-access book that explains how AI audits can help policymakers and everyday users better understand AI systems.
In a recent conversation, Metaxa discussed what motivated them to co-write the book, what AI audits can and cannot reveal, and how their lab at Penn Engineering is working to make AI systems more accountable.
