A friend runs security at a mid-size fintech. Last month she got a Slack ping from her GRC lead: the EU AI Act auditor wanted a full inventory of AI systems in production, plus provenance for each model, plus a mapping to the risk tiers by Friday. She figured, fine, we have an SBOM pipeline, we have a model registry, we have vendor questionnaires. Two hours of work.
Nine days later she was still reconciling.
The model registry knew about the four models the data science team had shipped. It did not know about the Copilot-style assistant a product team had built by wiring their app to Claude through a wrapper library. It did not know about the two internal RAG systems that had swapped their embedding model in April without telling anyone. It did not know that the "AI-powered search" feature was actually a fine-tuned Llama variant a contractor had left running on a GPU box in a colo, or that the customer support tool their vendor had "upgraded" now shipped user tickets to a third-party inference API in Singapore.
The part that got me when she told the story: the auditor wasn't even being aggressive. He was asking the baseline questions. Model name. Model version. Training data provenance. Deployment location. Data flows in. Data flows out. Who owns it. What it can do. What it can't do. And she couldn't answer half of them without hunting through Jira, Terraform, vendor contracts, and two Notion pages that hadn't been updated since 2024.
