A friend of mine runs security at a mid-sized fintech. About six weeks ago she called me, mildly furious, because her board had asked a question she couldn't answer: "How many AI models are running in production, and what data do they touch?"
She did what any reasonable CISO would do. Pulled the SBOM exports out of her SCA tool. Cross-referenced with the cloud asset inventory. Asked the platform team. Asked the ML team. Three different answers. The SCA tool said they had four models because it counted the transformers package four times. The cloud inventory found eleven SageMaker endpoints, two of which had been decommissioned a year ago but still had IAM roles attached. The ML team, who actually build the things, said the real number was somewhere around thirty — including a couple of fine-tuned Llama variants nobody on the security team had ever heard of, running on a self-hosted vLLM cluster behind an internal load balancer.
The part that really got her was the fine-tunes. She had no record of what base model they came from, what data was used to tune them, who approved the training run, or what guardrails were in place at inference. Her DLP team had no idea customer PII had been in the training corpus. Her compliance team had been telling auditors for two quarters that the company used "only approved foundation models from approved providers."
