A much-discussed MIT study from this past year suggested that more than 95% of artificial intelligence initiatives never make it to production or deliver measurable ROI."In healthcare, I would expect that number to be even higher," says Michael Privat, chief data and engineering officer of Availity, an IT services and consulting firm. "The pattern I see is consistent," Privat explains. "Teams build prototypes quickly and generate excitement, but the moment they try to operationalize those solutions, everything changes. Now you're dealing with real patient data, regulatory requirements, governance and security risks."We asked Privat to share some more thoughts about why and where AI projects stall or get abandoned in healthcare, and what can be done to realize a higher success rate. Here's what he had to say.Q. What are some must-haves to help ensure success in operationalizing health AI beyond pilots?A. Most companies skip the substrate. They buy a model, plug it into a notebook, and call it a pilot. Operationalizing health AI means building everything underneath first. That's data lineage and identity resolution at production scale, audit trails that satisfy HIPAA and SOC 2, and a continuous evaluation framework. Drift is real. A model that works in January will quietly stop working by July, if nobody watches it. It means one accountable owner per use case, on the business or clinical side, not the AI team. A pilot is a vendor demo until someone outside engineering can describe the outcome it produces in clinical or financial terms, without using the word "AI."Q. What are the pitfalls to avoid?A. The biggest one is using AI as an additive layer on top of a process that was already broken. AI amplifies what exists. If your prior authorization workflow has 14 handoffs and a two-week queue, generative AI will produce more material for the same queue. You haven't solved anything – you've automated the part that wasn't the bottleneck. The second pitfall is scoping pilots to demo – not to deliver. That means clean data, hand-picked cases, a special team and no integration constraints. Then, when someone tries to put it into the actual claims pipeline, the cost of integration eats every dollar of margin. The third is letting the vendor define success. If the only people measuring the model are the people selling the model, you don't have an evaluation framework. You have marketing.Q. What makes a lasting initiative with staying power, as opposed to one that's not worth continuing?A. The initiatives that last have an "honest owner" and a real metric. By honest owner, I mean someone on the business or clinical side who has to defend the outcome, not the engineer who built it and not the vendor who sold it. By real metric, I mean a number that ties to the profit and loss, or to patient outcomes – not engagement and not queries served.The lasting initiatives are also designed to be retired. They solve a defined problem, and when the problem changes, they end or evolve. The ones that don't last become infrastructure projects with no exit criteria, championed by an executive who left two years ago, still consuming budget because nobody knows how to shut them down. If you can't describe what would make this initiative finished, you're not running an initiative. You're funding a department.Q. What does it take to scale across an organization?A. Real scaling is consolidation – one inference platform, one evaluation framework, one identity and data model, and one set of guardrails. We run on Amazon Web Services Bedrock across roughly 60 engineering teams. That only works because no team is building its own stack. Standardization is what makes AI scale; novelty is what makes it expensive.It also takes governance as infrastructure, not governance as a committee. If every new use case requires three reviews, two architecture diagrams and a steering meeting, you don't have governance. You have a toll booth. Capability gates, behavioral guardrails and audit trails enforce policy at the platform level so teams ship without waiting on humans.In the end, scaling AI, just like scaling anything else, is mostly about economics: If the operational costs are too high, because there's too much diversity, then it won't scale. If the costs of being wrong are too high, then it won't scale. People talk about economies of scale. It applies here, too.Q. How can organizations ensure end-to-end observability after scaling AI?A. End-to-end observability for AI is not the same thing as monitoring. Monitoring tells you the model is responding. Observability tells you whether the response is still right. Most organizations only solve the first one.What you need: Input drift detection, so you know when the data feeding the model has shifted from what it was trained on. Output quality signals that are tied to a continuous evaluation harness – not a benchmark from launch day that nobody has run since. Latency and cost per call, because both will silently triple if nobody is watching. Full audit trails – every prompt, every response and every decision. In healthcare, you will be asked to reproduce why a specific case was handled a specific way, and "the model said so" is not an answer that survives a regulator.The thing that separates real observability from theater is whether your metrics tie to a business outcome. If your dashboards show 100% uptime and rising query volume while the clinical team is quietly working around the model because the answers got worse, you have monitoring. You don't have observability. Pair every model with a downstream metric that breaks when the model breaks. Otherwise, you're watching the wrong thing.Finally, understand that the model you launched a year ago is not the model you have today. Observability is what tells you the difference. Without it, you're paying for software you can no longer describe.Andrea Fox is senior editor of Healthcare IT News.Email: [email protected]Healthcare IT News is a HIMSS Media publication.