By Subba Rao Katragadda, Senior Principal Data Engineer, Johnson & Johnson.gettyPredictive machine learning is no longer a novel topic of conversation in healthcare. Models that identify patients at risk of deterioration, readmission, sepsis, medication complications or missed follow-up appointments are appearing in hospitals and health systems across the country.The potential benefits are hard to ignore. Machine learning models trained on clinical data can identify patterns and anomalies much faster than humans. That information can help care teams spend more time where it matters most. But before organizations deploy any predictive model, they should ask, "What happens if the model is unsure?"This question may matter more than how accurately the model predicts.In many high-risk healthcare decisions, a prediction should not always trigger an automated action. Sometimes, the best response is for the system to pause, acknowledge that it does not have enough information and escalate the case to a clinician. This is not a failure of machine learning. It is an example of responsible design.Why Prediction Is Not The Same As CertaintyPredictive models are helpful because they estimate risk based on patterns in historical data. They can predict the likelihood that a patient will experience a negative outcome, such as deterioration, missed appointments, readmissions, emergency department visits or complications.However, a risk score is not a diagnosis. It never considers every variable. It cannot capture the entirety of a patient or their situation, and it should not substitute for a trained medical professional’s clinical judgment.A machine learning model can highlight anomalies based on the information available to it, but healthcare data is often incomplete. Patients may receive care outside the system. Lab results and medication lists can fall behind. Socioeconomic factors, family dynamics and personal behavior rarely appear in a patient’s electronic health record.Low-confidence predictions are not the only problem. The larger issue is when models make high-confidence predictions based on incomplete information. That is why healthcare organizations should design their predictive systems to recognize and respond to uncertainty.The Case For A Safe Escalation PathIf predictive models should know when to pause, how will they know when to escalate? To answer that question, systems need a clear escalation path.An escalation path is a plan for what happens when the model identifies a high-risk patient, has low confidence in a prediction or encounters a patient whose information does not closely match the data the model was trained on.Consider a model that flags a patient as high risk for sepsis based on vital signs, lab values and clinical notes. That prediction should not automatically trigger a treatment decision. Instead, it should alert a responsible clinician, provide context and allow for rapid review.This logic applies to any model predicting risk, whether it is readmission, medication complications or patient deterioration. A model can flag who needs attention. Clinicians should determine the appropriate response.When designed responsibly, predictive modeling can create a healthy relationship between people and technology. Machine learning should automate prioritization, not replace human expertise.When A Model Should PauseThere are specific situations where a predictive model should invoke its escalation path and request human review. Engage human oversight when:• The model is unsure. Predictions made near the model’s decision threshold should automatically trigger review.• The data is incomplete or stale. Models should be transparent when they are making predictions based on missing information, such as outdated medication lists, missing lab results or incomplete clinical histories.• The patient does not fit the data. Models should pay extra attention to patients with rare diseases, complex comorbidities or healthcare histories that look significantly different from the data used for training.• The stakes are high. Organizations should consider the consequences of allowing a model to make an incorrect decision. The greater the potential risk, the more oversight that decision should require.Predictive models should invoke their escalation path in each of these situations. Yet too often, they continue making predictions or default to what the system believes is best. This is not something organizations should think about after deployment. These responses should be built into the model’s design.Building The Handoff Into The WorkflowSuccessful predictive systems should focus less on generating alerts and more on generating useful handoffs.Clinicians receive a large volume of alerts every day. Many are ignored, cause unnecessary delays or are dismissed as noise. But if the right people receive a well-structured alert, along with insight into how it was derived and why action may be required, it can support faster and more informed decisions.Effective alerts should share:• The factors that influenced the risk score• The model’s confidence level• Who should review the alert and when• Recommended next stepsThis is where thoughtful workflow design is as important as thoughtful model design.Additionally, teams should consider factors such as who should receive the alert, what information they need to assess its validity, how quickly they need to respond, what action they can take and how this information will be captured to improve the model. Without answers to these questions, predictive analytics can create inefficiencies instead of improving value.Measuring Safety, Not Just AccuracyMost teams look at accuracy, precision, recall and other statistical measures to understand model performance. These metrics are important, but organizations should also measure:• The frequency of uncertain escalations• Clinician agreement or override rates• Model performance across patient populations• Improvements in clinical outcomes after deploymentBy tracking these factors, teams can better measure a model’s safety in addition to its accuracy.A More Responsible Future For Predictive HealthcarePredictive healthcare should not be about automation for automation’s sake. It should be about how machines can responsibly support the clinicians who are ultimately accountable for decisions.The best predictive systems will know when to act autonomously, ask for more information, escalate to a clinician and allow clinicians to make the final decision.There will always be a role for automation in healthcare, but the value of predictive machine learning will be defined by when it knows to pause.​​Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?