The promise was straightforward enough. Large language models, trained on the sum total of medical literature, would help emergency physicians triage patients faster, assist radiologists in catching what the human eye missed, and give overwhelmed clinicians a second opinion when the waiting room was full and the clock was running. The reality, according to a growing body of peer-reviewed research, is considerably more uncomfortable. The most capable AI systems available today do not simply reflect the biases embedded in their training data. They amplify them, sometimes dramatically, and they do so in clinical contexts where the consequences land on real human bodies.
In September 2025, a team of researchers led by Mahmud Omar and Eyal Klang at the Icahn School of Medicine at Mount Sinai posted a preprint on medRxiv that tested OpenAI's GPT-5 across 500 physician-validated emergency department vignettes. Each case was replayed 32 times, with the only variable being the sociodemographic label attached to the patient: Black, white, low-income, high-income, LGBTQIA+, unhoused, and so on. The clinical details remained identical. The model's recommendations did not.
GPT-5 showed no improvement in sociodemographic-linked decision variation compared with its predecessor, GPT-4o. On several measures, it was worse. The model assigned higher urgency and recommended less advanced testing for historically marginalised groups. Most striking was the mental health screening disparity: several LGBTQIA+ labels were flagged for mental health evaluation in 100 per cent of cases, compared with roughly 41 to 73 per cent for comparable demographic groups under GPT-4o. The clinical presentation was the same. The only thing that changed was who the patient was described as being.
