When Your AI Medical Assistant Turns Into a Chemistry Tutor from Hell: A Lesson in LLM Overreliance
Hey dev.to!
I’m Jack, and my latest hobby beside gardening and cooking, I love poking at AI models in my sandbox to see what breaks. This time, I went after LLM09: Overreliance – one of the scarier items in the OWASP Top 10 for LLM Applications. The test subject? Gemini 3.5 Flash, wrapped in the persona of a helpful AI Medical Assistant.
Spoiler: it happily told me to create toxic gas to clear my sinuses. Let’s break down what happened, why it’s a massive facepalm, and how to fix it.
First, What the Heck Is “Overreliance” (LLM09)?
