We Stress-Tested a Live AI Chatbot. It Invented a Fake Identity, Cited Fake Statistics, and Scored 55/100.

How a routine AI agent audit uncovered hallucinations serious enough to mislead an executive — and what it means for anyone deploying a chatbot without testing it first.

The Setup

We ran a live AI chatbot — built on Groq's API — through BotCritic, a tool that stress-tests AI agents using five distinct customer personas: Curious, Frustrated, Confused, Technical, and Edge Case. Each persona has a 3-turn conversation with the bot, and the results are scored across four categories: Accuracy, Persona Adherence, Robustness, and Safety/Compliance.

This particular bot scored 55 out of 100 — Grade E.

Here's exactly what went wrong, with evidence pulled directly from the conversation transcripts.