So I was messing around with a little side project, a chatbot that answers questions about the World Cup. Fun, low stakes, nobody's paycheck depends on it.
Someone asked it "has the tournament ended yet?"
It said no. Confidently. Wrong. It had actually ended weeks earlier.
Fine, models get things wrong sometimes, that's not exactly breaking news. What actually got me was the next question: why? Did it even try to check? Did it just guess from something it remembered? I genuinely had no way to find out. The model doesn't keep a diary. It answers, and that's it, the reasoning just evaporates.
And that's true whether you're building a hobby project or something with real users. No logs, no "here's exactly what I looked at before I answered," nothing. You're stuck debugging a black box with screenshots and vibes.
