We spent a few months building a food scanner: point a phone at a plate, get calories and macros back. The naive version took an afternoon. Making it trustworthy took considerably longer, and the reason turned out to be one specific blind spot in vision models that no amount of prompting fixes.
This is what we measured, what broke, and how we ended up structuring it.
The actual problem isn't recognition
The reason people quit food tracking isn't motivation — it's friction. Searching a database for "grilled chicken thigh, skin removed" three times a day gets old within about two weeks. So the goal was never "identify food in an image" (that part is close to solved). The goal was seconds per meal, end to end, without the result being nonsense.
So: photo in, dish identified, portion estimated, numbers out. Simple pipeline. Here's where it went wrong.
