This is Part 2 of a two-part series. Part 1 covered the problem, the CLIP baseline, and BLIP. This part covers the OCR + VQA pipeline that replaced them for fine-grained product feature matching.
The Problem with Scores
At the end of Part 1, the conclusion was clear: CLIP and BLIP are good at catching obvious mismatches but fail at the fine-grained ones that matter most in a marketplace. A wrong brand scores 0.79 on BLIP ITM — comfortably inside "Likely match" territory. A wrong size is nearly invisible to both models.
The reason is architectural. Both CLIP and BLIP compare an image and a description as whole units. They never ask: does the brand in the image match the brand in the description? They encode everything together and produce a single score. That score cannot tell you which attribute caused a mismatch.
What a marketplace actually needs is not a similarity score. It needs answers to specific questions:
