From stuffing JSON into Claude to GraphRAG, hybrid search, CRAG, and adversarial evaluation — the complete honest account
The problem with FinMentor started before I had the vocabulary to describe it...
Users were asking reasonable questions about their portfolios. The system was answering them. Some answers were right. Some answers were wrong. And I couldn't explain the pattern because I hadn't looked at what was actually flowing into the model.
When I looked: every query was receiving the full IBKR portfolio snapshot. JSON format. Five positions, monthly P&L, thirty transactions, account metadata. The same 847 tokens regardless of what was asked. A question about sector concentration got the full transaction history. A question about a single ticker got every other position. Maybe 10% of the context was relevant to any given question. The other 90% was noise competing for attention and billing me for the privilege.
I wasn't doing retrieval. I was doing copy-paste with extra steps.
