A question I can't stop chewing on, and I want your answer more than I want your agreement.
How would you test whether an AI actually understands your codebase?
Not whether it writes good code. That's mostly settled, and benchmarks for it are everywhere. I mean understanding in the sense a senior engineer has it: knowing what depends on what, what breaks if you change this, where the bodies are buried. The thing you'd want before letting an agent touch a load-bearing model.
It's a slippery thing to measure, and I want to lay out why, show you the test I built, point at where I think it's wrong, and then hand it to you.
Why "understands" is hard to pin down
