I had a model refactor part of a backtesting engine last month. Gave it function signatures, call chain, test suite, the works. Output looked solid, tests passed. Then I renamed a variable and watched one assertion go sideways. Turned out the model had wired that variable to the wrong thing because I never explained what it actually meant. I added one sentence of context. Regenerated. Fixed.
Same week, different mood, I asked a model whether it understands sadness. Got back something careful and well-formed. It can recognize patterns of sadness in text, it said, and it can offer supportive responses, but it is not sure whether that constitutes understanding. Reads well. I genuinely could not tell whether it understood sadness or whether it had reproduced, very fluently, the statistical shape of how humans write about sadness. Its training set has seen more breakup posts than any therapist alive.
Then I looked at a multi-agent trading framework on GitHub. Tens of thousands of stars. Five LLM agents debating each other into a trade signal. The backtests looked spectacular.
Three interactions, three completely different feelings. The first one, the model genuinely helped. The second one, I am not sure what happened. The third one made me nervous.
