Kimi K3 was released this week, and like every model release it's being judged on leaderboard scores and screenshots. But a score is a bit like a football result, in that it tells you who won, not how the game was played. You wouldn't sign a player off a scoreline alone — you'd want to watch the tape. With code models, the tape is the code itself: how it's structured, whether you'd actually want to maintain it. That's what leaderboards can't show you.
So we built a benchmark that compares Kimi K3, Claude Fable 5 and Claude Opus 4.8 on exactly that — and it's open source, so you can run it yourself at github.com/dsplce-co/kimi-vs-fable-vs-opus.
This note summarises our findings after running it.
The task
Given a double-entry ledger module with an existing test suite, make it production-ready and add two features: transaction reversal and statement generation. There's no hint that there are any bugs to find.
