I keep a folder of ten small Solidity contracts with bugs I planted myself: a classic reentrancy in a withdraw function, a missing access modifier on an initializer, an ERC4626 vault that rounds in the depositor's favor, a signature check that never validates the signer, a few subtler logic bugs. Whenever I'm deciding which local model earns a slot on my disk, I run every candidate against all ten with the same prompts and compare notes. It's not a benchmark, there's no percentage at the end, but after doing this for months the patterns are consistent enough to be worth writing down.
This round: qwen2.5-coder against deepseek-coder, both running through Ollama on WSL2, both at the sizes that actually fit comfortably on my hardware, meaning 7B and below. If you can run 33B models locally, your conclusions will differ and you should mostly stop reading benchmarks of small models anyway.
Methodology, such as it is
Same ten contracts, none over 150 lines. Same three prompts per contract: an open "review this contract for security issues", a targeted "check specifically for reentrancy and access control problems", and a structured one demanding findings as a severity-tagged list. Each combination run a few times because small models are noisy, one lucky sample proves nothing. Temperature low but not zero. I score by hand: did it find the planted bug, did it describe it correctly, and how much noise came along for the ride.
