Evaluating LLMs on standardized leaderboards (like MMLU or HumanEval) is helpful, but it rarely...
Evaluating LLMs on standardized leaderboards (like MMLU or HumanEval) is helpful,
but it rarely tells you how a model performs on real-world edge cases.
In this benchmark, I tested three models on a specific dev-sec scenario:
Detecting hidden reentrancy and integer overflow vulnerabilities in a complex smart contract.
Models Tested:
TL;DR Last week I benchmarked 5 open-weight models (Llama 4 Scout, Llama 3.3 70B, Qwen3...
10 adversarial scenarios, 64 assertions, 3-tier evaluation pyramid. Llama, Qwen, GPT-OSS — none scored above 63%. Here's what…
I built GroundCheck, a small open model that checks whether an AI answer is actually supported by the...
This is a submission for the Gemma 4 Challenge: Write About Gemma 4 The Problem: Developers...
We've developed two new, challenging academic benchmarks measuring AI models’ ability to develop exploits, and an updated version…
Researchers at Carnegie Mellon University built a new benchmark that measures how far AI agents can go when exploiting real…
