After three weeks of running Qwen2.5-32B on a DGX Spark, the number that surprised me most wasn't the...
Qwen2.5-32B: zero errors on 2,859 code-gen tests (JSON, function calls, tool chains) via vLLM vs. 0.1–0.3% cloud; 50-step autonomous agents drop failure risk 14%→6%. Local inference unlocks production autonomous deployment without cloud-API fallback.
After three weeks of running Qwen2.5-32B on a DGX Spark, the number that surprised me most wasn't the throughput or latency. It was zero.
Zero structural errors across 2,859 code generation tests.
What I Tested
EvalScope with code generation tasks covering:
Structured JSON output
There are benchmarks for code an LLM writes. HumanEval, MBPP, SWE-Bench, LiveCodeBench. There are no...
Originally published at tengli.dev When I added an LLM-powered eval to mcpgrade, the first real run...
How I integrated Specmatic executable contracts into an NL-to-SQL engine as a guardrail against AI-agent drift -- across 12…
I pointed a 22-agent review at my own competition endgame expecting notes about the code. It came back with three errors in my…
AI code generation has scaled. Verification hasn’t — and that gap is becoming technical debt. The...
Submission for DEV's Summer Bug Smash — Clear the Lineup track. The...
