"Is Q4 safe for tool-calling?" gets asked constantly in local-LLM circles, and the answers are almost always anecdotal — a few hundred agent-hours on one model, extrapolated to everything. I wanted a benchmark where every degradation claim comes from bootstrapping the paired per-seed delta itself, not from eyeballing whether two confidence intervals happen to overlap. So I built one: QuantCall.
No cloud GPUs involved — everything below ran on my own hardware, an RTX 3050 Laptop with 4096 MiB of VRAM, which is exactly why the model choices below (0.6B–1.7B) look modest. That's the point: these are the models people are actually running on this class of hardware.
Setup: BFCL v4 (T1 simple/multiple + T6 irrelevance, n=200/seed, 3 seeds, greedy decoding, temperature=0). Metrics: Schema-Validity Rate (SVR), Tool-Selection Accuracy (TSA), Argument Correctness (AC), Abstention Accuracy, and Function-Calling Reliability (FCR — their weighted aggregate).
Headline result: model family beats model size as a predictor
Model
