TL;DR: This walkthrough shows how developers and coding agents can use Quantiles, an open-source AI evaluation platform licensed under Apache 2.0, to quickly run, analyze, and compare AI evaluations locally. We'll use the SimpleQA Verified benchmark as an example throughout this post, letting you follow the commands, inspect the evaluation results, and configure your own model for the same workflow.
Running an AI evaluation is rarely as simple as sending prompts to a model. Developers must connect datasets, model APIs, scoring logic, result storage, and comparison tooling before they can answer a basic question: did the system get better? When those pieces are spread across scripts, notebooks, and logs, every rerun becomes harder to reproduce and diagnose. A score alone cannot reveal whether the model changed or whether the dataset, prompt, scorer, or sample set changed with it.
Quickstart: Run an example benchmark
The Quantiles CLI is called qt on the command line. A simple curl ... | bash command supports macOS and Linux on X86-64 and Arm64 systems. First, use it to install the CLI:
curl -fsSL https://cli.quantiles.io/install.sh | bash
