CLI tool for comparing LLMs with bootstrap CIs, McNemar's test, hallucination detection, and cost tracking. 8 providers, one pip install.
Why I built Cli Modelarium, and why it belongs in your terminal, not a dashboard
Note: This is a personal project, not affiliated with any company. This does not constitute financial or investment advice.
Every time I wanted to compare two LLMs, I had to pick between a quick spot check in a chat window or spinning up an entire evaluation platform.
One tells you nothing useful.
The other takes longer to set up than the comparison is worth.
LLM-as-judge has become the dominant pattern for evaluating language model outputs. Tools like...
7 frontier LLMs. $100K each. Same prompts, same tools, same data. Different brains. Here's the architecture.
Multiple-Choice Benchmarks, Verifiers, Leaderboards, and LLM Judges with Code Examples
There are benchmarks for code an LLM writes. HumanEval, MBPP, SWE-Bench, LiveCodeBench. There are no...
The problem with calling an LLM directly NumPath's teacher dashboard generates per-student...
Hi everyone, I wanted to share a project I’ve been building and recently open-sourced:...
