How EvalPort's Grader System Works: 11 Types for LLM Evaluation

How EvalPort's Grader System Works

When designing EvalPort, the grader system was the hardest part to get right. Every eval framework has its own way of scoring LLM outputs — DeepEval uses metric classes, Promptfoo uses assertion objects, Inspect AI uses solver functions. We needed a system expressive enough to cover 90%+ of real-world eval needs, but simple enough that any framework could implement it.

The result: 11 grader types that carry their own semantics. A grader isn't just a name — it specifies its parameters, its model, its threshold. An eval suite is self-describing.

The 11 Grader Types

exact_match — Compare output to expected output, optionally ignoring case.

How EvalPort's Grader System Works: 11 Types for LLM Evaluation

Related reading

OpenEval: Why LLM Evaluation Needs a Standard Format

Who Grades the Grader? Your LLM Judge Is an Unvalidated Model in Production

LLM Evaluation Frameworks Compared: How to Actually Measure What Your Model…

Understanding the 4 Main Approaches to LLM Evaluation (From Scratch)

An open source LLM eval tool with two independent quality signals

LLM-as-Judge Shouldn't Aggregate Scores: Binary Checks as Evidence, One…