"Which model should we use?" is the most common question in every Bedrock project — and most teams answer it by eyeballing a few responses. That doesn't scale, it isn't reproducible, and it silently expires every time a new model version ships.

In this hands-on, we'll answer the question with data: Amazon Bedrock Model Evaluation, run in two modes — automatic metrics scored against reference answers, and LLM-as-a-Judge, where a stronger model grades each response. We'll build the evaluation dataset, run the jobs, and crunch the result files down to comparable numbers with jq and awk.

Prefer video? This entire hands-on is also on YouTube:

The two evaluation modes

Automatic evaluation runs your dataset through the target model and scores each response against your referenceResponse with built-in metrics — accuracy-style similarity scores, robustness, toxicity. Fast, cheap, objective, but only as good as your reference answers.