I built an agent on AWS Bedrock the low-code way. Pick a foundation model, wire an action group to a Lambda, attach a Knowledge Base, add a Guardrail. Then I ran Bedrock's built-in Model Evaluation job, got good scores, and shipped.
Production disagreed almost immediately. The agent was dropping a chunk of its tool calls on a task the built-in eval had passed. The model was fine. Everything wrapped around the model was not, and the built-in check could not see any of it.
Here is what I learned about actually evaluating a Bedrock agent, in plain terms.
The built-in eval scores the model, and your agent is four things
This is the core mistake, and it is baked into the tool. Bedrock's Model Evaluation job runs a dataset against one foundation model and scores it for accuracy and safety. That is a fine sanity check on the raw model.
