Evals stable

What it is

Evaluating an agent is harder than evaluating a completion because the output is a trajectory: a sequence of decisions, tool calls and intermediate results. A correct final answer reached by a wasteful or dangerous path is still a problem. Evals score what observability records; without tracing there is nothing to evaluate.

The layers

1. Ground truth on every step: tests, linters, type checks, schema validation. Free, deterministic, and they catch most regressions.
2. Trajectory metrics: did the agent pick the right tool, with the right arguments, in a sensible order, in a reasonable number of steps?
3. LLM-as-judge: a stronger model scores outputs against a written rubric. Use it for quality, tone and correctness that code cannot check.

Judges run in three modes: inline before the response goes out (rare, for high-stakes paths), asynchronously on every live trace, and offline in batch against a golden set.

Making judges trustworthy

• Judges have position bias. When comparing two outputs, score both orderings and treat disagreement as a tie.
• Calibrate against human ratings before trusting a judge. Aim for strong correlation, not perfect agreement.
• Build the golden set to a few hundred cases before reading aggregate numbers.
• Write rubrics as checklists, not adjectives.

When it matters

Before any change to the prompt, tools or model. Without evals, context engineering is guesswork.

Watch out for

• Judging with the same model family you are evaluating inflates scores.
• Traces contain user data. Your eval pipeline is a data pipeline.