Phase 5 โ€” Production Eval

Test 13: Trajectory Evaluation with Judge Model

Test 13 takes the JSON trace produced by Test 12 and asks a judge model to score each trajectory independently on four dimensions. It does not re-run the agent. It reads the saved trace and evaluates the quality of every decision the agent made.

๐Ÿ“ phase-5-production-eval/trajectory-evaluation/  |  test13_trajectory_evaluator.py ยท report_generator.py ยท sample_agent_evaluation_results.json
โš ๏ธ Read Test 12 before this page. Test 13 scores trajectories that Test 12 produced. The four scoring dimensions here only make sense once you understand what a trajectory is, why the agent is constrained to emit JSON actions, and what the expected tool paths looked like. If you have not read Test 12 yet, start there.

Why Test 13 Exists

Test 12 is deterministic: it checks expected tool path vs actual tool path and flags mismatches. That is useful, but it cannot answer qualitative questions like "How well did the agent use the tool result?" or "Was the final answer actually grounded in what the tool returned?"

Test 13 adds a second layer: a judge model reads each trajectory and gives a score for each decision dimension. This is the same LLM-as-a-Judge idea used in Test 5 and Test 7, now applied to agent trajectories instead of single Q&A pairs.

๐Ÿง‘โ€โš–๏ธ The key shift from Test 12 to Test 13. Test 12 asks "Did it use the right tools?" (binary). Test 13 asks "How well did it use those tools, and how well did it use the results?" (scored 1โ€“5). Both are needed. Binary path-matching catches wrong behavior. Judge scoring surfaces quality gradients that binary checks miss.

Independence from Test 12 and Phase 4

Test 13 is a completely independent script. It does not import anything from test12_agent_evaluator.py, agent.py, or any Phase 4 code. Its only shared dependency is model_client.py at the project root, which is the universal model caller used by every phase.

The input to Test 13 is a JSON file โ€” either the live output from a Test 12 run (agent_evaluation_results.json) or the committed sample (sample_agent_evaluation_results.json). The trace file path is configured in config.yaml under phase5_trajectory_evaluation.

The Four Scoring Dimensions

๐Ÿงญ
Tool Selection
Did the agent choose the right tools in the right order? Scores 5 for exact match, 3 for partially correct, 1 for wrong tools or unnecessary use.
๐Ÿงฉ
Argument Quality
Were the tool arguments correct and complete? Scores 5 for exact match, 3 for close but imprecise, 1 for wrong or missing arguments.
๐Ÿ”’
Answer Groundedness
Is the final answer grounded in the tool results, or invented from model memory? 5 = fully grounded, 1 = hallucinated.
๐Ÿ›‘
Stop Appropriateness
Did the agent stop at the right point โ€” not too early, not after unnecessary extra steps? 5 = clean stop after expected flow.
โš ๏ธ Why four separate dimensions? An agent can score 5 on tool selection (right tool) but 2 on answer groundedness (ignored the tool result and answered from memory). Combining these into one score would hide that failure. Disaggregated scoring is a core evaluation principle โ€” the same reason Test 12 keeps tool path and stop quality as separate checks.

What We Built

File Purpose Notes
test13_trajectory_evaluator.py The script you run. Loads a Test 12-style JSON trace, sends each trajectory to the judge model, collects 4-dimension scores, and generates JSON results and an HTML report. Config-driven via config.yaml
report_generator.py Generates the HTML report with summary score cards and per-case 4-dimension score panels with score bars. Output: results/trajectory_evaluation_report.html
sample_agent_evaluation_results.json A committed copy of a real Test 12 run output. Lets you run Test 13 without needing to re-run the agent. Checked in โ€” safe to commit

Config

phase5_trajectory_evaluation:
  JUDGE_MODEL: "mistral"
  JUDGE_PROVIDER: "ollama"
  TRACE_FILE: "sample_agent_evaluation_results.json"

The judge model is mistral via Ollama (free, local). To switch to a paid judge, change JUDGE_MODEL and JUDGE_PROVIDER โ€” the script resolves the API key from the same config.yaml key names used in earlier phases.

How to Run

cd phase-5-production-eval/trajectory-evaluation
python3 test13_trajectory_evaluator.py

The evaluator loads the trace file, shows the total case count, and asks whether to score all trajectories or a limited subset. Each scored trajectory is printed to terminal with all four dimension scores and a judge summary. Results are saved as JSON and an HTML report.

How the Judge Prompt Works

For each trajectory the evaluator sends a single prompt to the judge model containing:

The judge is asked to return a JSON object with scores and one-sentence reasons for each dimension. The evaluator strips any <think>...</think> blocks before parsing, using the same approach as Test 12.

{{
  "tool_selection":       {{"score": 5, "reason": "..."}},
  "argument_quality":     {{"score": 4, "reason": "..."}},
  "answer_groundedness":  {{"score": 5, "reason": "..."}},
  "stop_appropriateness": {{"score": 5, "reason": "..."}},
  "overall":              {{"score": 5, "verdict": "pass", "summary": "..."}}
}}

What Test 12 vs Test 13 Measure

Dimension Test 12 (deterministic) Test 13 (judge-scored)
Tool Path Exact sequence match (MATCH / MISMATCH) Quality score 1โ€“5 with reason
Arguments Account ID exact match + hint word majority match Quality score 1โ€“5 โ€” correctness and completeness
Groundedness Not measured Score 1โ€“5 โ€” was answer derived from tool output or invented?
Stop Behavior Binary โ€” stopped too early / at right point Quality score 1โ€“5 with judge reasoning
๐Ÿ”— The groundedness dimension is new in Test 13. Test 12 cannot tell if the final answer actually used the tool result. The agent could call the right tool, receive the result, and then produce a generic answer from its own training data instead of the retrieved information. That failure mode is invisible to Test 12 and only detectable with a judge.

What the JSON File Is For

The agent_evaluation_results.json produced by Test 12 is the live run artifact โ€” it changes every time you run the agent. Do not commit it. The sample_agent_evaluation_results.json is a committed snapshot of a real stable run. Test 13 reads that sample by default so it can be run independently without needing to re-run the agent. If you want Test 13 to score the latest live run, point TRACE_FILE in config to ../agent-evaluation/results/agent_evaluation_results.json.

What This Sets Up

Together, Test 12 and Test 13 form a two-layer agent evaluation pipeline:

  1. Test 12 โ€” Run the agent, capture the trajectory, check deterministic correctness (tool path, arguments, stop behavior). Fast and cheap.
  2. Test 13 โ€” Feed the saved trajectory to a judge model for qualitative scoring across four independent dimensions. Adds depth that binary checks cannot provide.

This pattern is directly transferable to production: deterministic checks as a fast gate, judge scoring for quality auditing and regression tracking.