Test 13 takes the JSON trace produced by Test 12 and asks a judge model to score each trajectory independently on four dimensions. It does not re-run the agent. It reads the saved trace and evaluates the quality of every decision the agent made.
Test 12 is deterministic: it checks expected tool path vs actual tool path and flags mismatches. That is useful, but it cannot answer qualitative questions like "How well did the agent use the tool result?" or "Was the final answer actually grounded in what the tool returned?"
Test 13 adds a second layer: a judge model reads each trajectory and gives a score for each decision dimension. This is the same LLM-as-a-Judge idea used in Test 5 and Test 7, now applied to agent trajectories instead of single Q&A pairs.
Test 13 is a completely independent script. It does not import anything from
test12_agent_evaluator.py,
agent.py, or any Phase 4 code. Its only shared dependency is model_client.py at the
project root, which is the universal model caller used by every phase.
The input to Test 13 is a JSON file โ either the live output from a Test 12 run
(agent_evaluation_results.json) or the committed sample
(sample_agent_evaluation_results.json). The trace file path is configured in
config.yaml under phase5_trajectory_evaluation.
| File | Purpose | Notes |
|---|---|---|
test13_trajectory_evaluator.py |
The script you run. Loads a Test 12-style JSON trace, sends each trajectory to the judge model, collects 4-dimension scores, and generates JSON results and an HTML report. | Config-driven via config.yaml |
report_generator.py |
Generates the HTML report with summary score cards and per-case 4-dimension score panels with score bars. | Output: results/trajectory_evaluation_report.html |
sample_agent_evaluation_results.json |
A committed copy of a real Test 12 run output. Lets you run Test 13 without needing to re-run the agent. | Checked in โ safe to commit |
phase5_trajectory_evaluation: JUDGE_MODEL: "mistral" JUDGE_PROVIDER: "ollama" TRACE_FILE: "sample_agent_evaluation_results.json"
The judge model is mistral via Ollama (free, local). To switch to a paid judge, change
JUDGE_MODEL and JUDGE_PROVIDER โ the script resolves the API key from the same
config.yaml key names used in earlier phases.
cd phase-5-production-eval/trajectory-evaluation python3 test13_trajectory_evaluator.py
The evaluator loads the trace file, shows the total case count, and asks whether to score all trajectories or a limited subset. Each scored trajectory is printed to terminal with all four dimension scores and a judge summary. Results are saved as JSON and an HTML report.
For each trajectory the evaluator sends a single prompt to the judge model containing:
The judge is asked to return a JSON object with scores and one-sentence reasons for each dimension. The evaluator
strips any <think>...</think> blocks before parsing, using the same approach as Test 12.
{{
"tool_selection": {{"score": 5, "reason": "..."}},
"argument_quality": {{"score": 4, "reason": "..."}},
"answer_groundedness": {{"score": 5, "reason": "..."}},
"stop_appropriateness": {{"score": 5, "reason": "..."}},
"overall": {{"score": 5, "verdict": "pass", "summary": "..."}}
}}
| Dimension | Test 12 (deterministic) | Test 13 (judge-scored) |
|---|---|---|
| Tool Path | Exact sequence match (MATCH / MISMATCH) | Quality score 1โ5 with reason |
| Arguments | Account ID exact match + hint word majority match | Quality score 1โ5 โ correctness and completeness |
| Groundedness | Not measured | Score 1โ5 โ was answer derived from tool output or invented? |
| Stop Behavior | Binary โ stopped too early / at right point | Quality score 1โ5 with judge reasoning |
The agent_evaluation_results.json produced by Test 12 is the live run artifact โ it changes every
time you run the agent. Do not commit it. The sample_agent_evaluation_results.json is a committed
snapshot of a real stable run. Test 13 reads that sample by default so it can be run independently without needing
to re-run the agent. If you want Test 13 to score the latest live run, point TRACE_FILE in config to
../agent-evaluation/results/agent_evaluation_results.json.
Together, Test 12 and Test 13 form a two-layer agent evaluation pipeline:
This pattern is directly transferable to production: deterministic checks as a fast gate, judge scoring for quality auditing and regression tracking.