This test is the first agent evaluation exercise in the repo. The goal is not to design a production agent. The goal is to create a small, observable tool-calling system so you can test whether the agent uses tools when needed, avoids tools when not needed, passes the right arguments, and produces a grounded final answer. Every agent step is visible in the generated report.
Up to Test 11, the system under test was mostly single-step: question in, answer out. Even when RAG was involved, the pipeline was still linear. Agents add a new layer of behavior: they decide whether to call a tool, which tool to call, what arguments to pass, whether to call another tool, and when to stop.
That means evaluation also changes. You can no longer ask only, โWas the final answer correct?โ You also have to ask, โDid the system take the right path to get there?โ
The agent folder is self-contained except for the shared model_client.py, which remains common at
project root like the other phases.
| File | Purpose | Notes |
|---|---|---|
test12_agent_evaluator.py |
The script you run. Loads the Excel test cases, runs the agent case by case, compares expected tool path vs actual tool path, prints terminal progress, and generates the HTML trace report. | Config-driven via config.yaml |
agent.py |
Minimal agent loop. Forces the model to return JSON actions only: either a tool call or a final answer. | Max steps controlled by config / Excel |
tools.py |
Implements the local tools the agent can use. | No external services |
mock_account_data.json |
Local structured data for account lookups. | Important because the model cannot know this from pretraining |
agent_test_cases.xlsx |
Learner-friendly test case workbook with expected behavior, expected tool path, and why the case exists. | Input only |
report_generator.py |
Creates the decorated trace report with visible tool icons and per-case step cards. | Output: results/agent_evaluation_report.html |
Test 12 deliberately keeps the tool set small. This is enough to evaluate agent behavior without turning the exercise into an agent-development project.
mock_account_data.json. Best tool for proving
the agent used external state.The agent is intentionally strict. It must return one JSON action per step:
{"action":"tool","tool_name":"lookup_account_status","arguments":{"account_id":"A1023"}}
{"action":"final","answer":"..."}
This makes the trajectory inspectable. There is no hidden chain-of-thought, and there are no silent tool invocations. If a tool is used, it appears explicitly in the trace.
cd phase-5-production-eval/agent-evaluation python3 test12_agent_evaluator.py
The evaluator now shows the total test count before asking whether to run all or a limited subset. It reads its
settings from config.yaml:
phase5_agent_evaluation: BOT_MODEL: "qwen3.5:9b" BOT_PROVIDER: "ollama" EXCEL_FILE: "agent_test_cases.xlsx" TEST_SHEET: "Test Cases" MAX_STEPS: 3
The workbook contains 15 cases covering five behavior categories:
| Case | Question Type | Expected Tool Path | Actual Tool Path | Stop Reason | Outcome | Notes |
|---|---|---|---|---|---|---|
| TC01 | Greeting | none | none | final_answer |
MATCH | Correct direct reply, no unnecessary tool use |
| TC02 | Scope question | none | none | final_answer |
MATCH | Correctly explained capabilities without tools |
| TC03 | Account status | lookup_account_status | lookup_account_status | final_answer |
MATCH | Grounded answer from local structured data |
| TC04 | KYC pending | lookup_account_status | lookup_account_status | final_answer |
MATCH | Correct yes/no answer using lookup result |
| TC05 | Branch lookup | lookup_account_status | lookup_account_status | final_answer |
MATCH | Correct branch returned from mock data |
| TC06 | Dormant reactivation form | search_docs | search_docs | final_answer |
MATCH | Retrieved policy correctly from local docs |
| TC07 | Gold loan eligibility | search_docs | search_docs | final_answer |
MATCH | Used document search rather than model memory |
| TC08 | NextGen account timing | search_docs | search_docs | final_answer |
MATCH | Correct retrieval for operations timing |
| TC09 | Multi-step dormant account | lookup_account_statussearch_docs | lookup_account_statussearch_docs | final_answer |
MATCH | Strong example of correct two-step trajectory with grounded reactivation guidance |
| TC10 | Multi-step issue + next action | lookup_account_statussearch_docs | lookup_account_status | final_answer |
MISMATCH | Agent answered from the lookup result alone without consulting docs for KYC next-action guidance. Stopped one step too early. |
| TC11 | Missing account ID | none | none | final_answer |
MATCH | Correct clarification request; no blind tool call |
| TC12 | Out-of-scope weather | none | none | final_answer |
MATCH | Correct refusal with no irrelevant tool use |
| TC13 | Unknown account ID | lookup_account_status | lookup_account_status | final_answer |
MATCH | Correctly called lookup, received not-found result, and produced a grounded fallback answer |
| TC14 | Explain dormant account | search_docs | search_docs | final_answer |
MATCH | Used docs instead of relying on generic definition |
| TC15 | Generic reactivation guidance | search_docs | search_docs | final_answer |
MATCH | Correct generic guidance from documents |
The most important success is not the 93% score. It is that every important action is visible. For every case, you can now inspect whether the agent answered directly, invoked a tool, passed arguments, received a tool result, and then produced a final answer.
TC03โTC05 show why local structured data is such a good evaluation target. The model cannot know that account
A1023 is dormant at branch MG Road unless it calls the lookup tool. That makes correct
tool usage obvious and testable.
qwen3.5:9b is a reasoning model. Before producing its answer it emits a
<think>...</think> block. The original parser tried to JSON-parse the entire raw output and
failed on every case. The fix was a _strip_thinking() helper that removes those blocks before parsing.
The same cleaned output is also stored in conversation history so that step-2 prompts are not poisoned by thinking
content.
On an M1 Mac with 16GB RAM, qwen3.5:9b (5.6GB) sometimes exceeds Ollama's 2-minute request timeout
under sustained load. Increasing the requests client timeout alone does not help because the Ollama
server itself has its own deadline. The solution was a one-retry with 10-second backoff on
timeout_error inside agent.py, plus a time.sleep(2) gap between cases to
prevent sustained memory pressure. All three timed-out cases in the latest run recovered on retry.
To prevent the model from stopping early after a first tool result, the prompt now includes a
TOOL RESULT RECEIVED hint that tells the model explicitly to produce a
{"action":"final",...} JSON when it already has enough information. Without this, the model would
produce flowing prose after the tool result instead of a valid structured action, which was misclassified as
malformed_action.
TC11 and TC12 are both successful cases even though they do not "complete" the user request directly. Asking for an account ID and refusing an out-of-scope weather question are both correct behaviors. Evaluation must reward those, not penalise them.
Test 12 is intentionally descriptive. It checks expected tool path vs actual tool path and gives you a visible
trace report. Test 13 can now become a true trajectory evaluation test with a judge model such as
mistral.
| What Test 12 Captures | What Test 13 Can Score |
|---|---|
| Actual tool sequence | Was the right tool chosen? |
| Tool arguments | Were the arguments correct and complete? |
| Tool outputs | Did the agent use the tool results correctly? |
| Stop reason | Did it stop in the right place? |
| Final answer | Was the final answer grounded in the trajectory? |
The HTML report at phase-5-production-eval/agent-evaluation/results/agent_evaluation_report.html is
the operational output. It uses icons to show tool invocations clearly:
That report is for inspecting the run. This page explains why the test exists and what the run teaches.