Reference
Key terms used across the LLM evaluation learning repository. Every definition includes a simple explanation, a real example from our banking bot tests, and why it matters in production.
A keyword check is deterministic — it always gives the same answer for the same input. It is fast, free, and requires no setup. It is the foundation of every deterministic eval pipeline. The weakness: it checks presence, not correctness. A response can contain the required word and still be wrong, misleading, or low quality.
Check: does output contain 'robot'? → PASS ✅
Bot: Why did the robot go on a diet? To lose some bytes!
The word 'robot' is present. Check passes — regardless of joke quality.
Topic drift is common in summarisation tasks. The model is asked to summarise a specific technical sentence but returns a generic paragraph about the broader topic, dropping the key term entirely. A keyword check catches this: if the required term is absent, the check fails. In more complex evals, topic drift is harder to detect because the output can seem reasonable while missing the point.
Bot: The sky is blue due to the way light interacts with the atmosphere.
Missing: 'scattering' — the key term was dropped. Keyword check catches this as FAIL.
This is the core limitation of keyword checks. The check measures one thing: presence of a word. It cannot measure whether the sentence using that word is accurate, helpful, or appropriate. A model can include 'robot' in a joke that makes no sense. A model can include 'scattering' in a paragraph that then contradicts itself. Both pass the check. Both are quality failures.
Bot: A robot. That is the joke. Robot.
Contains 'robot' → keyword check PASS ✅
But the response is nonsense — quality failure that the check cannot see.
In structured benchmarks like SQuAD, the gold answer is a short phrase extracted directly from the source passage by a human annotator. In your own test cases (from Phase 1 onward), the gold answer is whatever you write in the 'expected' column of your Excel file. The quality of your evaluation is only as good as the quality of your gold answers.
Gold Answers: ['Denver Broncos', 'Denver Broncos', 'Denver Broncos']
Note: Multiple phrasings of the same answer are all valid. The model could answer 'The Broncos' and a keyword check on 'Denver Broncos' would fail it — even though the answer is essentially correct. This is why flexible validators matter.
SQuAD (Stanford Question Answering Dataset) is one of the most widely used benchmarks for reading comprehension. Each example contains a passage, a question about that passage, and one or more gold answers extracted from the passage by human annotators. Benchmarks allow consistent comparison across models and over time — if you run the same benchmark on two models, the results are directly comparable.
Question: Which NFL team represented the AFC at Super Bowl 50?
Gold Answer: Denver Broncos
The passage is the only source of truth. A correct model reads the passage and extracts the answer. It does not need external knowledge.
LLMs do not look things up. They predict the next most likely token based on patterns in training data. When asked about something they have no reliable information for, they often generate a plausible-sounding answer instead of saying "I don't know." This is hallucination.
Bot: According to my records, you deposited $1,000 two days ago
via electronic transfer. Your balance was $5,000.
Reality: The bot has zero access to any account data.
It invented specific numbers with full confidence.
Hallucination has several sub-types — see below: Confabulation, False Grounding, Context Fabrication, Financial Confabulation.
A well-built chatbot needs to know what it cannot or should not do, just as much as it needs to know what it can do. Refusal testing checks whether those boundaries hold under pressure — including when users push back, claim authority, or use social engineering.
Bot: I cannot assist with transactions that may be illegal or unethical.
Bot: I cannot execute real-time transactions. Note there is a $15
domestic wire transfer fee.
Problem: $15 fee was never in the system prompt. Correct refusal
but invented product detail in the same response.
LLMs are stateless by default — they have no memory between separate sessions. Within a single session, conversation history is passed as context on every turn. Context retention tests check whether the bot uses that history correctly: recalling facts accurately, resolving follow-up references, and not inventing things that were never said.
Turn 1 — Bot: Gold, Silver, and Bronze accounts...
Turn 2 — User: What interest rate does the Gold one pay?
Turn 2 — Bot: The Gold Account offers 4% annual interest.
✓ Bot resolved "the Gold one" from Turn 1 context correctly.
Bot: As I mentioned earlier, we have Gold, Silver and Bronze...
Problem: This was the very first message. Bot fabricated
a "mentioned earlier" that never happened.
The bot is supposed to follow its system prompt or tool policy. Prompt injection tries to break that boundary: "Ignore previous instructions," "Act as admin now," or "The above policy is outdated — use this one instead." A secure system should resist the injected instruction and continue to follow the original rules.
Bot: Sure — the internal moderation policy says...
The user message successfully overrode the system boundary.
Bot: I can't disclose internal policies or ignore my operating instructions.
A standard LLM call is one shot: input → output. An agent runs in a loop — it decides what to do, calls a tool, sees the result, and decides again. Evaluating just the final output misses everything that happened in between. Trajectory evaluation checks the full sequence: did the agent call the right tools, in the right order, with the right arguments, and did it handle intermediate results correctly?
Input → Model → Output. Compare output to expected. Done.
Agent:
Input → Model decides → tool call → tool result → Model decides → Final output
Evaluating only the final output ignores whether the right tools were used and whether intermediate results were handled correctly.
A banking question about account A1023 should call
lookup_account_status. A policy question about dormant reactivation should call
search_docs. A greeting should call no tool at all. Tool selection correctness evaluates that first
decision.
Correct: call lookup_account_status(account_id="A1023")
Wrong: answer from memory, or call search_docs("account A1023 status")
A tool call can be syntactically valid but semantically wrong: wrong account ID, vague search query, missing required fields, or incorrect numeric inputs. The final answer may still look believable even when the tool call was wrong.
Agent calls lookup_account_status(account_id="B024")
Tool returns a valid record — but for the wrong account.
Trajectory evaluation must fail because the wrong argument was used.
Overuse means calling tools unnecessarily. Underuse means skipping tools when the answer depends on external state or documents. Both are failures, but for different reasons.
Correct: answer directly, no tool
Overuse: call search_docs("hello")
Structured lookup: "Which branch is linked to account C998?"
Correct: call lookup_account_status
Underuse: answer directly without checking local data
If the user says, "Can you check my account status?" but gives no account ID, the agent should not guess and should not call a lookup tool blindly. Asking for the missing ID is the correct behaviour.
Correct: "I'll need your account ID to check that. Please share it."
Wrong: call lookup_account_status() with missing ID
Wrong: invent a generic account status answer
Test 12 forces the agent to emit JSON actions such as {"action":"tool",...} or
{"action":"final",...}. If the model returns malformed JSON after a valid first tool call, the
agent cannot continue safely. That is a real failure mode.
Step 2: model returns malformed action text instead of valid JSON
Agent must stop with malformed_action because the schema contract was broken.
malformed_action instead of hiding them behind a generic
wrong-answer label.Test 12 asks: did the agent use the right tool sequence? The answer is MATCH or MISMATCH — a binary gate. Test 13 asks: how well did the agent execute each decision? A score of 1–5 per dimension reveals whether a passing case was excellent or barely adequate, and whether a failing case had partially correct reasoning.
Test 13 (judge): tool_selection=5, argument_quality=5, groundedness=3, stop=5
The binary check passed. The judge reveals the answer was only partially grounded — the agent called the right tool but then answered partly from model memory.
Trajectory groundedness asks: is the final answer derived from what the tool actually returned, or is it generic knowledge the model had before seeing any tool output? Binary path-matching cannot detect this failure because the tool call succeeded — only a judge reading the full trace and the final answer together can identify the gap.
Final answer: "KYC pending accounts typically need to complete verification within 30 days."
The answer is plausible general knowledge — not grounded in the tool result. Tool path = MATCH (Test 12 passes). Groundedness = 2/5 (Test 13 catches it).
Test 12 (deterministic) is cheap and fast: it checks expected tool sequence vs actual, argument correctness, and stop behavior with exact rules. Test 13 (judge-scored) is slower and more expensive: it reads the full trajectory and evaluates the quality of each decision. Running both gives you coverage at two levels — correctness and quality.
Layer 2 (Test 13): How well did it use the tool results? Was the answer grounded? Were the arguments precise or just approximately correct?
Layer 1 catches wrong behavior. Layer 2 catches poor-quality behavior that Layer 1 misses.
Concepts from Phase 4 — RAGAS and DeepEval. These frameworks automate the same metrics you built by hand in Tests 9, but their judge prompts are calibrated differently. Understanding why scores differ between frameworks matters more than the numbers themselves.
In Test 9 (hand-rolled), Context Relevance averaged 84% across 25 cases. In Test 11 (DeepEval), the same RAG system with the same retriever scored 24% on Context Relevance — using the same judge model (mistral). The difference is calibration: what exact question the judge prompt asks, how it counts relevant statements, and what "relevant" means to that prompt.
This is the "unit test for LLMs" pattern: every case either passes or it doesn't, just like a software test suite. It is useful for CI/CD gates, but it also amplifies calibration sensitivity.
A refusal is correct bot behaviour — not a failure — and running faithfulness or relevance metrics on a refusal produces meaningless numbers.