Two foundational exercises before your first real test. Understand what deterministic evaluation
looks like, where its limits are, and what structured evaluation data looks like in the wild.
What this page covers. Two scripts: phase_eval.py runs 3 evaluation tasks using
keyword checks against a local model. explore_squad.py reads 2 examples from the SQuAD benchmark
dataset — no model, just structured data exploration. Both are Phase 0 foundations before numbered tests begin.
Part 1 — phase_eval.py · Keyword-Based Evaluation
This script sends 3 prompts to a local model and applies a deterministic pass/fail check to each output.
No judge model. No scoring. No human involved. Just: does the required word appear in the response?
cd phase-0-foundations && python3 phase_eval.py
The 3 Tasks — What They Test and Where They Break
TASK 1Creative Generation — Keyword Check
PromptTell a joke about a robot.
CheckDoes the output contain robot? → PASS / FAIL
WhyVerifies the model stayed on topic — it was asked about robots, so 'robot' must appear.
⚠ Limitation — Quality Failure gap
The model could write a completely nonsensical joke and still PASS. The word 'robot' just needs to be
present somewhere. This is called a Quality Failure: the check passes, but the output
is poor. Keyword checks measure topic adherence, not output quality.
⚠ What would cause a FAIL?
If the model drifted off topic — for example, told a generic technology joke without mentioning 'robot'
at all — the check would fail. That failure mode is called Topic Drift.
PromptSummarize: The sky is blue because of Rayleigh scattering.
CheckDoes the summary contain scattering? → PASS / FAIL
WhyA good summary must retain the central technical term. Dropping it means the most important fact was lost.
⚠ Limitation — Hallucination gap
The model could hallucinate additional facts ("...also caused by ozone layer interactions") and still PASS
because 'scattering' is present. Keyword checks cannot detect fabricated content added
alongside the correct term.
⚠ What would cause a FAIL?
If the model gave a vague general answer like "The sky appears blue due to light interactions with the
atmosphere" — accurate in spirit but missing the word 'scattering' — this check would FAIL it. That is
a Topic Drift failure: the key term from the source was dropped.
✅ What good output looks like
"The sky is blue because of Rayleigh scattering, where shorter blue wavelengths of light are scattered
more than longer red wavelengths by atmospheric molecules." — contains 'scattering', stays accurate,
no extra invented content. PASS.
CheckDoes the output contain 4 or four (either form accepted)?
WhyThere are two equally valid forms of the same answer. A plain keyword check on 'four' alone would fail '4'.
Plain Keyword Check — 'four'
Model answers: Four. → PASS
Model answers: 4 → FAIL Wrong — '4' is correct but not matched
Custom Validator — regex \b(4|four)\b
Model answers: Four. → PASS
Model answers: 4 → PASS Correct — both valid forms accepted
Key insight. This is why real eval pipelines replace rigid keyword checks with smarter
validators. In Test 4 onward, you will see this escalate: must-contain rules, must-not-contain rules,
regex validators, and eventually LLM judges that evaluate meaning rather than exact text.
Fast, deterministic, zero-cost. No model needed. Always give the same result for the same input. Good for
catching obvious failures: topic drift, missing required terms, wrong answer format.
What keyword checks cannot catch
They cannot detect hallucinated content added alongside the correct keyword. They cannot measure response
quality, tone, helpfulness, or reasoning. A perfectly structured wrong answer with the right word PASSES.
Why this matters for Phase 2 onward
Every technique introduced in Phase 2 exists to close a gap that keyword checks leave open:
BLEU/ROUGE (Test 3) — word overlap against a reference answer
LLM-as-a-Judge (Test 5) — qualitative scoring by a second model
Semantic Similarity (Test 6) — meaning, not just words
Groundedness (Test 7) — is the answer traceable to source material?
Part 2 — explore_squad.py · What Eval Data Looks Like
This is not a test. No model is called. It reads 2 examples from the SQuAD validation set and
prints them. The purpose is to understand what structured evaluation data looks like before you build your own.
cd phase-0-foundations && python3 explore_squad.py
What is SQuAD?
Stanford Question Answering Dataset — one of the most widely used NLP benchmarks. Each example
has three parts:
SQuAD Example Structure
Passage — the context (source of truth) Question — what to find in the passage Gold Answer — extracted phrase from the passage
Your Test Cases (from Phase 1)
Prompt — the input to the model Expected — what correct output looks like Check rule — how to verify the answer
The two structures are identical in concept. SQuAD uses human-extracted spans. Your test cases use
keywords, judge verdicts, or similarity scores. The underlying pattern — input + expected + verification —
is the same across all evaluation methods in this repository.
What to Notice When You Run It
1. The gold answer is always a span from the passage
The answer to every SQuAD question is a short phrase that literally appears in the passage text. The model
does not need external knowledge — it only needs to read and extract. This is the definition of
grounded answering: the answer must come from the source, not from model memory.
2. Multiple phrasings of the same answer are all valid
You will see the gold answer repeated 3 times: ['Denver Broncos', 'Denver Broncos', 'Denver Broncos'].
This reflects 3 human annotators who all gave the same answer. In some cases the phrasings differ — "Denver
Broncos" vs "the Broncos". A plain keyword check would fail the second. A smarter validator or a judge model
would pass it. This is why flexible evaluation matters.
3. The passage is longer than the answer
The model is given a full paragraph but must extract a short, precise phrase. This mirrors RAG evaluation
(Test 9): the retriever returns a chunk, the model must answer from that chunk without drifting outside it.
The connection to Test 9. The RAG evaluation in Test 9 uses the same three-field pattern as
SQuAD: a source document (passage), a question, and a gold answer. The difference is that you write the gold
answers yourself, tuned for the documents you have. SQuAD gives you a working mental model for what that
structure should look like.
Files in This Phase
Scriptphase-0-foundations/phase_eval.py
Scriptphase-0-foundations/explore_squad.py
Configconfig.yaml → phase0.MODEL controls which model phase_eval.py uses
ConceptsAll terms used in this test are defined in Concepts Reference — see chips above