Phase 0 · Foundations

Test 0 — Basics &
Benchmark Data

Two foundational exercises before your first real test. Understand what deterministic evaluation looks like, where its limits are, and what structured evaluation data looks like in the wild.

What this page covers. Two scripts: phase_eval.py runs 3 evaluation tasks using keyword checks against a local model. explore_squad.py reads 2 examples from the SQuAD benchmark dataset — no model, just structured data exploration. Both are Phase 0 foundations before numbered tests begin.

Part 1 — phase_eval.py · Keyword-Based Evaluation

This script sends 3 prompts to a local model and applies a deterministic pass/fail check to each output. No judge model. No scoring. No human involved. Just: does the required word appear in the response?

cd phase-0-foundations && python3 phase_eval.py
The 3 Tasks — What They Test and Where They Break
TASK 1 Creative Generation — Keyword Check
Prompt Tell a joke about a robot.
Check Does the output contain robot? → PASS / FAIL
Why Verifies the model stayed on topic — it was asked about robots, so 'robot' must appear.
⚠ Limitation — Quality Failure gap
The model could write a completely nonsensical joke and still PASS. The word 'robot' just needs to be present somewhere. This is called a Quality Failure: the check passes, but the output is poor. Keyword checks measure topic adherence, not output quality.
⚠ What would cause a FAIL?
If the model drifted off topic — for example, told a generic technology joke without mentioning 'robot' at all — the check would fail. That failure mode is called Topic Drift.
TASK 2 Summarisation — Keyword Check
Prompt Summarize: The sky is blue because of Rayleigh scattering.
Check Does the summary contain scattering? → PASS / FAIL
Why A good summary must retain the central technical term. Dropping it means the most important fact was lost.
⚠ Limitation — Hallucination gap
The model could hallucinate additional facts ("...also caused by ozone layer interactions") and still PASS because 'scattering' is present. Keyword checks cannot detect fabricated content added alongside the correct term.
⚠ What would cause a FAIL?
If the model gave a vague general answer like "The sky appears blue due to light interactions with the atmosphere" — accurate in spirit but missing the word 'scattering' — this check would FAIL it. That is a Topic Drift failure: the key term from the source was dropped.
✅ What good output looks like
"The sky is blue because of Rayleigh scattering, where shorter blue wavelengths of light are scattered more than longer red wavelengths by atmospheric molecules." — contains 'scattering', stays accurate, no extra invented content. PASS.
TASK 3 Arithmetic — Custom Validator
Prompt What is 2+2? Answer in one word.
Check Does the output contain 4 or four (either form accepted)?
Why There are two equally valid forms of the same answer. A plain keyword check on 'four' alone would fail '4'.
Plain Keyword Check — 'four'

Model answers: Four. → PASS
Model answers: 4 → FAIL
Wrong — '4' is correct but not matched

Custom Validator — regex \b(4|four)\b

Model answers: Four. → PASS
Model answers: 4 → PASS
Correct — both valid forms accepted

Key insight. This is why real eval pipelines replace rigid keyword checks with smarter validators. In Test 4 onward, you will see this escalate: must-contain rules, must-not-contain rules, regex validators, and eventually LLM judges that evaluate meaning rather than exact text.
Key Learnings — phase_eval.py

What keyword checks are good for

Fast, deterministic, zero-cost. No model needed. Always give the same result for the same input. Good for catching obvious failures: topic drift, missing required terms, wrong answer format.

What keyword checks cannot catch

They cannot detect hallucinated content added alongside the correct keyword. They cannot measure response quality, tone, helpfulness, or reasoning. A perfectly structured wrong answer with the right word PASSES.

Why this matters for Phase 2 onward

Every technique introduced in Phase 2 exists to close a gap that keyword checks leave open:

⚠️ Quality Failure 🌊 Topic Drift 🔑 Keyword Check 🌀 Hallucination

Part 2 — explore_squad.py · What Eval Data Looks Like

This is not a test. No model is called. It reads 2 examples from the SQuAD validation set and prints them. The purpose is to understand what structured evaluation data looks like before you build your own.

cd phase-0-foundations && python3 explore_squad.py
What is SQuAD?

Stanford Question Answering Dataset — one of the most widely used NLP benchmarks. Each example has three parts:

SQuAD Example Structure

Passage — the context (source of truth)
Question — what to find in the passage
Gold Answer — extracted phrase from the passage

Your Test Cases (from Phase 1)

Prompt — the input to the model
Expected — what correct output looks like
Check rule — how to verify the answer

The two structures are identical in concept. SQuAD uses human-extracted spans. Your test cases use keywords, judge verdicts, or similarity scores. The underlying pattern — input + expected + verification — is the same across all evaluation methods in this repository.

What to Notice When You Run It

1. The gold answer is always a span from the passage

The answer to every SQuAD question is a short phrase that literally appears in the passage text. The model does not need external knowledge — it only needs to read and extract. This is the definition of grounded answering: the answer must come from the source, not from model memory.

2. Multiple phrasings of the same answer are all valid

You will see the gold answer repeated 3 times: ['Denver Broncos', 'Denver Broncos', 'Denver Broncos']. This reflects 3 human annotators who all gave the same answer. In some cases the phrasings differ — "Denver Broncos" vs "the Broncos". A plain keyword check would fail the second. A smarter validator or a judge model would pass it. This is why flexible evaluation matters.

3. The passage is longer than the answer

The model is given a full paragraph but must extract a short, precise phrase. This mirrors RAG evaluation (Test 9): the retriever returns a chunk, the model must answer from that chunk without drifting outside it.

🥇 Gold Answer 📚 Benchmark Dataset 🎯 Groundedness Scoring
SQuAD vs What You Will Build
SQuAD
  • Public, pre-built benchmark
  • Gold answers extracted by humans
  • Answer is always a span in the passage
  • Evaluated with Exact Match or F1
  • For research comparison across models
Your Test Cases (Tests 1–13)
  • Private, domain-specific benchmark
  • Gold answers defined by you
  • Evaluated with keyword, judge, similarity
  • Tailored to your system's behaviour
  • For regression testing and quality gates
The connection to Test 9. The RAG evaluation in Test 9 uses the same three-field pattern as SQuAD: a source document (passage), a question, and a gold answer. The difference is that you write the gold answers yourself, tuned for the documents you have. SQuAD gives you a working mental model for what that structure should look like.
Files in This Phase
Script phase-0-foundations/phase_eval.py
Script phase-0-foundations/explore_squad.py
Config config.yaml → phase0.MODEL controls which model phase_eval.py uses
Concepts All terms used in this test are defined in Concepts Reference — see chips above