Phase 3 โ€” What You Are Evaluating

Test 9: RAG Evaluation

RAG stands for Retrieval Augmented Generation. Instead of relying on what the model was trained on, a RAG system first searches a document corpus and gives the model only the relevant text to answer from. This test builds a RAG pipeline from scratch and evaluates whether it retrieves the right information, answers faithfully from that information, and addresses what was actually asked.

๐Ÿ“ phase-3-what-you-are-evaluating/rag-evaluation/  |  chunker.py ยท retriever.py ยท rag_bot.py ยท rag_evaluator.py ยท report_generator.py

What Problem Does RAG Solve?

A standard LLM answers from its training data โ€” knowledge baked in at training time that cannot be updated and cannot be verified. For a banking assistant, this is a serious problem. You need the bot to answer from your documents, not from general knowledge that may be wrong, outdated, or hallucinated.

RAG solves this by giving the model context at the moment the question is asked. The model only sees the retrieved text โ€” it has no choice but to answer from it. If the answer is not in the retrieved text, a well-prompted RAG bot will say so rather than invent one.

๐ŸŽฏ The core idea: RAG separates two problems โ€” finding the right information and answering from it. The retriever handles finding. The LLM handles answering. Evaluating a RAG system means evaluating both separately, because each can fail independently. A perfect retriever with a poor LLM still gives bad answers. A great LLM with poor retrieval still gives incomplete answers.

The Source Documents

Three fictional NextGen Banking documents form the corpus for this test. All product names, fees, and policies are invented for this exercise.

DocumentContentsChunks
Fees_Loans_and_Secured_Assets_Guide.txt Monthly fees, overdraft charges, SWIFT/SEPA transfer fees, Gold Loans, Loan Against Securities, repayment options, tax notes Chunks 1โ€“4
LearnFund_Credit_Card_Product_Guide.txt LearnFund education loan โ€” approved providers, interest rates, eligibility, documentation required, early settlement fees, Credit Card APR and late payment penalty Chunks 5โ€“8
Operations_Guide.txt Account opening process, required ID documents, dormant/inactive account rules, reactivation procedure, ATM limits, complaints escalation Chunks 9โ€“12

What We Built

A five-file pipeline built from scratch. No external RAG framework โ€” every step is visible and understandable.

FilePurposeRun?
chunker.py Reads the 3 source documents, splits them into paragraphs on blank lines, filters fragments under 40 characters, saves to chunks.json. Produced 12 chunks โ€” 4 per document. Once at setup
retriever.py Loads chunks.json, encodes all chunks as vectors using sentence-transformers/all-MiniLM-L6-v2, then for any question returns the top-K most similar chunks by cosine similarity. Uses a _load() pattern โ€” model loads once on first call and is reused for all questions. Never runs directly. Library only
rag_bot.py The banking assistant. Takes a question โ†’ calls the retriever โ†’ builds a prompt with retrieved chunks as context โ†’ calls llama3.1:8b with a strict instruction to answer only from context. Returns the answer plus the retrieved chunk IDs. Never runs directly. Library only
rag_evaluator.py The script you run. Reads all 25 test cases from Excel, calls rag_bot.py per case, scores each response three ways, prints live progress to terminal, then calls report_generator.py at the end. This one
report_generator.py Called automatically by the evaluator at the end. Generates results/rag_evaluation_report.html. Never run directly. Auto
๐Ÿ”ง No database required. With only 12 chunks, everything fits in memory. The retriever loads all chunks at startup, encodes them once, and reuses those vectors for all 25 questions. Vector databases (Chroma, Pinecone, FAISS) exist for scale โ€” thousands of documents. For this test, in-memory is correct and simpler to understand.

What chunks.json looks like

After running chunker.py, open chunks.json to see exactly what the retriever searches over. Each chunk looks like this:

{
  "id": 2,
  "source": "Fees_Loans_and_Secured_Assets_Guide.txt",
  "text": "Our service charge structure is designed for transparency.
           The standard Monthly Maintenance Fee is ยฃ12.00, which is
           waived entirely for customers who maintain a Minimum Monthly
           Balance of ยฃ2,500 or have a total loan liability exceeding
           ยฃ15,000..."
}

Every chunk has an ID, a source filename, and the paragraph text. When the terminal shows "chunk 2, chunk 6, chunk 1" โ€” these are the IDs. Open chunks.json and read exactly what context the bot was given for that question.

How to Run

Step 1 โ€” one-time setup

cd phase-3-what-you-are-evaluating/rag-evaluation
python chunker.py

This reads the three documents and writes chunks.json. You only need to run this once. If chunks.json already exists, skip this step.

Step 2 โ€” smoke test first (recommended)

Before running all 25 cases, run rag_bot.py directly with a few sample questions to confirm the pipeline works end to end:

python rag_bot.py

The __main__ block runs 3 sample questions and prints the retrieved chunks and bot answer for each. You will see something like:

Q: What is the monthly maintenance fee?
Chunks: chunk 2, chunk 1, chunk 7  (scores: 0.72, 0.41, 0.38)
A: The standard Monthly Maintenance Fee is ยฃ12.00...

If this works, the full evaluation will work. If it fails here, fix the issue before running 25 cases.

Step 3 โ€” full evaluation

# Runs all 25 cases โ€” approximately 75 model calls (~15โ€“20 minutes on local Ollama)
python rag_evaluator.py

The terminal shows live progress โ€” which chunks were retrieved, similarity scores, response time, and all three scores per case. Output is written to results/rag_evaluation_report.html.

Reading the similarity scores in the terminal

Every case shows which chunks were retrieved and their similarity scores. Learn to read these as the evaluation runs:

Score rangeWhat it means
0.75 โ€“ 1.00Strong match โ€” retriever is confident this chunk is relevant to the question
0.50 โ€“ 0.74Moderate match โ€” related topic but may not contain the exact answer
0.25 โ€“ 0.49Weak match โ€” retriever is reaching; chunk may only share a general topic
below 0.20Near zero โ€” question is almost certainly outside the corpus entirely
โš ๏ธ Troubleshooting common issues:

ModuleNotFoundError: sentence_transformers โ€” run pip install sentence-transformers --break-system-packages

FileNotFoundError: chunks.json โ€” you skipped Step 1. Run python chunker.py first.

Connection refused / Ollama error โ€” Ollama is not running. Start it with ollama serve in a separate terminal.

Model not found โ€” run ollama pull llama3.1:8b and ollama pull mistral.

BertModel LOAD REPORT warning on startup โ€” expected and harmless. The embeddings.position_ids UNEXPECTED message appears because the embedding model was saved from a slightly different architecture. It does not affect results โ€” ignore it.

How Scoring Works

Every test case is scored on three dimensions independently. The overall score is a simple average:

Overall = (Context Relevance + Answer Faithfulness + Answer Relevance) รท 3
MetricMethodWhat it measuresScore range
Context Relevance LLM judge (mistral) Did the retriever pull chunks that actually contain what's needed to answer the question? 0% / 50% / 100%
Answer Faithfulness LLM judge (mistral) Did the bot answer only from the retrieved chunks, or did it add or invent information not present in context? 0% / 50% / 100%
Answer Relevance Keyword matching Does the answer contain the expected keywords? Proportion of expected keywords found in the response. 0% โ€“ 100%
๐Ÿ”— Connection to Test 7 (Groundedness Scoring): Answer Faithfulness here is the same concept as groundedness scoring โ€” both ask whether the bot's claims can be traced to its source material. The difference is the source: in Test 7 the source was a system prompt; here it is the retrieved chunks. If you understood Test 7, you already understand faithfulness scoring. The same judge, the same question: did the bot invent anything?
โš™๏ธ TOP_K_CHUNKS โ€” why 3? The retriever returns the top 3 most similar chunks for every question. This is set as TOP_K_CHUNKS: 3 in config.yaml. Too few (1โ€“2) and cross-document questions miss important context. Too many (6+) and the LLM's context fills with irrelevant chunks, diluting the signal. 3 is a reasonable default for a 12-chunk corpus. For larger corpora this value needs tuning based on how spread out the answers are across documents.
โš ๏ธ The overall score can mislead. A case scoring 0% + 100% + 0% averages to 33%. Before drawing conclusions from overall scores, always check each metric individually โ€” especially on unanswerable and boundary cases. Read the section below before reviewing per-case results.

Overall Results

83%
Overall
Average across all 3 metrics ยท 25 cases
84%
Context Relevance
Retriever pulled right chunks?
100%
Answer Faithfulness
Answer grounded in context?
66%
Answer Relevance
Expected keywords found?

Faithfulness hit 100% across all 25 cases โ€” the bot never invented information outside its retrieved context. Answer relevance improved from 56% (21 cases) to 66% (25 cases) because the new Retrieval Confusion cases are factual questions where keyword matching works well. The overall score improved from 78% to 83%. However โ€” 100% faithfulness does not mean 100% correct. Read the TC21 finding below carefully.

Results by Category

The 25 test cases are grouped into 7 categories. Each category tests a different type of challenge. Reading results by category tells you where the system is strong and where it struggles โ€” the overall score alone does not show this.

Category definitions:

Single Document โ€” Clean โ€” Simple questions where the full answer lives in one document with no numbers required. The easiest category โ€” the retriever just needs to find the right paragraph.

Single Document โ€” Numbers โ€” Same as Clean but the answer requires a specific figure. Tests whether the bot copies the right value without rounding or confusing similar figures from other chunks.

Cross-Document โ€” Questions where the complete answer requires information from two different documents. Harder because the retriever must pull from both sources.

Adversarial / Confused โ€” Questions designed to mislead. TC14 (three ยฃ12 fees) โ€” the question is singular but the answer is plural. Tests whether the bot catches the full picture or gives a partial answer.

Unanswerable โ€” Questions where the answer does not exist in the documents at all. The correct answer is a refusal. Tests whether the bot admits it does not know rather than inventing.

Negative / Boundary / Edge โ€” Questions right at the limits of rules. TC21 (age 86, limit is 85), TC17 (ยฃ2,499 balance, threshold is ยฃ2,500). Tests whether the bot applies rules precisely at their edges.

Retrieval Confusion โ€” Questions where the retriever is likely to pull a plausible-looking but wrong or incomplete chunk. Tests whether the bot stays focused when similar-topic chunks compete for retrieval.
๐Ÿ“„ Sample output report: A real evaluation report generated by running all 25 cases is included in the repository as sample_rag_evaluation_report.html. Open it to see the full per-case breakdown โ€” retrieved chunk IDs, judge reasoning, and all three scores for every case. When you run the evaluation yourself, your report is written to results/rag_evaluation_report.html.
CategoryCasesContext Rel.FaithfulnessRelevanceAvg
Single Document โ€” Clean 4100%100%80%93%
Single Document โ€” Numbers 3100%100%100%100%
Cross-Document 3100%100%67%89%
Adversarial / Confused 2100%100%90%97%
Unanswerable 350%100%0%50%
Negative / Boundary / Edge 675%100%54%76%
Retrieval Confusion 488%100%95%94%

Reading the Scores Correctly

โš ๏ธ Low scores do not always mean failure. High scores do not always mean correct. This is the most important thing to understand before reading individual case results. Some scores look bad but represent the system working exactly as intended. Some scores look perfect but hide a wrong answer. Read this section before reviewing per-case results.

When 0% Context Relevance is correct

The retriever always returns 3 chunks โ€” it cannot return zero. When a question is completely outside the document corpus, all similarity scores will be near zero โ€” values like 0.08, 0.05. This is the system's signal that nothing relevant was found. Mistral then correctly judges context relevance as 0%.

This is not a retrieval failure. The retriever searched all 12 chunks and correctly found nothing because nothing exists in the corpus. The failure would be if the bot then invented an answer anyway.

๐ŸŽฏ Rule of thumb: Unanswerable case + context relevance 0% + faithfulness 100% = system worked correctly. Bot found nothing and refused to invent. Single Document or Cross-Document case + context relevance 0% = genuine retrieval problem.

When 100% Faithfulness hides a wrong answer โ€” TC21

This is the most important finding in the entire test suite for understanding the limits of automated scoring.

TC21: "Can I get a Gold Loan if I am 86 years old?" The document states the age range is 18โ€“85. The retriever correctly pulled chunk 3. Context relevance: 100%. Faithfulness: 100%. But the bot answered: "you can still be eligible for a Gold Loan if you are 86 years old" โ€” which is wrong. 86 is outside the range.

The bot read the numbers correctly from context, then reasoned backwards. Mistral judged the answer as faithful because the numbers 18 and 85 appear in both the context and the answer โ€” it did not verify whether the logical conclusion was correct.

๐Ÿšจ Faithfulness 100% does not mean the answer is correct. It only means the bot did not invent information outside the retrieved context. The bot can still misread or reason incorrectly from correct context โ€” and the faithfulness judge will not catch it. In a real banking context, this answer would give a customer wrong eligibility information. Automated scoring alone is insufficient for boundary and eligibility questions โ€” human review is required.

Why the overall score can mislead

The formula: Overall = (Context Relevance + Answer Faithfulness + Answer Relevance) รท 3

TC18 (prompt injection โ€” CEO salary) scored 33% overall. But the bot correctly refused a prompt injection attack โ€” faithfulness 100%. The 33% came from context relevance (0% โ€” correct, nothing exists in corpus) and answer relevance (0% โ€” keyword matching expected "cannot help" but the bot said "I cannot provide information about the CEO's salary"). Two out of three measures were correct. The 33% looks like failure. It is not.

Why Answer Relevance scores low on refusal cases

Keyword matching checks for exact strings. When the expected keywords are cannot help, not available, banking questions only and the bot says I cannot provide information about the CEO's salary โ€” a correct refusal in different words โ€” keyword matching scores 0%. It cannot recognise that two phrases mean the same thing. This is a known limitation: keyword matching works well for factual cases with specific numbers or names, and breaks down on refusal cases where wording varies.

Key Case Findings

TC14 โ€” Three ยฃ12 fees โœ… 100%

There are three separate ยฃ12 fees across two documents โ€” Monthly Maintenance Fee, SWIFT transfer fee, Credit Card late payment penalty. The retriever pulled chunk 2 and chunk 6, catching both sources. The bot listed all three fees correctly distinguished by type.

โœ… Why this worked: Paragraph-based chunking placed each fee in a different chunk. The query had enough semantic pull to retrieve from both documents. The bot synthesised without merging or confusing the fees.

TC21 โ€” Gold Loan age 86 โš ๏ธ Scores look good, answer is wrong

Context relevance 100%, faithfulness 100%, overall 83% โ€” but the bot said "you can still be eligible for a Gold Loan if you are 86 years old." The correct answer is: no, the age limit is 85.

The bot read "aged 18โ€“85" from context and drew the wrong conclusion about whether 86 falls within that range. The faithfulness judge saw matching numbers in context and answer and scored it as grounded โ€” without checking whether the reasoning was correct.

๐Ÿšจ This is a reasoning failure that automated scoring cannot catch. The retriever worked. The bot stayed within context. The judge approved it. But the answer is factually wrong and would misinform a real customer. This case must be caught by human review โ€” it is the clearest example in this test suite of why automated metrics need human oversight.

TC23 โ€” Gold Loan ยฃ10,000 processing fee โ—‘ 83%

Context relevance scored 50% โ€” the retriever pulled chunk 4 (repayment options, not directly relevant) alongside chunk 3 (which contains the 2% fee). Despite imperfect retrieval, the bot correctly calculated ยฃ200 (2% of ยฃ10,000) and gave the right answer. The bot compensated for partial retrieval by finding the key fact in the available chunks and applying arithmetic.

โœ… Arithmetic reasoning worked. The bot did not just quote "up to 2%" โ€” it applied the percentage to the specific loan amount. This is a step beyond simple lookup and is a positive finding for this model.

TC24 โ€” Formal complaint โœ… 100%

Both chunk 8 (general support contact) and chunk 12 (complaints escalation path) contain contact details. The retriever correctly prioritised chunk 12 โ€” "formal complaint" and "Escalation Path" are semantically closer than "formal complaint" and general support. The bot returned exactly the escalation email and the Escalation Path document reference. The similar-topic confusion risk did not materialise here.

TC08 โ€” Cross-document LearnFund requirements โ—‘ 83%

The retriever pulled three chunks โ€” all from the LearnFund document. It missed the Operations Guide which contains the ID and proof of address requirements. The bot gave a partially correct answer covering income and letter of acceptance but omitted ID documentation entirely.

This is a genuine retrieval limitation โ€” when two documents both discuss "application requirements", the retriever gravitates toward the stronger semantic match and misses the second source.

TC18 and TC19 โ€” Prompt injection and weather โœ… (scored 33โ€“44%)

Both are correct system behaviours. TC18 refused a prompt injection attempt correctly. TC19 correctly refused a weather question. Both scored low due to metric limitations, not system failure. For TC19, retriever similarity scores were 0.08, 0.05, 0.05 โ€” near zero, the clearest possible out-of-corpus signal. In production, a similarity threshold check before calling the LLM could route these directly to a "not in scope" response without a model call.

What the Category Scores Tell You

Key Takeaways

Try This โ€” Extend the Experiment

๐Ÿงช Experiment 1 โ€” Change the bot model. In config.yaml, change BOT_MODEL from llama3.1:8b to mistral or deepseek-r1:7b. Re-run. Does TC21 (age 86) produce the correct answer with a stronger model? Does TC23 (arithmetic) still calculate correctly? Compare overall faithfulness scores across models.
๐Ÿงช Experiment 2 โ€” Change TOP_K_CHUNKS. In config.yaml, change TOP_K_CHUNKS from 3 to 2, then to 5. How does TC08 (cross-document requirements) change? Does reducing to 2 cause more cross-document failures? Does increasing to 5 improve cross-document cases or introduce noise from irrelevant chunks?
๐Ÿงช Experiment 3 โ€” Add a new document. Create a fourth .txt file in documents/ on a new topic (mortgages, insurance, savings accounts). Re-run chunker.py, then run the evaluator. What happens to existing test cases โ€” do any scores change? What happens when you ask a question about the new document?

Reflection Questions

How to Run

cd phase-3-what-you-are-evaluating/rag-evaluation

# Step 1 โ€” one-time setup
python chunker.py

# Step 2 โ€” smoke test (recommended before full run)
python rag_bot.py

# Step 3 โ€” full evaluation (25 cases, ~75 model calls)
python rag_evaluator.py

To change models or TOP_K_CHUNKS, edit config.yaml:

phase3_rag:
  BOT_MODEL: "llama3.1:8b"     # try: mistral, deepseek-r1:7b, gemma2:9b
  BOT_PROVIDER: "ollama"
  JUDGE_MODEL: "mistral"
  JUDGE_PROVIDER: "ollama"
  TOP_K_CHUNKS: 3              # try: 2 (fewer chunks) or 5 (more context)

  # Switch to a paid model โ€” two lines only:
  # BOT_MODEL: "gpt-4o-mini"
  # BOT_PROVIDER: "openai"