RAG stands for Retrieval Augmented Generation. Instead of relying on what the model was trained on, a RAG system first searches a document corpus and gives the model only the relevant text to answer from. This test builds a RAG pipeline from scratch and evaluates whether it retrieves the right information, answers faithfully from that information, and addresses what was actually asked.
A standard LLM answers from its training data โ knowledge baked in at training time that cannot be updated and cannot be verified. For a banking assistant, this is a serious problem. You need the bot to answer from your documents, not from general knowledge that may be wrong, outdated, or hallucinated.
RAG solves this by giving the model context at the moment the question is asked. The model only sees the retrieved text โ it has no choice but to answer from it. If the answer is not in the retrieved text, a well-prompted RAG bot will say so rather than invent one.
Three fictional NextGen Banking documents form the corpus for this test. All product names, fees, and policies are invented for this exercise.
| Document | Contents | Chunks |
|---|---|---|
Fees_Loans_and_Secured_Assets_Guide.txt |
Monthly fees, overdraft charges, SWIFT/SEPA transfer fees, Gold Loans, Loan Against Securities, repayment options, tax notes | Chunks 1โ4 |
LearnFund_Credit_Card_Product_Guide.txt |
LearnFund education loan โ approved providers, interest rates, eligibility, documentation required, early settlement fees, Credit Card APR and late payment penalty | Chunks 5โ8 |
Operations_Guide.txt |
Account opening process, required ID documents, dormant/inactive account rules, reactivation procedure, ATM limits, complaints escalation | Chunks 9โ12 |
A five-file pipeline built from scratch. No external RAG framework โ every step is visible and understandable.
| File | Purpose | Run? |
|---|---|---|
chunker.py |
Reads the 3 source documents, splits them into paragraphs on blank lines, filters fragments under 40 characters, saves to chunks.json. Produced 12 chunks โ 4 per document. |
Once at setup |
retriever.py |
Loads chunks.json, encodes all chunks as vectors using sentence-transformers/all-MiniLM-L6-v2, then for any question returns the top-K most similar chunks by cosine similarity. Uses a _load() pattern โ model loads once on first call and is reused for all questions. Never runs directly. |
Library only |
rag_bot.py |
The banking assistant. Takes a question โ calls the retriever โ builds a prompt with retrieved chunks as context โ calls llama3.1:8b with a strict instruction to answer only from context. Returns the answer plus the retrieved chunk IDs. Never runs directly. |
Library only |
rag_evaluator.py |
The script you run. Reads all 25 test cases from Excel, calls rag_bot.py per case, scores each response three ways, prints live progress to terminal, then calls report_generator.py at the end. |
This one |
report_generator.py |
Called automatically by the evaluator at the end. Generates results/rag_evaluation_report.html. Never run directly. |
Auto |
After running chunker.py, open chunks.json to see exactly what the retriever searches over. Each chunk looks like this:
{
"id": 2,
"source": "Fees_Loans_and_Secured_Assets_Guide.txt",
"text": "Our service charge structure is designed for transparency.
The standard Monthly Maintenance Fee is ยฃ12.00, which is
waived entirely for customers who maintain a Minimum Monthly
Balance of ยฃ2,500 or have a total loan liability exceeding
ยฃ15,000..."
}
Every chunk has an ID, a source filename, and the paragraph text. When the terminal shows "chunk 2, chunk 6, chunk 1" โ these are the IDs. Open chunks.json and read exactly what context the bot was given for that question.
cd phase-3-what-you-are-evaluating/rag-evaluation python chunker.py
This reads the three documents and writes chunks.json. You only need to run this once. If chunks.json already exists, skip this step.
Before running all 25 cases, run rag_bot.py directly with a few sample questions to confirm the pipeline works end to end:
python rag_bot.py
The __main__ block runs 3 sample questions and prints the retrieved chunks and bot answer for each. You will see something like:
Q: What is the monthly maintenance fee? Chunks: chunk 2, chunk 1, chunk 7 (scores: 0.72, 0.41, 0.38) A: The standard Monthly Maintenance Fee is ยฃ12.00...
If this works, the full evaluation will work. If it fails here, fix the issue before running 25 cases.
# Runs all 25 cases โ approximately 75 model calls (~15โ20 minutes on local Ollama) python rag_evaluator.py
The terminal shows live progress โ which chunks were retrieved, similarity scores, response time, and all three scores per case. Output is written to results/rag_evaluation_report.html.
Every case shows which chunks were retrieved and their similarity scores. Learn to read these as the evaluation runs:
| Score range | What it means |
|---|---|
0.75 โ 1.00 | Strong match โ retriever is confident this chunk is relevant to the question |
0.50 โ 0.74 | Moderate match โ related topic but may not contain the exact answer |
0.25 โ 0.49 | Weak match โ retriever is reaching; chunk may only share a general topic |
below 0.20 | Near zero โ question is almost certainly outside the corpus entirely |
pip install sentence-transformers --break-system-packagespython chunker.py first.ollama serve in a separate terminal.ollama pull llama3.1:8b and ollama pull mistral.embeddings.position_ids UNEXPECTED message appears because the embedding model was saved from a slightly different architecture. It does not affect results โ ignore it.
Every test case is scored on three dimensions independently. The overall score is a simple average:
| Metric | Method | What it measures | Score range |
|---|---|---|---|
| Context Relevance | LLM judge (mistral) | Did the retriever pull chunks that actually contain what's needed to answer the question? | 0% / 50% / 100% |
| Answer Faithfulness | LLM judge (mistral) | Did the bot answer only from the retrieved chunks, or did it add or invent information not present in context? | 0% / 50% / 100% |
| Answer Relevance | Keyword matching | Does the answer contain the expected keywords? Proportion of expected keywords found in the response. | 0% โ 100% |
TOP_K_CHUNKS: 3 in config.yaml. Too few (1โ2) and cross-document questions miss important context. Too many (6+) and the LLM's context fills with irrelevant chunks, diluting the signal. 3 is a reasonable default for a 12-chunk corpus. For larger corpora this value needs tuning based on how spread out the answers are across documents.
Faithfulness hit 100% across all 25 cases โ the bot never invented information outside its retrieved context. Answer relevance improved from 56% (21 cases) to 66% (25 cases) because the new Retrieval Confusion cases are factual questions where keyword matching works well. The overall score improved from 78% to 83%. However โ 100% faithfulness does not mean 100% correct. Read the TC21 finding below carefully.
The 25 test cases are grouped into 7 categories. Each category tests a different type of challenge. Reading results by category tells you where the system is strong and where it struggles โ the overall score alone does not show this.
sample_rag_evaluation_report.html. Open it to see the full per-case breakdown โ retrieved chunk IDs, judge reasoning, and all three scores for every case. When you run the evaluation yourself, your report is written to results/rag_evaluation_report.html.
| Category | Cases | Context Rel. | Faithfulness | Relevance | Avg |
|---|---|---|---|---|---|
| Single Document โ Clean | 4 | 100% | 100% | 80% | 93% |
| Single Document โ Numbers | 3 | 100% | 100% | 100% | 100% |
| Cross-Document | 3 | 100% | 100% | 67% | 89% |
| Adversarial / Confused | 2 | 100% | 100% | 90% | 97% |
| Unanswerable | 3 | 50% | 100% | 0% | 50% |
| Negative / Boundary / Edge | 6 | 75% | 100% | 54% | 76% |
| Retrieval Confusion | 4 | 88% | 100% | 95% | 94% |
The retriever always returns 3 chunks โ it cannot return zero. When a question is completely outside the document corpus, all similarity scores will be near zero โ values like 0.08, 0.05. This is the system's signal that nothing relevant was found. Mistral then correctly judges context relevance as 0%.
This is not a retrieval failure. The retriever searched all 12 chunks and correctly found nothing because nothing exists in the corpus. The failure would be if the bot then invented an answer anyway.
This is the most important finding in the entire test suite for understanding the limits of automated scoring.
TC21: "Can I get a Gold Loan if I am 86 years old?" The document states the age range is 18โ85. The retriever correctly pulled chunk 3. Context relevance: 100%. Faithfulness: 100%. But the bot answered: "you can still be eligible for a Gold Loan if you are 86 years old" โ which is wrong. 86 is outside the range.
The bot read the numbers correctly from context, then reasoned backwards. Mistral judged the answer as faithful because the numbers 18 and 85 appear in both the context and the answer โ it did not verify whether the logical conclusion was correct.
The formula: Overall = (Context Relevance + Answer Faithfulness + Answer Relevance) รท 3
TC18 (prompt injection โ CEO salary) scored 33% overall. But the bot correctly refused a prompt injection attack โ faithfulness 100%. The 33% came from context relevance (0% โ correct, nothing exists in corpus) and answer relevance (0% โ keyword matching expected "cannot help" but the bot said "I cannot provide information about the CEO's salary"). Two out of three measures were correct. The 33% looks like failure. It is not.
Keyword matching checks for exact strings. When the expected keywords are cannot help, not available, banking questions only and the bot says I cannot provide information about the CEO's salary โ a correct refusal in different words โ keyword matching scores 0%. It cannot recognise that two phrases mean the same thing. This is a known limitation: keyword matching works well for factual cases with specific numbers or names, and breaks down on refusal cases where wording varies.
There are three separate ยฃ12 fees across two documents โ Monthly Maintenance Fee, SWIFT transfer fee, Credit Card late payment penalty. The retriever pulled chunk 2 and chunk 6, catching both sources. The bot listed all three fees correctly distinguished by type.
Context relevance 100%, faithfulness 100%, overall 83% โ but the bot said "you can still be eligible for a Gold Loan if you are 86 years old." The correct answer is: no, the age limit is 85.
The bot read "aged 18โ85" from context and drew the wrong conclusion about whether 86 falls within that range. The faithfulness judge saw matching numbers in context and answer and scored it as grounded โ without checking whether the reasoning was correct.
Context relevance scored 50% โ the retriever pulled chunk 4 (repayment options, not directly relevant) alongside chunk 3 (which contains the 2% fee). Despite imperfect retrieval, the bot correctly calculated ยฃ200 (2% of ยฃ10,000) and gave the right answer. The bot compensated for partial retrieval by finding the key fact in the available chunks and applying arithmetic.
Both chunk 8 (general support contact) and chunk 12 (complaints escalation path) contain contact details. The retriever correctly prioritised chunk 12 โ "formal complaint" and "Escalation Path" are semantically closer than "formal complaint" and general support. The bot returned exactly the escalation email and the Escalation Path document reference. The similar-topic confusion risk did not materialise here.
The retriever pulled three chunks โ all from the LearnFund document. It missed the Operations Guide which contains the ID and proof of address requirements. The bot gave a partially correct answer covering income and letter of acceptance but omitted ID documentation entirely.
This is a genuine retrieval limitation โ when two documents both discuss "application requirements", the retriever gravitates toward the stronger semantic match and misses the second source.
Both are correct system behaviours. TC18 refused a prompt injection attempt correctly. TC19 correctly refused a weather question. Both scored low due to metric limitations, not system failure. For TC19, retriever similarity scores were 0.08, 0.05, 0.05 โ near zero, the clearest possible out-of-corpus signal. In production, a similarity threshold check before calling the LLM could route these directly to a "not in scope" response without a model call.
config.yaml, change BOT_MODEL from llama3.1:8b to mistral or deepseek-r1:7b. Re-run. Does TC21 (age 86) produce the correct answer with a stronger model? Does TC23 (arithmetic) still calculate correctly? Compare overall faithfulness scores across models.
config.yaml, change TOP_K_CHUNKS from 3 to 2, then to 5. How does TC08 (cross-document requirements) change? Does reducing to 2 cause more cross-document failures? Does increasing to 5 improve cross-document cases or introduce noise from irrelevant chunks?
documents/ on a new topic (mortgages, insurance, savings accounts). Re-run chunker.py, then run the evaluator. What happens to existing test cases โ do any scores change? What happens when you ask a question about the new document?
cd phase-3-what-you-are-evaluating/rag-evaluation # Step 1 โ one-time setup python chunker.py # Step 2 โ smoke test (recommended before full run) python rag_bot.py # Step 3 โ full evaluation (25 cases, ~75 model calls) python rag_evaluator.py
To change models or TOP_K_CHUNKS, edit config.yaml:
phase3_rag: BOT_MODEL: "llama3.1:8b" # try: mistral, deepseek-r1:7b, gemma2:9b BOT_PROVIDER: "ollama" JUDGE_MODEL: "mistral" JUDGE_PROVIDER: "ollama" TOP_K_CHUNKS: 3 # try: 2 (fewer chunks) or 5 (more context) # Switch to a paid model โ two lines only: # BOT_MODEL: "gpt-4o-mini" # BOT_PROVIDER: "openai"