Phase 4 โ€” Frameworks & Tools

Test 11: DeepEval

DeepEval is a pytest-style LLM testing framework. Instead of writing your own scoring logic, you declare what you want to test and DeepEval runs the assertions. Every metric produces a PASS or FAIL verdict โ€” not just a score. This test runs the same 25 RAG cases from Test 9 through DeepEval to compare framework-based evaluation against hand-rolled evaluation.

๐Ÿ“ phase-4-frameworks-and-tools/deepeval-evaluation/  |  test11_deepeval_evaluator.py ยท report_generator.py ยท rag_bot.py ยท retriever.py ยท chunker.py

What Is DeepEval?

Every evaluator you have built so far โ€” Test 5 LLM-as-a-Judge, Test 6 Semantic Similarity, Test 7 Groundedness, Test 9 RAG โ€” required you to write the scoring logic yourself. You decided what a good answer looks like, how to prompt the judge, and how to interpret the score.

DeepEval wraps all of that. You pass it a test case and a metric, and it handles the judge prompt, the scoring, the threshold check, and the pass/fail verdict. The output is closer to a test runner than a measurement tool โ€” which is exactly the mental model a QA engineer already has.

๐Ÿงช The unit test analogy. In software testing, a test either passes or fails. You don't get a "72% correct" result from a unit test โ€” you get green or red. DeepEval applies the same model to LLM evaluation. Each metric has a threshold (default 0.5 in this test). Score at or above threshold = PASS. Below = FAIL. This is a deliberate design choice โ€” it forces you to define what "good enough" means before you run, not after you see the numbers.

What We Built

Unlike Test 9 which required five files, Test 11 is self-contained โ€” all support files are copied locally so the folder runs independently without relying on any other phase.

File Purpose Source
test11_deepeval_evaluator.py The script you run. Loads test cases, calls the RAG bot per case, builds a LLMTestCase object, runs three DeepEval metrics per case, prints live progress, generates report. New for Test 11
report_generator.py Called automatically at the end. Generates results/deepeval_report.html with PASS/FAIL badges, score cards, and per-case breakdown. New for Test 11
rag_bot.py Same banking RAG assistant as Test 9. Copied locally so this folder has no dependency on phase-3. Copied from Test 9
retriever.py Vector search over chunks.json using sentence-transformers. Copied locally. Copied from Test 9
chunker.py Reads documents/, splits into paragraphs, writes chunks.json. Run once at setup. Copied from Test 9
deepeval_test_cases.xlsx Same 25 test cases as Test 9/10, renamed for clarity. Input only โ€” never written by the script. Renamed from Test 10
๐Ÿ”ง Self-contained design. Every file the evaluator needs lives in the deepeval-evaluation folder. No imports from other phases. Anyone can clone the repo, run python chunker.py once to generate chunks.json, then run python test11_deepeval_evaluator.py โ€” without needing Test 9 or Test 10 to have been run first. This is the correct design for evaluation scripts that will be run in isolation or by other team members.

How DeepEval Works โ€” The Key Concepts

LLMTestCase โ€” the unit of evaluation

Everything in DeepEval is built around a LLMTestCase object. You provide three things:

Field What it is In this test
input The question asked The Question column from Excel
actual_output What the bot said llama3.1:8b response via Ollama
retrieval_context The chunks the RAG bot used Top-3 chunks from retriever.py โ€” as a list of strings

Once a LLMTestCase is built, you pass it to any metric. The metric sends it to the judge model and returns a score, a reason, and a pass/fail verdict.

OllamaModel โ€” running without a paid API

DeepEval was designed with OpenAI as the default judge. To use a local Ollama model instead, you register it once via the CLI:

deepeval set-ollama --model=mistral --base-url="http://localhost:11434"

This writes a config file on your machine. After that, DeepEval uses mistral for all judge calls โ€” no OpenAI key needed. In code, you instantiate OllamaModel and pass it to each metric.

โš ๏ธ async_mode=False is required for local models. DeepEval defaults to async_mode=True โ€” it fires multiple concurrent requests to the judge. Ollama cannot handle concurrent requests locally. It queues them, the async calls time out waiting, and scores default to 0.0 โ€” appearing as FAILs. Always set async_mode=False when using Ollama. This is a one-line fix per metric: ContextualRelevancyMetric(threshold=0.5, model=judge, async_mode=False).

The three metrics

Metric DeepEval class What it judges Equivalent in Test 9
Contextual Relevancy ContextualRelevancyMetric Do the retrieved chunks contain what's needed to answer the question? Context Relevance (LLM judge)
Faithfulness FaithfulnessMetric Is every claim in the answer supported by the retrieved context, or was something invented? Answer Faithfulness (LLM judge)
Answer Relevancy AnswerRelevancyMetric Does the answer directly address what was asked? Answer Relevance (keyword matching in Test 9 โ€” LLM judge here)
๐Ÿ”— Key difference from Test 9: In Test 9, Answer Relevance used keyword matching โ€” it checked whether expected keywords appeared in the bot's response. In DeepEval, AnswerRelevancyMetric uses the LLM judge โ€” it asks whether the answer semantically addresses the question. This is a stronger, more flexible measure. It can recognise that "I cannot provide information about the CEO's salary" is a correct refusal even if none of the expected keywords appear.

How to Run

Step 1 โ€” one-time setup

pip install deepeval --break-system-packages
deepeval set-ollama --model=mistral --base-url="http://localhost:11434"

Step 2 โ€” generate chunks (first time only)

cd phase-4-frameworks-and-tools/deepeval-evaluation
python chunker.py

Step 3 โ€” run the evaluator

python test11_deepeval_evaluator.py

The terminal shows live progress for all 25 cases. Each case prints the bot answer, chunk count, and three scoring lines. Output is written to results/deepeval_report.html.

โš ๏ธ Runtime warning โ€” this is slow on local Ollama. With async_mode=False, all 75 judge calls (25 cases ร— 3 metrics) run sequentially. Each call waits for mistral to respond. Expect 10โ€“20 minutes for the first 15 cases โ€” then timeouts may begin as Ollama's memory fills under sustained load. See the findings section below for the full explanation of what happened in this run and how to address it.

Overall Results

โš ๏ธ Read before interpreting these numbers. TC01โ€“TC15 scored fully โ€” mistral judged all three metrics successfully. TC16โ€“TC25 ran in a second batch: ContextualRelevancy and Faithfulness timed out on several cases (defaulting to 0.00), but AnswerRelevancy scored correctly on all. The numbers below reflect all 25 cases with the second-run scores applied. TC19 (weather question) is a correct refusal and was excluded from scoring. Only TC15 officially passed (all 3 metrics โ‰ฅ 0.5). Read the findings section for the full explanation of the timeout problem.
62%
Overall Avg
All 25 cases ยท 24 scored (TC19 refusal excluded)
24%
Context Relevance
3/24 passed โ€” timeouts on TC16โ€“25 defaulted to 0.00
82%
Faithfulness
22/24 passed โ€” strongest metric overall
81%
Answer Relevance
20/24 passed โ€” LLM judge, not keyword matching
1/25
Overall PASS
TC15 โ€” only case where all 3 metrics โ‰ฅ 0.5
0.5
Threshold
Score below this = FAIL on that metric
TC01โ€“15
Fully Scored
All three metrics judged by mistral correctly
TC16โ€“25
Partially Scored
Answer Rel. valid; Context Rel. / Faithfulness timed out on some cases

The Timeout Problem โ€” What Happened and Why

This is the most important finding from Test 11 โ€” not the scores, but what caused the scores to break down.

What happened

TC01โ€“TC15 ran correctly โ€” mistral judged all three metrics per case and returned real scores. From TC16 onward, the terminal began printing RetryError[TimeoutError] warnings. When a metric times out, DeepEval catches the exception and defaults the score to 0.0, marking it as FAIL. By TC20โ€“TC25, all three metrics were timing out on almost every case.

Why it happened

DeepEval's ContextualRelevancyMetric and FaithfulnessMetric are internally decomposed โ€” they send multiple sub-prompts to the judge to build their score. Each case makes significantly more than 3 judge calls. After 15 cases (roughly 2โ€“3 hours of sustained Ollama load), the machine's available RAM was exhausted, Ollama was swapping memory, and response times grew beyond DeepEval's internal timeout window โ€” even with async_mode=False.

๐Ÿšจ This is not a DeepEval bug โ€” it is a local resource constraint. DeepEval is designed to run against a paid API (OpenAI, Anthropic) where response times are fast and consistent. Running it against a local Ollama model on a 16GB laptop for 5 hours is outside its intended use case. The framework works โ€” the infrastructure does not scale to this volume locally.

All 25 cases โ€” full results

TC01โ€“TC15 were scored fully in the first run. TC16โ€“TC25 ran in a second batch: ContextualRelevancy and Faithfulness timed out on several cases (shown as 0.00*), but AnswerRelevancy scored correctly on all. TC19 is a correct refusal (weather question) and is excluded from metric scoring.

Case Category Context Rel. Faithfulness Answer Rel. Verdict Notes
TC01 Single Doc โ€” Clean 0.41 F 1.00 P 1.00 P FAIL Context fails despite correct retrieval โ€” judge penalises supporting chunks
TC02 Single Doc โ€” Clean 0.18 F 1.00 P 1.00 P FAIL Context 0.18 โ€” correct answer from correct chunk still fails context check
TC03 Single Doc โ€” Clean 0.20 F 1.00 P 1.00 P FAIL Same calibration pattern โ€” context judge harsh on supporting chunks
TC04 Single Doc โ€” Clean 0.35 F 1.00 P 1.00 P FAIL LearnFund degree rate โ€” context just below threshold
TC05 Single Doc โ€” Numbers 0.06 F 0.50 P 1.00 P FAIL Unarranged overdraft โ€” context near zero; faithfulness borderline at 0.50
TC06 Single Doc โ€” Numbers 0.41 F 1.00 P 1.00 P FAIL Account opening docs โ€” 4/14 context statements relevant per judge
TC07 Single Doc โ€” Numbers 0.13 F 1.00 P 1.00 P FAIL ATM limit โ€” answer correct; judge finds only 1 directly relevant statement
TC08 Cross-Document 0.28 F 1.00 P 1.00 P FAIL LearnFund loan docs โ€” 1/8 context statements relevant per judge
TC09 Cross-Document 0.52 P 1.00 P 0.33 F FAIL Monthly fee โ€” context passes; answer relevance fails (LLM judge stricter than keywords)
TC10 Cross-Document 0.38 F 1.00 P 1.00 P FAIL Cross-doc retrieval โ€” context borderline below 0.5
TC11 Unanswerable 0.50 P 0.00 F 0.50 P FAIL Personal loan โ€” correct refusal; faithfulness 0.00 (refusal has no grounding claim)
TC12 Unanswerable 0.20 F 1.00 P 0.00 F FAIL Unanswerable โ€” context and answer relevance both fail
TC13 Unanswerable 0.33 F 1.00 P 1.00 P FAIL Student account refusal โ€” LLM judge correctly scores relevance; context still fails
TC14 Negative / Boundary 0.13 F 1.00 P 0.50 P FAIL Boundary case โ€” context very low; answer and faithfulness ok
TC15 Adversarial / Confused 0.58 P 1.00 P 1.00 P PASS Inactive vs dormant โ€” only case to fully pass all 3 metrics
* TC16โ€“TC25: second run โ€” ContextualRelevancy and Faithfulness timed out on several cases (0.00 = timeout default). AnswerRelevancy scored correctly on all.
TC16 Negative / Boundary 0.00* F 0.00* F 1.00 P FAIL Context and faith timed out; answer relevance correctly scored 1.00
TC17 Negative / Boundary 0.14* F 0.50 P 0.50 P FAIL Context partial timeout; faith and answer rel. borderline pass
TC18 Negative / Boundary 0.00* F 1.00 P 0.50 P FAIL Context timed out; faithfulness strong at 1.00
TC19 Negative / Boundary โŠ˜ Refusal โ€” weather question is out of scope. Excluded from scoring. N/A Only refusal in TC16โ€“25 batch
TC20 Retrieval Confusion 0.20* F 0.50 P 0.67 P FAIL Context timed out; faith borderline; answer rel. correctly scored
TC21 Retrieval Confusion 0.17* F 1.00 P 0.00 F FAIL Answer relevance 0.00 โ€” judge found answer did not address question
TC22 Retrieval Confusion 0.20* F 1.00 P 1.00 P FAIL Faith and rel. both strong; context timed out keeping it from passing
TC23 Retrieval Confusion 0.16* F 0.50 P 0.67 P FAIL Context timed out; other metrics borderline pass
TC24 Adversarial / Confused 0.20* F 0.50 P 1.00 P FAIL Answer relevance full pass; context timeout prevents overall pass
TC25 Negative / Boundary 0.23* F 0.50 P 1.00 P FAIL Same pattern โ€” context timeout; answer correctly scored at 1.00

The Context Relevance Calibration Problem

The single most striking pattern in TC01โ€“TC15 is how harshly DeepEval's ContextualRelevancyMetric scored context relevance compared to Test 9.

Test 9 โ€” Context Relevance (hand-rolled)
84%
Average across 25 cases. Most clean single-document cases scored 100%.
Test 11 โ€” Context Relevance (DeepEval)
24%
Average across all 24 scored cases. TC01โ€“TC15 had real judgements; TC16โ€“TC25 had timeouts defaulting to 0.00 on most.

TC01 is a clean example: "How long does it take to open a NextGen account?" The retriever pulled the correct chunks. The bot gave the correct answer. Test 9 scored this context relevance at 100%. DeepEval scored it at 0.41 โ€” a FAIL.

This is a calibration difference, not a system failure. DeepEval's ContextualRelevancyMetric internally asks whether each retrieved chunk is directly relevant to the question โ€” it penalises any chunk that is topically related but not specifically needed. With TOP_K=3, the third chunk is often a supporting or nearby paragraph that adds background but is not the primary source of the answer. DeepEval's judge penalises this. Test 9's hand-rolled judge was more lenient โ€” it asked "is there enough context here to answer?" rather than "is every chunk directly relevant?"

๐ŸŽฏ This is the key learning from Test 11. Two different judges, same metric name, meaningfully different scores. The label "context relevance" does not guarantee identical evaluation โ€” it depends entirely on how the judge prompt is written. When you compare scores across evaluation frameworks, always check whether the underlying prompt is asking the same question. A score of 84% in Test 9 and 27% in Test 11 does not mean one system is wrong โ€” it means the two judges have different calibration.

TC11 โ€” Faithfulness 0.0 on a Correct Refusal

TC11: "What is the interest rate on a NextGen personal loan?" The bot answered: "The information provided in the context does not mention the interest rate on a NextGen personal loan." This is a correct answer โ€” the documentation does not contain this information.

DeepEval scored faithfulness at 0.0. Why? FaithfulnessMetric decomposes the answer into claims and checks each claim against the context. A refusal statement like "the information is not available" is not a claim supported by a chunk โ€” it is an absence of information. DeepEval's faithfulness judge does not have a category for "correct absence of claim" โ€” it treats uncorroborated statements as unfaithful.

โš ๏ธ Faithfulness metrics struggle with correct refusals. This is a known limitation across all faithfulness evaluators โ€” not just DeepEval. When the correct answer is "I don't know", there is no chunk to support that statement. The judge marks it as a faithfulness failure. Human review is required for unanswerable cases โ€” automated faithfulness scoring alone will systematically penalise correct refusals.

TC15 โ€” The One PASS

TC15: "What is the difference between an inactive and a dormant account?" This is the only case that passed all three metrics.

Context relevance 0.58, faithfulness 1.00, answer relevance 1.00. The question has a clear factual answer that maps directly to the retrieved chunks. The distinction between inactive (12 months, no transactions) and dormant (24 months, no transactions) is precisely described in one chunk. No boundary reasoning is required. No multi-document synthesis. The retriever pulled the right chunk and the bot quoted it accurately.

โœ… TC15 is the clearest signal that the pipeline works. When retrieval is clean and the answer is a direct lookup, DeepEval correctly issues a PASS. The 1/25 pass rate is not evidence that the RAG system is broken โ€” it is evidence that the context relevance threshold (0.5) is strict relative to DeepEval's judge calibration, and that the timeout problem corrupted TC16โ€“TC25.

Comparing Test 9 vs Test 11 โ€” Valid Cases Only

Using only TC01โ€“TC15 where both tests scored correctly:

Metric Test 9 (hand-rolled) Test 11 DeepEval (TC01โ€“15) Key difference
Context Relevance 84% ~27% DeepEval penalises supporting chunks; Test 9 asked "enough context?"
Faithfulness 100% ~87% DeepEval scores refusals as 0.0; Test 9 scored TC11 correctly
Answer Relevance 66% ~83% DeepEval uses LLM judge โ€” recognises correct refusals; Test 9 used keyword matching
๐Ÿ“Š Answer relevance improved with a better judge. Test 9's keyword matching scored TC13 (student account โ€” not in docs) at 0% answer relevance because none of the expected keywords appeared in the refusal. DeepEval's LLM judge correctly recognised the refusal as relevant and scored it higher. This is the concrete benefit of LLM-as-judge over keyword matching for open-ended and refusal cases.

What DeepEval Does That Your Hand-Rolled Evaluator Doesn't

Capability Test 9 hand-rolled Test 11 DeepEval
Pass/Fail verdict No โ€” score only Yes โ€” per metric, per case
Threshold configuration No โ€” you interpret the score Yes โ€” set at metric instantiation
Judge reasoning Partial โ€” one reason per case Yes โ€” per metric with include_reason=True
Metric decomposition No โ€” single judge prompt Yes โ€” faithfulness decomposes into per-claim checks
pytest integration No Yes โ€” can run as deepeval test run in CI/CD
Local model support Yes โ€” built in from the start Yes โ€” with OllamaModel and async_mode=False
Calibration transparency High โ€” you wrote the prompt Low โ€” judge prompt is internal to DeepEval

When to Use DeepEval vs Hand-Rolled

Test 11 surfaces a genuine trade-off that you will face in production evaluation work:

๐ŸŽฏ The calibration lesson. Before adopting any evaluation framework in production, run it on a set of cases where you already know the correct verdict โ€” either from human review or from a trusted hand-rolled evaluator. Check how well the framework agrees. If it systematically disagrees on a category (e.g. all refusal cases), understand why before deploying it as your primary quality gate.

The Local vs Paid API Problem

The 5-hour runtime and timeout cascade from TC16 onward is a concrete demonstration of a trade-off you will face on every evaluation project:

Local Ollama
Free
Zero cost. Full privacy. But slow, memory-constrained, and not suitable for sustained multi-metric evaluation runs with frameworks like DeepEval.
Paid API (OpenAI / Anthropic)
~$0.50
25 cases ร— 3 metrics at gpt-4o-mini pricing. Fast, consistent, no timeouts. DeepEval was designed for this environment.

For learning and prototyping, Ollama is the right choice โ€” it allowed you to run hundreds of iterations through Tests 1โ€“10 without any cost. For production evaluation pipelines that need to run reliably, on schedule, and at scale, a paid API is the practical choice. The cost is lower than it appears โ€” 25 evaluation cases costs less than a cup of coffee.

Key Takeaways

Reflection Questions

Try This โ€” Extend the Experiment

๐Ÿงช Experiment 1 โ€” Run with a paid API judge. When OpenAI credits are available, change JUDGE_MODEL to gpt-4o-mini and JUDGE_PROVIDER to openai in config.yaml. Re-run. Compare context relevance scores against the local mistral run. Does the calibration gap close? Does the pass rate improve?
๐Ÿงช Experiment 2 โ€” Adjust the threshold. Change METRIC_THRESHOLD in config.yaml from 0.5 to 0.3. Re-run TC01โ€“TC15. How many cases now pass? What does this tell you about the relationship between threshold selection and reported pass rates?
๐Ÿงช Experiment 3 โ€” Run fewer cases to avoid timeouts. Add MAX_CASES: 10 to the phase4_deepeval config section and update the script to respect it. Run only the first 10 cases. Confirm all 10 score without timeouts. Compare those 10 results against the same 10 cases from Test 9.

How to Run

cd phase-4-frameworks-and-tools/deepeval-evaluation

# Step 1 โ€” one-time setup (install + register local judge)
pip install deepeval --break-system-packages
deepeval set-ollama --model=mistral --base-url="http://localhost:11434"

# Step 2 โ€” generate chunks (first time only)
python chunker.py

# Step 3 โ€” run evaluation
python test11_deepeval_evaluator.py

To switch to a paid judge โ€” two lines in config.yaml:

phase4_deepeval:
  JUDGE_MODEL: "gpt-4o-mini"
  JUDGE_PROVIDER: "openai"
  # OPENAI_API_KEY: "sk-..."   # or use env var