DeepEval is a pytest-style LLM testing framework. Instead of writing your own scoring logic, you declare what you want to test and DeepEval runs the assertions. Every metric produces a PASS or FAIL verdict โ not just a score. This test runs the same 25 RAG cases from Test 9 through DeepEval to compare framework-based evaluation against hand-rolled evaluation.
Every evaluator you have built so far โ Test 5 LLM-as-a-Judge, Test 6 Semantic Similarity, Test 7 Groundedness, Test 9 RAG โ required you to write the scoring logic yourself. You decided what a good answer looks like, how to prompt the judge, and how to interpret the score.
DeepEval wraps all of that. You pass it a test case and a metric, and it handles the judge prompt, the scoring, the threshold check, and the pass/fail verdict. The output is closer to a test runner than a measurement tool โ which is exactly the mental model a QA engineer already has.
Unlike Test 9 which required five files, Test 11 is self-contained โ all support files are copied locally so the folder runs independently without relying on any other phase.
| File | Purpose | Source |
|---|---|---|
test11_deepeval_evaluator.py |
The script you run. Loads test cases, calls the RAG bot per case, builds a
LLMTestCase object, runs three DeepEval metrics per case, prints live progress, generates report.
|
New for Test 11 |
report_generator.py |
Called automatically at the end. Generates results/deepeval_report.html with PASS/FAIL badges,
score cards, and per-case breakdown. |
New for Test 11 |
rag_bot.py |
Same banking RAG assistant as Test 9. Copied locally so this folder has no dependency on phase-3. | Copied from Test 9 |
retriever.py |
Vector search over chunks.json using sentence-transformers. Copied locally. | Copied from Test 9 |
chunker.py |
Reads documents/, splits into paragraphs, writes chunks.json. Run once at setup. | Copied from Test 9 |
deepeval_test_cases.xlsx |
Same 25 test cases as Test 9/10, renamed for clarity. Input only โ never written by the script. | Renamed from Test 10 |
python chunker.py once to generate
chunks.json, then run python test11_deepeval_evaluator.py โ without needing Test 9 or Test 10 to have
been run first. This is the correct design for evaluation scripts that will be run in isolation or by other team
members.
Everything in DeepEval is built around a LLMTestCase object. You provide three things:
| Field | What it is | In this test |
|---|---|---|
input |
The question asked | The Question column from Excel |
actual_output |
What the bot said | llama3.1:8b response via Ollama |
retrieval_context |
The chunks the RAG bot used | Top-3 chunks from retriever.py โ as a list of strings |
Once a LLMTestCase is built, you pass it to any metric. The metric sends it to the judge model and
returns a score, a reason, and a pass/fail verdict.
DeepEval was designed with OpenAI as the default judge. To use a local Ollama model instead, you register it once via the CLI:
deepeval set-ollama --model=mistral --base-url="http://localhost:11434"
This writes a config file on your machine. After that, DeepEval uses mistral for all judge calls โ no OpenAI key
needed. In code, you instantiate OllamaModel and pass it to each metric.
async_mode=True
โ it fires multiple concurrent requests to the judge. Ollama cannot handle concurrent requests locally. It queues
them, the async calls time out waiting, and scores default to 0.0 โ appearing as FAILs. Always set
async_mode=False when using Ollama. This is a one-line fix per metric:
ContextualRelevancyMetric(threshold=0.5, model=judge, async_mode=False).
| Metric | DeepEval class | What it judges | Equivalent in Test 9 |
|---|---|---|---|
| Contextual Relevancy | ContextualRelevancyMetric |
Do the retrieved chunks contain what's needed to answer the question? | Context Relevance (LLM judge) |
| Faithfulness | FaithfulnessMetric |
Is every claim in the answer supported by the retrieved context, or was something invented? | Answer Faithfulness (LLM judge) |
| Answer Relevancy | AnswerRelevancyMetric |
Does the answer directly address what was asked? | Answer Relevance (keyword matching in Test 9 โ LLM judge here) |
AnswerRelevancyMetric uses the
LLM judge โ it asks whether the answer semantically addresses the question. This is a stronger, more flexible
measure. It can recognise that "I cannot provide information about the CEO's salary" is a correct refusal even if
none of the expected keywords appear.
pip install deepeval --break-system-packages deepeval set-ollama --model=mistral --base-url="http://localhost:11434"
cd phase-4-frameworks-and-tools/deepeval-evaluation python chunker.py
python test11_deepeval_evaluator.py
The terminal shows live progress for all 25 cases. Each case prints the bot answer, chunk count, and three scoring
lines. Output is written to results/deepeval_report.html.
async_mode=False, all 75 judge
calls (25 cases ร 3 metrics) run sequentially. Each call waits for mistral to respond. Expect 10โ20 minutes for the
first 15 cases โ then timeouts may begin as Ollama's memory fills under sustained load. See the findings section
below for the full explanation of what happened in this run and how to address it.
This is the most important finding from Test 11 โ not the scores, but what caused the scores to break down.
TC01โTC15 ran correctly โ mistral judged all three metrics per case and returned real scores. From TC16 onward, the
terminal began printing RetryError[TimeoutError] warnings. When a metric times out, DeepEval catches
the exception and defaults the score to 0.0, marking it as FAIL. By TC20โTC25, all three metrics were timing out on
almost every case.
DeepEval's ContextualRelevancyMetric and FaithfulnessMetric are internally decomposed โ
they send multiple sub-prompts to the judge to build their score. Each case makes significantly more than 3 judge
calls. After 15 cases (roughly 2โ3 hours of sustained Ollama load), the machine's available RAM was exhausted,
Ollama was swapping memory, and response times grew beyond DeepEval's internal timeout window โ even with
async_mode=False.
TC01โTC15 were scored fully in the first run. TC16โTC25 ran in a second batch: ContextualRelevancy and Faithfulness timed out on several cases (shown as 0.00*), but AnswerRelevancy scored correctly on all. TC19 is a correct refusal (weather question) and is excluded from metric scoring.
| Case | Category | Context Rel. | Faithfulness | Answer Rel. | Verdict | Notes |
|---|---|---|---|---|---|---|
| TC01 | Single Doc โ Clean | 0.41 F | 1.00 P | 1.00 P | FAIL | Context fails despite correct retrieval โ judge penalises supporting chunks |
| TC02 | Single Doc โ Clean | 0.18 F | 1.00 P | 1.00 P | FAIL | Context 0.18 โ correct answer from correct chunk still fails context check |
| TC03 | Single Doc โ Clean | 0.20 F | 1.00 P | 1.00 P | FAIL | Same calibration pattern โ context judge harsh on supporting chunks |
| TC04 | Single Doc โ Clean | 0.35 F | 1.00 P | 1.00 P | FAIL | LearnFund degree rate โ context just below threshold |
| TC05 | Single Doc โ Numbers | 0.06 F | 0.50 P | 1.00 P | FAIL | Unarranged overdraft โ context near zero; faithfulness borderline at 0.50 |
| TC06 | Single Doc โ Numbers | 0.41 F | 1.00 P | 1.00 P | FAIL | Account opening docs โ 4/14 context statements relevant per judge |
| TC07 | Single Doc โ Numbers | 0.13 F | 1.00 P | 1.00 P | FAIL | ATM limit โ answer correct; judge finds only 1 directly relevant statement |
| TC08 | Cross-Document | 0.28 F | 1.00 P | 1.00 P | FAIL | LearnFund loan docs โ 1/8 context statements relevant per judge |
| TC09 | Cross-Document | 0.52 P | 1.00 P | 0.33 F | FAIL | Monthly fee โ context passes; answer relevance fails (LLM judge stricter than keywords) |
| TC10 | Cross-Document | 0.38 F | 1.00 P | 1.00 P | FAIL | Cross-doc retrieval โ context borderline below 0.5 |
| TC11 | Unanswerable | 0.50 P | 0.00 F | 0.50 P | FAIL | Personal loan โ correct refusal; faithfulness 0.00 (refusal has no grounding claim) |
| TC12 | Unanswerable | 0.20 F | 1.00 P | 0.00 F | FAIL | Unanswerable โ context and answer relevance both fail |
| TC13 | Unanswerable | 0.33 F | 1.00 P | 1.00 P | FAIL | Student account refusal โ LLM judge correctly scores relevance; context still fails |
| TC14 | Negative / Boundary | 0.13 F | 1.00 P | 0.50 P | FAIL | Boundary case โ context very low; answer and faithfulness ok |
| TC15 | Adversarial / Confused | 0.58 P | 1.00 P | 1.00 P | PASS | Inactive vs dormant โ only case to fully pass all 3 metrics |
| * TC16โTC25: second run โ ContextualRelevancy and Faithfulness timed out on several cases (0.00 = timeout default). AnswerRelevancy scored correctly on all. | ||||||
| TC16 | Negative / Boundary | 0.00* F | 0.00* F | 1.00 P | FAIL | Context and faith timed out; answer relevance correctly scored 1.00 |
| TC17 | Negative / Boundary | 0.14* F | 0.50 P | 0.50 P | FAIL | Context partial timeout; faith and answer rel. borderline pass |
| TC18 | Negative / Boundary | 0.00* F | 1.00 P | 0.50 P | FAIL | Context timed out; faithfulness strong at 1.00 |
| TC19 | Negative / Boundary | โ Refusal โ weather question is out of scope. Excluded from scoring. | N/A | Only refusal in TC16โ25 batch | ||
| TC20 | Retrieval Confusion | 0.20* F | 0.50 P | 0.67 P | FAIL | Context timed out; faith borderline; answer rel. correctly scored |
| TC21 | Retrieval Confusion | 0.17* F | 1.00 P | 0.00 F | FAIL | Answer relevance 0.00 โ judge found answer did not address question |
| TC22 | Retrieval Confusion | 0.20* F | 1.00 P | 1.00 P | FAIL | Faith and rel. both strong; context timed out keeping it from passing |
| TC23 | Retrieval Confusion | 0.16* F | 0.50 P | 0.67 P | FAIL | Context timed out; other metrics borderline pass |
| TC24 | Adversarial / Confused | 0.20* F | 0.50 P | 1.00 P | FAIL | Answer relevance full pass; context timeout prevents overall pass |
| TC25 | Negative / Boundary | 0.23* F | 0.50 P | 1.00 P | FAIL | Same pattern โ context timeout; answer correctly scored at 1.00 |
The single most striking pattern in TC01โTC15 is how harshly DeepEval's ContextualRelevancyMetric
scored context relevance compared to Test 9.
TC01 is a clean example: "How long does it take to open a NextGen account?" The retriever pulled the correct chunks. The bot gave the correct answer. Test 9 scored this context relevance at 100%. DeepEval scored it at 0.41 โ a FAIL.
This is a calibration difference, not a system failure. DeepEval's
ContextualRelevancyMetric internally asks whether each retrieved chunk is directly relevant to the
question โ it penalises any chunk that is topically related but not specifically needed. With TOP_K=3, the third
chunk is often a supporting or nearby paragraph that adds background but is not the primary source of the answer.
DeepEval's judge penalises this. Test 9's hand-rolled judge was more lenient โ it asked "is there enough context
here to answer?" rather than "is every chunk directly relevant?"
TC11: "What is the interest rate on a NextGen personal loan?" The bot answered: "The information provided in the context does not mention the interest rate on a NextGen personal loan." This is a correct answer โ the documentation does not contain this information.
DeepEval scored faithfulness at 0.0. Why? FaithfulnessMetric decomposes the answer into claims and
checks each claim against the context. A refusal statement like "the information is not available" is not a claim
supported by a chunk โ it is an absence of information. DeepEval's faithfulness judge does not have a category for
"correct absence of claim" โ it treats uncorroborated statements as unfaithful.
TC15: "What is the difference between an inactive and a dormant account?" This is the only case that passed all three metrics.
Context relevance 0.58, faithfulness 1.00, answer relevance 1.00. The question has a clear factual answer that maps directly to the retrieved chunks. The distinction between inactive (12 months, no transactions) and dormant (24 months, no transactions) is precisely described in one chunk. No boundary reasoning is required. No multi-document synthesis. The retriever pulled the right chunk and the bot quoted it accurately.
Using only TC01โTC15 where both tests scored correctly:
| Metric | Test 9 (hand-rolled) | Test 11 DeepEval (TC01โ15) | Key difference |
|---|---|---|---|
| Context Relevance | 84% | ~27% | DeepEval penalises supporting chunks; Test 9 asked "enough context?" |
| Faithfulness | 100% | ~87% | DeepEval scores refusals as 0.0; Test 9 scored TC11 correctly |
| Answer Relevance | 66% | ~83% | DeepEval uses LLM judge โ recognises correct refusals; Test 9 used keyword matching |
| Capability | Test 9 hand-rolled | Test 11 DeepEval |
|---|---|---|
| Pass/Fail verdict | No โ score only | Yes โ per metric, per case |
| Threshold configuration | No โ you interpret the score | Yes โ set at metric instantiation |
| Judge reasoning | Partial โ one reason per case | Yes โ per metric with include_reason=True |
| Metric decomposition | No โ single judge prompt | Yes โ faithfulness decomposes into per-claim checks |
| pytest integration | No | Yes โ can run as deepeval test run in CI/CD |
| Local model support | Yes โ built in from the start | Yes โ with OllamaModel and async_mode=False |
| Calibration transparency | High โ you wrote the prompt | Low โ judge prompt is internal to DeepEval |
Test 11 surfaces a genuine trade-off that you will face in production evaluation work:
The 5-hour runtime and timeout cascade from TC16 onward is a concrete demonstration of a trade-off you will face on every evaluation project:
For learning and prototyping, Ollama is the right choice โ it allowed you to run hundreds of iterations through Tests 1โ10 without any cost. For production evaluation pipelines that need to run reliably, on schedule, and at scale, a paid API is the practical choice. The cost is lower than it appears โ 25 evaluation cases costs less than a cup of coffee.
JUDGE_MODEL to gpt-4o-mini and JUDGE_PROVIDER to openai in
config.yaml. Re-run. Compare context relevance scores against the local mistral run. Does the
calibration gap close? Does the pass rate improve?
METRIC_THRESHOLD in
config.yaml from 0.5 to 0.3. Re-run TC01โTC15. How many cases now pass? What
does this tell you about the relationship between threshold selection and reported pass rates?
MAX_CASES: 10 to the
phase4_deepeval config section and update the script to respect it. Run only the first 10 cases.
Confirm all 10 score without timeouts. Compare those 10 results against the same 10 cases from Test 9.
cd phase-4-frameworks-and-tools/deepeval-evaluation # Step 1 โ one-time setup (install + register local judge) pip install deepeval --break-system-packages deepeval set-ollama --model=mistral --base-url="http://localhost:11434" # Step 2 โ generate chunks (first time only) python chunker.py # Step 3 โ run evaluation python test11_deepeval_evaluator.py
To switch to a paid judge โ two lines in config.yaml:
phase4_deepeval: JUDGE_MODEL: "gpt-4o-mini" JUDGE_PROVIDER: "openai" # OPENAI_API_KEY: "sk-..." # or use env var