TestNeural LLM Academy
Course navigation

LLM Evaluation Engineering

18 modules · 53 topics · 238h core study · Your progress

Choose Your Learning Journey

Quick Learning

Learn what you need. Skip what you don't.

Choose a goal for a route with every required topic and checkpoint. Plans use the full topic material and share your progress with Complete Course.

Compare all learning paths and preparation options

LLM Evaluation Fundamentals

8 topics · 34h learner time

Translate an evaluation goal into checks, references and human review.

For New evaluators, QA engineers and developers adding a first evaluation

What you’ll build Build a small deterministic scorer and compare its results with human labels and overlap metrics.

Prerequisites and checkpoint

Required foundations included M02-T01, M02-T02

Final checkpoint Score a frozen set, distinguish execution errors from wrong answers and explain a false pass and a false fail.

Open learning plan

LLM-as-a-Judge Specialist

7 topics · 34h 30m learner time

Design and calibrate pointwise and pairwise judges on the same saved answers.

For Engineers evaluating open-ended responses or introducing a model judge

What you’ll build Produce a judge report that measures coverage, agreement, class errors and order sensitivity.

Prerequisites and checkpoint

Required foundations included M02-T01, M02-T02, M03-T02, M03-T03

Final checkpoint Reproduce a swapped pair, reject an unscorable verdict and justify what the judge can reliably decide.

Open learning plan

RAG Evaluation Specialist

5 topics · 28h learner time

Measure retrieval, source support and answer correctness separately.

For Engineers evaluating systems that retrieve documents before answering

What you’ll build Build a labeled retrieval and claim report with citation checks and a controlled comparison using known-good passages.

Prerequisites and checkpoint

Required foundations included M02-T01, M02-T02

Final checkpoint Verify a ranking metric by hand, then show which stage caused a held-out failure and how to detect its return.

Open learning plan

Agent Evaluation Specialist

5 topics · 27h learner time

Evaluate tool use, task completion, final state and recovery from saved traces.

For Engineers testing tool calls or multi-step workflows

What you’ll build Create a trace replay scorecard that accepts equivalent valid workflows and detects wrong state.

Prerequisites and checkpoint

Required foundations included M02-T01, M03-T02

Final checkpoint Pass two valid traces, reject a plausible wrong-state trace and explain the recovery and cost trade-off.

Open learning plan

Production Evaluation Engineer

15 topics · 78h learner time

Turn reproducible evaluation evidence into ongoing release and monitoring decisions.

For Engineers owning evaluation pipelines, release gates or production monitoring

What you’ll build Build a versioned gate and use synthetic traces to diagnose an incident and add a regression case.

Prerequisites and checkpoint

Required foundations included M02-T01, M02-T02, M02-T03, M03-T01, M03-T02, M04-T01, M04-T03, M07-T01, M07-T02

Final checkpoint Replay pass, fail and insufficient-evidence outcomes. Verify the trace diagnosis and justify a rollback trigger.

Open learning plan

Broader plans

Complete Beginner

44 topics · 238h learner time

Design, run and defend an end-to-end evaluation.

For Learners seeking a structured foundation and a production portfolio piece

What you’ll build Produce a versioned evaluator, replayable run, incident diagnosis and release recommendation.

Prerequisites and checkpoint

Required foundations included See the ordered plan for the required topics.

Final checkpoint Have a peer reproduce the scores and the incident regression, then defend the release decision with evidence.

Open learning plan

Complete Expert

53 topics · 292h 30m learner time

Complete the core and extend evaluation to the expert domains.

For Learners seeking the whole curriculum plus specialized evaluation practice

What you’ll build Produce the core capstone and specialized evidence such as a field-level scorecard, coordinated workflow audit or migration assessment.

Prerequisites and checkpoint

Required foundations included See the ordered plan for the required topics.

Final checkpoint Reproduce the core capstone and defend the evidence and limits of the expert module submissions.

Open learning plan
Learning Paths and preparation

Quick Learning sequences the required topics for a specific goal. Complete Course follows the full core in order and offers elective topics separately. Both use the same topic material and completion checkpoints. A learning plan describes the intended exercises and links to the existing material where it is available.

Preparation / Getting Started

Allow 7 learner hours if all six readiness skills are new. Skip any row once you can demonstrate its checkpoint. Practice locally with synthetic files and saved responses. No model downloads or API credentials are needed. Preparation hours are separate from all path and curriculum totals.

Preparation skills
IDReadiness skillLearning objective and practiceReadiness checkpointLearner total (hrs)
P01Files, folders and the terminalNavigate a sample project with pwd, cd and a directory listing. Recognize absolute and relative paths and save into a new output file.From the project root, locate a fixture and explain where a relative output path will be written.1h
P02Basic Python for evaluationRead strings, numbers, lists and dictionaries. Use a loop, a conditional, a small function and an assertion to count outcomes.Given three saved outcomes, return the number that pass and verify an empty input.1h 30m
P03Run Python scriptsConfirm the Python version, run a local script from the right directory, pass it a file path and read standard output and a traceback.Run a supplied script and fix a deliberately mistyped input path.1h
P04JSON and JSONLRead and write JSON objects and arrays. Read a JSONL file one record at a time with Python and distinguish syntax errors from missing fields.Convert three sample cases to JSONL and identify the invalid line in a second fixture.1h 30m
P05API requests and responsesRead a saved HTTP request and response. Identify method, endpoint, headers, body, status and timeout. Keep sample credentials out of fixtures.Explain the difference between a successful request with a wrong answer and an unsuccessful request.1h
P06Numerical reasoningCalculate rates, percentages, means and percentage-point changes on a small score sheet. Identify the denominator and how an excluded row changes it.Explain why 8 of 10 scored cases and 8 of 12 attempted cases make different claims.1h

Paths A–G

Learning path estimates
PathNameTargetsTopicsLearner total (hrs)
AComplete BeginnerCore44238h
BLLM Evaluation FundamentalsM01-T01, M02-T03, M03-T01, M03-T02, M03-T03, M05-T01834h
CLLM-as-a-Judge SpecialistM06734h 30m
DRAG Evaluation SpecialistM08528h
EAgent Evaluation SpecialistM10527h
FProduction Evaluation EngineerM13, M141578h
GComplete ExpertAll53292h 30m
  • A — Complete Beginner. Use the preparation checkpoints first, then follow the full core in curriculum order. Start at M01. The capstone verifies a reproducible release decision.
  • B — LLM Evaluation Fundamentals. Start with the objective and references, then learn deterministic and structured checks, human review and overlap metrics. Exit by scoring a small frozen set and explaining a false pass.
  • C — LLM-as-a-Judge Specialist. Complete the required reference, schema and rubric topics, then pointwise scoring, calibration and comparisons in M06. Extend with advanced judge design if needed. Embeddings and RAG can be skipped.
  • D — RAG Evaluation Specialist. Start with objectives and references, then work through M08. The ranked-list and claim exercises use supplied fixtures. Take embeddings or judge setup only for the extensions that require them.
  • E — Agent Evaluation Specialist. Learn JSON schema and evaluation objectives, then study M10. The required labs use mock tools and saved traces. Study retrieval only for the document-tool extension.
  • F — Production Evaluation Engineer. Follow the minimum route to release gates and production monitoring. The route includes versioned cases, sample design and uncertainty. RAG, agent and defensive assessment topics are optional for this path unless they apply to the release under review.
  • G — Complete Expert. Complete core and the electives. Choose the preparation rows you need and add their hours if applicable.

Path goals and checkpoints

Learning path goals
PathAudienceLearning goalPractical outcomeFinal completion checkpointSuggested extensions
ALearners seeking a structured foundation and a production portfolio pieceDesign, run and defend an end-to-end evaluation.Produce a versioned evaluator, replayable run, incident diagnosis and release recommendation.Have a peer reproduce the scores and the incident regression, then defend the release decision with evidence.—
BNew evaluators, QA engineers and developers adding a first evaluationTranslate an evaluation goal into checks, references and human review.Build a small deterministic scorer and compare its results with human labels and overlap metrics.Score a frozen set, distinguish execution errors from wrong answers and explain a false pass and a false fail.M05-T02
CEngineers evaluating open-ended responses or introducing a model judgeDesign and calibrate pointwise and pairwise judges on the same saved answers.Produce a judge report that measures coverage, agreement, class errors and order sensitivity.Reproduce a swapped pair, reject an unscorable verdict and justify what the judge can reliably decide.M18-T01
DEngineers evaluating systems that retrieve documents before answeringMeasure retrieval, source support and answer correctness separately.Build a labeled retrieval and claim report with citation checks and a controlled comparison using known-good passages.Verify a ranking metric by hand, then show which stage caused a held-out failure and how to detect its return.M05-T02, M06-T01, M09-T02
EEngineers testing tool calls or multi-step workflowsEvaluate tool use, task completion, final state and recovery from saved traces.Create a trace replay scorecard that accepts equivalent valid workflows and detects wrong state.Pass two valid traces, reject a plausible wrong-state trace and explain the recovery and cost trade-off.M08-T01, M06-T01, M17-T03
FEngineers owning evaluation pipelines, release gates or production monitoringTurn reproducible evaluation evidence into ongoing release and monitoring decisions.Build a versioned gate and use synthetic traces to diagnose an incident and add a regression case.Replay pass, fail and insufficient-evidence outcomes. Verify the trace diagnosis and justify a rollback trigger.M11-T03, M09-T03
GLearners seeking the whole curriculum plus specialized evaluation practiceComplete the core and extend evaluation to the expert domains.Produce the core capstone and specialized evidence such as a field-level scorecard, coordinated workflow audit or migration assessment.Reproduce the core capstone and defend the evidence and limits of the expert module submissions.—

Using the dependency roadmap

Open Learning Roadmap and select a module or topic by name or ID. The required route includes the target and every transitive mandatory topic, each counted once, in suggested study order. Selecting a module uses every topic in that module as a target. Recommended preparation adds context and may be skipped. A conditional link applies only to the exercise named beside it. Implementation readiness describes skills or tool setup you can demonstrate independently, so those notes and preparation hours sit outside the academic dependency graph. A dash means that category has no prerequisite.

M01-T01, M02-T01 and M03-T02 have no mandatory course prerequisites. Start with M01-T01 for a first look at model behavior, M02-T01 for evaluation design, or M03-T02 if you already know basic Python and JSON and want to validate structured outputs.

  • I want to learn RAG evaluation. Select M08 or Path D. Check the reference-answer prerequisites first. For an interpretation-only refresher, select M08-T02. Use the conditional notes before changing a dense encoder or adding a judge.
  • I already know Python and deterministic evaluation. Verify the preparation checkpoints and use Path B's fixture as a diagnostic. Mark familiar topics completed only when you can demonstrate their stated result. For structured data and provenance, enter at M04-T01 once its prerequisites are met.
  • I only want to learn LLM-as-a-Judge. Select M06-T01 for pointwise scoring, or Path C to add calibration and pairwise ranking. Use optional extensions as needed.
  • I want to become a complete evaluation engineer. Use Path A through the production capstone, then choose specialization topics or Path G. A short route establishes the advertised skills for its target. Course-wide competency requires the broader assessments and capstone.

Use the module overview to orient yourself and the topic view to inspect exact links. Arrows point to the topic or module that uses the prerequisite. The unlocked list uses your saved completion status and identifies topics that would become available by completing the current target. Use Export graph to share the full module or topic diagram, or choose a target or path and export its required route. Image downloads use the full graph for that scope regardless of pan and zoom. The standalone HTML download opens on the roadmap and supports offline interaction.

In Quick Learning, use Download Learning Plan to share an offline copy that opens on the selected path. Hosted path links use fragments such as #quick-d. For a local copy, share the downloaded HTML file. A local file address only works on the machine where that file is stored. Completion marks belong to the browser and are excluded from the shared plan.

Instructor preparation

Replay the chosen lab offline and grade a good submission and a borderline one. Verify the topic's requirements, exact fixtures, code and expected outcomes before teaching. Prepare a demonstration fallback for any live extension. Pilot the preparation checkpoints and required route timings with your class before planning further sessions.

Open the Learning Roadmap

← All Quick Learning paths

Complete Beginner

44 topics · 238h learner time

Design, run and defend an end-to-end evaluation.

Who this is for
Learners seeking a structured foundation and a production portfolio piece
Practical outcome
Produce a versioned evaluator, replayable run, incident diagnosis and release recommendation.
Required foundations included
The route includes all its required foundations.
Open path in Learning Roadmap

Learn these topics in order

Open a topic to use the same material as Complete Course. Mark it complete after you can verify the checkpoint.

  1. Complete first None

    Exercise and checkpoint Mark which evidence the model saw and predict when the reply should abstain.

  2. Complete first M01-T01

    Exercise and checkpoint Explain which difference the dataset can support and what metadata it lacks.

  3. Complete first M01-T01

    Exercise and checkpoint Classify timeout, malformed output and plausible-but-wrong response, then name which stage owns each.

  4. Complete first None

    Exercise and checkpoint Write one measurable requirement and matching counterexample for each of three dimensions.

  5. Complete first M02-T01

    Exercise and checkpoint Write a case with a source, acceptable answers, an abstention label and one known false positive.

  6. Complete first M02-T01

    Exercise and checkpoint Classify evidence sources for an offline bug, a silent production failure and a user outcome metric.

  7. Complete first M02-T02

    Exercise and checkpoint Write a truth table for punctuation, negation, an amount mismatch and an acceptable paraphrase.

  8. Complete first None

    Exercise and checkpoint Write fixtures for missing, malformed, invalid and business-correct records.

  9. Complete first M02-T02

    Exercise and checkpoint Submit a revised rubric with a low, middle and high anchor and justify one disputed case.

  10. Complete first M02-T02, M03-T02

    Exercise and checkpoint Create a revision that changes a label and generate a manifest explaining the effect on previous results.

  11. Complete first M03-T03

    Exercise and checkpoint Submit a label guide, one disagreement record and a synthetic-data acceptance rule.

  12. Complete first M04-T01

    Exercise and checkpoint Explain how to handle a new policy release without exposing or silently changing the holdout.

  13. Complete first M02-T02

    Exercise and checkpoint Rank a paraphrase and a false high-overlap answer, state the metric scale and explain the mismatch.

  14. Complete first M02-T02

    Exercise and checkpoint Evaluate two thresholds on held-out data and explain which factual errors go to deterministic checks.

  15. Complete first M03-T01, M03-T03

    Exercise and checkpoint Propose and defend an escalation rule with coverage and error counts.

  16. Complete first M03-T02, M03-T03

    Exercise and checkpoint Reject malformed, out-of-range and unsupported verdicts and record why each was unscorable.

  17. Complete first M06-T01

    Exercise and checkpoint Recalibrate on anchors and report held-out error rates, sample sizes and unresolved disagreements.

  18. Complete first M03-T03

    Exercise and checkpoint Detect an order-sensitive pair and justify whether the evidence supports a ranking.

  19. Complete first M02-T01

    Exercise and checkpoint Report attempted, valid and scorable rates separately and explain what would reproduce the result.

  20. Complete first M07-T01

    Exercise and checkpoint Vary sample size or minimum effect and explain why a result is inconclusive without overclaiming equivalence.

  21. Complete first M02-T03, M07-T02

    Exercise and checkpoint Write a rollout recommendation with effect size, uncertainty, guardrails, confounds and stopping criteria.

  22. Complete first M02-T02

    Exercise and checkpoint Add a graded relevance judgment and explain a case where MRR and Recall@k disagree.

  23. Complete first M08-T01

    Exercise and checkpoint Grade a fabricated citation, an unsupported correct answer and an unwarranted refusal separately.

  24. Complete first M08-T02

    Exercise and checkpoint Identify the root cause for a replayed failure and specify a regression test with a failure-stage assertion.

  25. Complete first M04-T01

    Exercise and checkpoint Write a tool-choice memo covering local replay, lockfile/platform, data boundary and migration cost.

  26. Complete first M08-T02, M09-T01

    Exercise and checkpoint Explain one matched-input metric disagreement, required reference, expected runtime and optional hosted cost.

  27. Complete first M09-T01

    Exercise and checkpoint Recommend a tool for a CI matrix and another for a team using managed grading, with limits and a fixture fallback.

  28. Complete first M03-T02

    Exercise and checkpoint Identify an incorrect argument and verify the corresponding result, action and trust boundary.

  29. Complete first M02-T01, M10-T01

    Exercise and checkpoint Explain why a perfect trajectory score can hide missed behavior and defend an outcome-first rubric.

  30. Complete first M10-T02

    Exercise and checkpoint Demonstrate a safe retry, then choose a threshold for recovery within budget.

  31. Complete first M03-T03

    Exercise and checkpoint Report correct refusals and false refusals with their own denominators and adjudicate a borderline case.

  32. Complete first M11-T01

    Exercise and checkpoint Explain confounds and identify what evidence and review are needed before a fairness or privacy claim.

  33. Complete first M11-T01

    Exercise and checkpoint Design a slice-sensitive safety gate and state what the benchmark cannot establish.

  34. Complete first M11-T01

    Exercise and checkpoint Specify assets, failure outcomes, negative controls, test boundaries and responsible reporting.

  35. Complete first M12-T01

    Exercise and checkpoint Calculate observed violation rate by scenario and distinguish authorization failures from correct refusals and unavailable evidence.

  36. Complete first M12-T02

    Exercise and checkpoint Write a red-team finding and verification memo with coverage, confirmed failures, false refusals, remediation and remaining risk.

  37. Complete first M03-T01, M04-T01

    Exercise and checkpoint Reconstruct a run manifest and classify invalid input, evaluator timeout and valid failure.

  38. Complete first M07-T02, M13-T01

    Exercise and checkpoint Implement a three-way decision and justify how a critical regression overrides aggregate improvement.

  39. Complete first M02-T03, M13-T02

    Exercise and checkpoint Present the owner, decision, rollback trigger, evidence link and follow-up dataset change.

  40. Complete first M02-T03

    Exercise and checkpoint Annotate a trace and show which fields to retain, redact or drop before collection.

  41. Complete first M04-T03, M07-T02, M14-T01

    Exercise and checkpoint Distinguish drift from instrumentation loss and promote reviewed cases into an offline suite.

  42. Complete first M07-T01, M14-T01

    Exercise and checkpoint Defend a cost and latency budget and diagnose an incident where quality improved but reliability worsened.

  43. Complete first M13-T02

    Exercise and checkpoint Submit a verified manifest and a suite containing good, incorrect, ambiguous and unscorable cases.

  44. Complete first M13-T03, M14-T02, M15-T01

    Exercise and checkpoint Defend the release decision and show a peer can verify the corrected outcome from the submitted artifacts.

Verify your understanding

Have a peer reproduce the scores and the incident regression, then defend the release decision with evidence.

Recommended extensions (0)

These topics are optional for this goal. Their hours are separate from the required route.

There are no additional recommended topics for this path.

Other topics you can skip for this goal (9)

← All Quick Learning paths

LLM Evaluation Fundamentals

8 topics · 34h learner time

Translate an evaluation goal into checks, references and human review.

Who this is for
New evaluators, QA engineers and developers adding a first evaluation
Practical outcome
Build a small deterministic scorer and compare its results with human labels and overlap metrics.
Required foundations included
M02-T01, M02-T02
Open path in Learning Roadmap

Learn these topics in order

Open a topic to use the same material as Complete Course. Mark it complete after you can verify the checkpoint.

  1. Complete first None

    Exercise and checkpoint Mark which evidence the model saw and predict when the reply should abstain.

  2. Complete first None

    Exercise and checkpoint Write one measurable requirement and matching counterexample for each of three dimensions.

  3. Complete first M02-T01

    Exercise and checkpoint Write a case with a source, acceptable answers, an abstention label and one known false positive.

  4. Complete first M02-T01

    Exercise and checkpoint Classify evidence sources for an offline bug, a silent production failure and a user outcome metric.

  5. Complete first M02-T02

    Exercise and checkpoint Write a truth table for punctuation, negation, an amount mismatch and an acceptable paraphrase.

  6. Complete first None

    Exercise and checkpoint Write fixtures for missing, malformed, invalid and business-correct records.

  7. Complete first M02-T02

    Exercise and checkpoint Submit a revised rubric with a low, middle and high anchor and justify one disputed case.

  8. Complete first M02-T02

    Exercise and checkpoint Rank a paraphrase and a false high-overlap answer, state the metric scale and explain the mismatch.

Verify your understanding

Score a frozen set, distinguish execution errors from wrong answers and explain a false pass and a false fail.

Recommended extensions (2)

These topics are optional for this goal. Their hours are separate from the required route.

Other topics you can skip for this goal (43)

← All Quick Learning paths

LLM-as-a-Judge Specialist

7 topics · 34h 30m learner time

Design and calibrate pointwise and pairwise judges on the same saved answers.

Who this is for
Engineers evaluating open-ended responses or introducing a model judge
Practical outcome
Produce a judge report that measures coverage, agreement, class errors and order sensitivity.
Required foundations included
M02-T01, M02-T02, M03-T02, M03-T03
Open path in Learning Roadmap

Learn these topics in order

Open a topic to use the same material as Complete Course. Mark it complete after you can verify the checkpoint.

  1. Complete first None

    Exercise and checkpoint Write one measurable requirement and matching counterexample for each of three dimensions.

  2. Complete first M02-T01

    Exercise and checkpoint Write a case with a source, acceptable answers, an abstention label and one known false positive.

  3. Complete first None

    Exercise and checkpoint Write fixtures for missing, malformed, invalid and business-correct records.

  4. Complete first M02-T02

    Exercise and checkpoint Submit a revised rubric with a low, middle and high anchor and justify one disputed case.

  5. Complete first M03-T02, M03-T03

    Exercise and checkpoint Reject malformed, out-of-range and unsupported verdicts and record why each was unscorable.

  6. Complete first M06-T01

    Exercise and checkpoint Recalibrate on anchors and report held-out error rates, sample sizes and unresolved disagreements.

  7. Complete first M03-T03

    Exercise and checkpoint Detect an order-sensitive pair and justify whether the evidence supports a ranking.

Verify your understanding

Reproduce a swapped pair, reject an unscorable verdict and justify what the judge can reliably decide.

Recommended extensions (7)

These topics are optional for this goal. Their hours are separate from the required route.

Other topics you can skip for this goal (39)

← All Quick Learning paths

RAG Evaluation Specialist

5 topics · 28h learner time

Measure retrieval, source support and answer correctness separately.

Who this is for
Engineers evaluating systems that retrieve documents before answering
Practical outcome
Build a labeled retrieval and claim report with citation checks and a controlled comparison using known-good passages.
Required foundations included
M02-T01, M02-T02
Open path in Learning Roadmap

Learn these topics in order

Open a topic to use the same material as Complete Course. Mark it complete after you can verify the checkpoint.

  1. Complete first None

    Exercise and checkpoint Write one measurable requirement and matching counterexample for each of three dimensions.

  2. Complete first M02-T01

    Exercise and checkpoint Write a case with a source, acceptable answers, an abstention label and one known false positive.

  3. Complete first M02-T02

    Exercise and checkpoint Add a graded relevance judgment and explain a case where MRR and Recall@k disagree.

  4. Complete first M08-T01

    Exercise and checkpoint Grade a fabricated citation, an unsupported correct answer and an unwarranted refusal separately.

  5. Complete first M08-T02

    Exercise and checkpoint Identify the root cause for a replayed failure and specify a regression test with a failure-stage assertion.

Verify your understanding

Verify a ranking metric by hand, then show which stage caused a held-out failure and how to detect its return.

Recommended extensions (9)

These topics are optional for this goal. Their hours are separate from the required route.

Other topics you can skip for this goal (39)

← All Quick Learning paths

Agent Evaluation Specialist

5 topics · 27h learner time

Evaluate tool use, task completion, final state and recovery from saved traces.

Who this is for
Engineers testing tool calls or multi-step workflows
Practical outcome
Create a trace replay scorecard that accepts equivalent valid workflows and detects wrong state.
Required foundations included
M02-T01, M03-T02
Open path in Learning Roadmap

Learn these topics in order

Open a topic to use the same material as Complete Course. Mark it complete after you can verify the checkpoint.

  1. Complete first None

    Exercise and checkpoint Write one measurable requirement and matching counterexample for each of three dimensions.

  2. Complete first None

    Exercise and checkpoint Write fixtures for missing, malformed, invalid and business-correct records.

  3. Complete first M03-T02

    Exercise and checkpoint Identify an incorrect argument and verify the corresponding result, action and trust boundary.

  4. Complete first M02-T01, M10-T01

    Exercise and checkpoint Explain why a perfect trajectory score can hide missed behavior and defend an outcome-first rubric.

  5. Complete first M10-T02

    Exercise and checkpoint Demonstrate a safe retry, then choose a threshold for recovery within budget.

Verify your understanding

Pass two valid traces, reject a plausible wrong-state trace and explain the recovery and cost trade-off.

Recommended extensions (8)

These topics are optional for this goal. Their hours are separate from the required route.

Other topics you can skip for this goal (40)

← All Quick Learning paths

Production Evaluation Engineer

15 topics · 78h learner time

Turn reproducible evaluation evidence into ongoing release and monitoring decisions.

Who this is for
Engineers owning evaluation pipelines, release gates or production monitoring
Practical outcome
Build a versioned gate and use synthetic traces to diagnose an incident and add a regression case.
Required foundations included
M02-T01, M02-T02, M02-T03, M03-T01, M03-T02, M04-T01, M04-T03, M07-T01, M07-T02
Open path in Learning Roadmap

Learn these topics in order

Open a topic to use the same material as Complete Course. Mark it complete after you can verify the checkpoint.

  1. Complete first None

    Exercise and checkpoint Write one measurable requirement and matching counterexample for each of three dimensions.

  2. Complete first M02-T01

    Exercise and checkpoint Write a case with a source, acceptable answers, an abstention label and one known false positive.

  3. Complete first M02-T01

    Exercise and checkpoint Classify evidence sources for an offline bug, a silent production failure and a user outcome metric.

  4. Complete first M02-T02

    Exercise and checkpoint Write a truth table for punctuation, negation, an amount mismatch and an acceptable paraphrase.

  5. Complete first None

    Exercise and checkpoint Write fixtures for missing, malformed, invalid and business-correct records.

  6. Complete first M02-T02, M03-T02

    Exercise and checkpoint Create a revision that changes a label and generate a manifest explaining the effect on previous results.

  7. Complete first M04-T01

    Exercise and checkpoint Explain how to handle a new policy release without exposing or silently changing the holdout.

  8. Complete first M02-T01

    Exercise and checkpoint Report attempted, valid and scorable rates separately and explain what would reproduce the result.

  9. Complete first M07-T01

    Exercise and checkpoint Vary sample size or minimum effect and explain why a result is inconclusive without overclaiming equivalence.

  10. Complete first M03-T01, M04-T01

    Exercise and checkpoint Reconstruct a run manifest and classify invalid input, evaluator timeout and valid failure.

  11. Complete first M07-T02, M13-T01

    Exercise and checkpoint Implement a three-way decision and justify how a critical regression overrides aggregate improvement.

  12. Complete first M02-T03, M13-T02

    Exercise and checkpoint Present the owner, decision, rollback trigger, evidence link and follow-up dataset change.

  13. Complete first M02-T03

    Exercise and checkpoint Annotate a trace and show which fields to retain, redact or drop before collection.

  14. Complete first M04-T03, M07-T02, M14-T01

    Exercise and checkpoint Distinguish drift from instrumentation loss and promote reviewed cases into an offline suite.

  15. Complete first M07-T01, M14-T01

    Exercise and checkpoint Defend a cost and latency budget and diagnose an incident where quality improved but reliability worsened.

Verify your understanding

Replay pass, fail and insufficient-evidence outcomes. Verify the trace diagnosis and justify a rollback trigger.

Recommended extensions (12)

These topics are optional for this goal. Their hours are separate from the required route.

Other topics you can skip for this goal (26)

← All Quick Learning paths

Complete Expert

53 topics · 292h 30m learner time

Complete the core and extend evaluation to the expert domains.

Who this is for
Learners seeking the whole curriculum plus specialized evaluation practice
Practical outcome
Produce the core capstone and specialized evidence such as a field-level scorecard, coordinated workflow audit or migration assessment.
Required foundations included
The route includes all its required foundations.
Open path in Learning Roadmap

Learn these topics in order

Open a topic to use the same material as Complete Course. Mark it complete after you can verify the checkpoint.

  1. Complete first None

    Exercise and checkpoint Mark which evidence the model saw and predict when the reply should abstain.

  2. Complete first M01-T01

    Exercise and checkpoint Explain which difference the dataset can support and what metadata it lacks.

  3. Complete first M01-T01

    Exercise and checkpoint Classify timeout, malformed output and plausible-but-wrong response, then name which stage owns each.

  4. Complete first None

    Exercise and checkpoint Write one measurable requirement and matching counterexample for each of three dimensions.

  5. Complete first M02-T01

    Exercise and checkpoint Write a case with a source, acceptable answers, an abstention label and one known false positive.

  6. Complete first M02-T01

    Exercise and checkpoint Classify evidence sources for an offline bug, a silent production failure and a user outcome metric.

  7. Complete first M02-T02

    Exercise and checkpoint Write a truth table for punctuation, negation, an amount mismatch and an acceptable paraphrase.

  8. Complete first None

    Exercise and checkpoint Write fixtures for missing, malformed, invalid and business-correct records.

  9. Complete first M02-T02

    Exercise and checkpoint Submit a revised rubric with a low, middle and high anchor and justify one disputed case.

  10. Complete first M02-T02, M03-T02

    Exercise and checkpoint Create a revision that changes a label and generate a manifest explaining the effect on previous results.

  11. Complete first M03-T03

    Exercise and checkpoint Submit a label guide, one disagreement record and a synthetic-data acceptance rule.

  12. Complete first M04-T01

    Exercise and checkpoint Explain how to handle a new policy release without exposing or silently changing the holdout.

  13. Complete first M02-T02

    Exercise and checkpoint Rank a paraphrase and a false high-overlap answer, state the metric scale and explain the mismatch.

  14. Complete first M02-T02

    Exercise and checkpoint Evaluate two thresholds on held-out data and explain which factual errors go to deterministic checks.

  15. Complete first M03-T01, M03-T03

    Exercise and checkpoint Propose and defend an escalation rule with coverage and error counts.

  16. Complete first M03-T02, M03-T03

    Exercise and checkpoint Reject malformed, out-of-range and unsupported verdicts and record why each was unscorable.

  17. Complete first M06-T01

    Exercise and checkpoint Recalibrate on anchors and report held-out error rates, sample sizes and unresolved disagreements.

  18. Complete first M03-T03

    Exercise and checkpoint Detect an order-sensitive pair and justify whether the evidence supports a ranking.

  19. Complete first M02-T01

    Exercise and checkpoint Report attempted, valid and scorable rates separately and explain what would reproduce the result.

  20. Complete first M07-T01

    Exercise and checkpoint Vary sample size or minimum effect and explain why a result is inconclusive without overclaiming equivalence.

  21. Complete first M02-T03, M07-T02

    Exercise and checkpoint Write a rollout recommendation with effect size, uncertainty, guardrails, confounds and stopping criteria.

  22. Complete first M02-T02

    Exercise and checkpoint Add a graded relevance judgment and explain a case where MRR and Recall@k disagree.

  23. Complete first M08-T01

    Exercise and checkpoint Grade a fabricated citation, an unsupported correct answer and an unwarranted refusal separately.

  24. Complete first M08-T02

    Exercise and checkpoint Identify the root cause for a replayed failure and specify a regression test with a failure-stage assertion.

  25. Complete first M04-T01

    Exercise and checkpoint Write a tool-choice memo covering local replay, lockfile/platform, data boundary and migration cost.

  26. Complete first M08-T02, M09-T01

    Exercise and checkpoint Explain one matched-input metric disagreement, required reference, expected runtime and optional hosted cost.

  27. Complete first M09-T01

    Exercise and checkpoint Recommend a tool for a CI matrix and another for a team using managed grading, with limits and a fixture fallback.

  28. Complete first M03-T02

    Exercise and checkpoint Identify an incorrect argument and verify the corresponding result, action and trust boundary.

  29. Complete first M02-T01, M10-T01

    Exercise and checkpoint Explain why a perfect trajectory score can hide missed behavior and defend an outcome-first rubric.

  30. Complete first M10-T02

    Exercise and checkpoint Demonstrate a safe retry, then choose a threshold for recovery within budget.

  31. Complete first M03-T03

    Exercise and checkpoint Report correct refusals and false refusals with their own denominators and adjudicate a borderline case.

  32. Complete first M11-T01

    Exercise and checkpoint Explain confounds and identify what evidence and review are needed before a fairness or privacy claim.

  33. Complete first M11-T01

    Exercise and checkpoint Design a slice-sensitive safety gate and state what the benchmark cannot establish.

  34. Complete first M11-T01

    Exercise and checkpoint Specify assets, failure outcomes, negative controls, test boundaries and responsible reporting.

  35. Complete first M12-T01

    Exercise and checkpoint Calculate observed violation rate by scenario and distinguish authorization failures from correct refusals and unavailable evidence.

  36. Complete first M12-T02

    Exercise and checkpoint Write a red-team finding and verification memo with coverage, confirmed failures, false refusals, remediation and remaining risk.

  37. Complete first M03-T01, M04-T01

    Exercise and checkpoint Reconstruct a run manifest and classify invalid input, evaluator timeout and valid failure.

  38. Complete first M07-T02, M13-T01

    Exercise and checkpoint Implement a three-way decision and justify how a critical regression overrides aggregate improvement.

  39. Complete first M02-T03, M13-T02

    Exercise and checkpoint Present the owner, decision, rollback trigger, evidence link and follow-up dataset change.

  40. Complete first M02-T03

    Exercise and checkpoint Annotate a trace and show which fields to retain, redact or drop before collection.

  41. Complete first M04-T03, M07-T02, M14-T01

    Exercise and checkpoint Distinguish drift from instrumentation loss and promote reviewed cases into an offline suite.

  42. Complete first M07-T01, M14-T01

    Exercise and checkpoint Defend a cost and latency budget and diagnose an incident where quality improved but reliability worsened.

  43. Complete first M13-T02

    Exercise and checkpoint Submit a verified manifest and a suite containing good, incorrect, ambiguous and unscorable cases.

  44. Complete first M13-T03, M14-T02, M15-T01

    Exercise and checkpoint Defend the release decision and show a peer can verify the corrected outcome from the submitted artifacts.

  45. Complete first M04-T03, M07-T02

    Exercise and checkpoint Produce a per-locale scorecard and identify translation leakage and subgroup uncertainty.

  46. Complete first M02-T02, M03-T03

    Exercise and checkpoint Identify unsupported claims and report errors by modality and extraction stage.

  47. Complete first M01-T01, M07-T01

    Exercise and checkpoint Report controlled position and context-length effects across slices.

  48. Complete first M03-T02, M05-T01

    Exercise and checkpoint Explain how micro F1 hides a critical rare-field miss and add a matching gate.

  49. Complete first M03-T02, M07-T01

    Exercise and checkpoint Distinguish a compilation failure, wrong answer and timeout and interpret pass@k assumptions.

  50. Complete first M10-T02

    Exercise and checkpoint Verify global invariants, handoff quality and team cost from saved events.

  51. Complete first M06-T02

    Exercise and checkpoint Justify a human escalation boundary and explain one shared judge blind spot.

  52. Complete first M04-T03, M07-T03

    Exercise and checkpoint Identify confounds and defend a migration plan for rare and severe slices.

  53. Complete first M04-T03, M07-T02

    Exercise and checkpoint Write a method card with construct, assumptions, evidence limits and a replication plan.

Verify your understanding

Reproduce the core capstone and defend the evidence and limits of the expert module submissions.

Recommended extensions (0)

These topics are optional for this goal. Their hours are separate from the required route.

There are no additional recommended topics for this path.

Other topics you can skip for this goal (0)

This route covers every other topic or lists it above as an extension.

Complete Course

From beginner to advanced evaluation engineering.

Open the full dependency roadmap

Beginner3 modules · 36h 30m
M01 How LLM applications behave 3 topics · 11h

Beginner · Core · Foundations

Description
Follow a request from user input to model output. Learn which settings and components can change the evidence an evaluator sees.
Objectives
Trace a generation request, explain output variability, and locate failure boundaries in a model-backed application.
Labs
Replay a fictional bank help request as plain text, then read its saved request and result envelopes.
Competencies
Label the request, model configuration, response and failure stage in a fresh replay.
Completion
Submit a labeled request/response trace. Mark message roles, missing context and one failure boundary. Compare your labels to the supplied replay key. A submission fails if a transport error is labeled a wrong model answer.
Theory (hrs)
3h
Guided (hrs)
4h
Experiment (hrs)
2h 30m
Exercises (hrs)
1h 30m
Learner total (hrs)
11h
Instructor-led (hrs)
7h 30m
M01-T01 Tokens, messages and context windows Needs Improvement · 3h 30m

Beginner · Core

Link to topic
Why it matters
Missing evidence or truncation can look like a reasoning failure.
Concepts
Tokens versus words, system and user roles, context budget, truncation, input/output limits and message ordering.
Mandatory prerequisites
—
Recommended prerequisites
—
Conditional prerequisites
—
Implementation readiness
Read a short text fixture. The optional Python replay uses P01 and P03.
Demonstration
Compare two saved prompts that differ only in whether the fee schedule fits in the supplied context.
Exercise and assessment
Mark which evidence the model saw and predict when the reply should abstain.
Theory (hrs)
1h
Guided (hrs)
1h
Experiment (hrs)
1h
Exercises (hrs)
0h 30m
Learner total (hrs)
3h 30m
Instructor-led (hrs)
2h 30m

Repository mapping

Existing files
docs/test1.html, model_client.py
Audit findings
S08
Reuse strategy
Reuse the single request and shared adapter as diagrams.
Teaching and experiment gaps
Add a token-budget worksheet and trace with a truncated prompt.
Technical corrections
Keep model selection and effective request settings visible.
M01-T02 Inference, sampling and repeatability Needs Correction · 4h

Beginner · Core

Link to topic
Why it matters
A fixed prompt can yield different responses, and generation errors need their own outcome.
Concepts
Next-token sampling, temperature, top-p, seed limits, maximum output tokens, stop conditions and model snapshots.
Mandatory prerequisites
M01-T01
Recommended prerequisites
—
Conditional prerequisites
—
Implementation readiness
Read a table of saved completions. Live comparisons use P03 and P05 plus a prepared local model.
Demonstration
Inspect repeated saved completions and compare a changed setting with repeated runs under one setting.
Exercise and assessment
Explain which difference the dataset can support and what metadata it lacks.
Theory (hrs)
1h
Guided (hrs)
1h 30m
Experiment (hrs)
1h
Exercises (hrs)
0h 30m
Learner total (hrs)
4h
Instructor-led (hrs)
2h 30m

Repository mapping

Existing files
docs/test8.html, phase-3-what-you-are-evaluating/temperature-sensitivity/temperature_evaluator.py
Audit findings
C08,V02
Reuse strategy
Keep the observed temperature comparison as an experiment-design critique.
Teaching and experiment gaps
Add replayable repeated draws and a parameter glossary.
Technical corrections
Remove universal conclusions based on a single fixed-order run.
M01-T03 Application architecture and a first smoke test Reusable · 3h 30m

Beginner · Core

Link to topic
Why it matters
A working request establishes connectivity and enables stage-specific diagnosis.
Concepts
Client, prompt assembly, inference server, retrieval, tools, post-processing, transport errors and end-to-end elapsed time.
Mandatory prerequisites
M01-T01
Recommended prerequisites
M01-T02
Conditional prerequisites
—
Implementation readiness
Use P01, P03 and P04 for Python replay. The optional local request also uses P05.
Demonstration
Follow a frozen request through a tiny Python fixture reader. Optionally compare with a single local model call after setup.
Exercise and assessment
Classify timeout, malformed output and plausible-but-wrong response, then name which stage owns each.
Theory (hrs)
1h
Guided (hrs)
1h 30m
Experiment (hrs)
0h 30m
Exercises (hrs)
0h 30m
Learner total (hrs)
3h 30m
Instructor-led (hrs)
2h 30m

Repository mapping

Existing files
docs/test1.html, phase-1-first-experiments/test1_basic_test.py, config.yaml
Audit findings
C11,S06,S08,V02
Reuse strategy
Retain the smoke-test idea and worked terminal output.
Teaching and experiment gaps
Add setup preflight and a one-model replay option.
Technical corrections
Qualify timing and creativity observations and verify direct requirements before teaching live setup.
M02 Design the evaluation 3 topics · 12h

Beginner · Core · Foundations

Description
Translate a user goal and potential harm into an evaluation contract with cases, acceptance criteria and decisions.
Objectives
Select an evaluation unit, define outcomes, and explain when to use offline and online evidence.
Labs
Draft a one-page evaluation contract for fictional fee and account-support questions.
Competencies
Given an ambiguous feature, specify intended behavior, cases, labels, failure categories and a decision owner.
Completion
Submit an evaluation contract with a unit, two quality dimensions, expected behavior and a decision owner. Match one reference and one unanswerable case to the contract in a peer walkthrough. A score with no decision or acceptance criterion fails.
Theory (hrs)
3h
Guided (hrs)
3h 30m
Experiment (hrs)
3h
Exercises (hrs)
2h 30m
Learner total (hrs)
12h
Instructor-led (hrs)
8h
M02-T01 Objectives, dimensions and failure taxonomy Needs Improvement · 4h 30m

Beginner · Core

Link to topic
Why it matters
A score helps only if it answers a decision the team can act on.
Concepts
Correctness, completeness, relevance, groundedness, usability, safety, cost, latency, evaluation units and error versus failure.
Mandatory prerequisites
—
Recommended prerequisites
M01-T03
Conditional prerequisites
—
Implementation readiness
Read and label saved outputs. This discussion and worksheet need no Python.
Demonstration
Classify saved fee answers that are incorrect, incomplete, unsupported or unavailable due to infrastructure.
Exercise and assessment
Write one measurable requirement and matching counterexample for each of three dimensions.
Theory (hrs)
1h
Guided (hrs)
1h 30m
Experiment (hrs)
1h
Exercises (hrs)
1h
Learner total (hrs)
4h 30m
Instructor-led (hrs)
3h

Repository mapping

Audit findings
C05,S02
Reuse strategy
Start with the audit's false-pass examples and glossary.
Teaching and experiment gaps
Add objective-to-check worksheet and worked failure tree.
Technical corrections
Make failure, evaluator error and N/A explicit.
M02-T02 Ground truth, references and benchmarks Reusable · 4h

Beginner · Core

Link to topic
Why it matters
Valid answers can differ in wording, and some cases require abstention or multiple acceptable labels.
Concepts
Gold evidence, reference answer sets, expected behavior, benchmark validity, dataset rows and uncertain or unanswerable examples.
Mandatory prerequisites
M02-T01
Recommended prerequisites
M01-T01
Conditional prerequisites
—
Implementation readiness
Read policy records and benchmark rows. P04 helps when inspecting JSON directly.
Demonstration
Inspect a saved extractive QA example and a synthetic banking record with two acceptable answers.
Exercise and assessment
Write a case with a source, acceptable answers, an abstention label and one known false positive.
Theory (hrs)
1h
Guided (hrs)
1h
Experiment (hrs)
1h
Exercises (hrs)
1h
Learner total (hrs)
4h
Instructor-led (hrs)
2h 30m

Repository mapping

Existing files
docs/test0.html, phase-0-foundations/explore_squad.py, docs/concepts.html
Audit findings
S06,V03
Reuse strategy
Retain the SQuAD schema and multiple gold answers walkthrough.
Teaching and experiment gaps
Supply a reviewed local fixture and contrast extractive versus generative tasks.
Technical corrections
Describe the download dependency or use the offline fixture.
M02-T03 Offline, online and the evaluation lifecycle Needs Improvement · 3h 30m

Beginner · Core

Link to topic
Why it matters
Offline quality estimates and live user outcomes answer different questions.
Concepts
Development replay, shadow evaluation, canary, online experiments, regression suites, incident feedback and release evidence.
Mandatory prerequisites
M02-T01
Recommended prerequisites
M01-T03
Conditional prerequisites
—
Implementation readiness
Use the supplied lifecycle worksheet and offline scenario.
Demonstration
Follow a fictional fee-policy failure from labeled case to offline gate, monitored release and incident review.
Exercise and assessment
Classify evidence sources for an offline bug, a silent production failure and a user outcome metric.
Theory (hrs)
1h
Guided (hrs)
1h
Experiment (hrs)
1h
Exercises (hrs)
0h 30m
Learner total (hrs)
3h 30m
Instructor-led (hrs)
2h 30m

Repository mapping

Existing files
docs/test14.html, README.md
Audit findings
V04
Reuse strategy
Reuse the offline-to-monitoring distinction.
Teaching and experiment gaps
Add a lifecycle diagram, evidence worksheet and decision scenarios.
Technical corrections
Describe offline traces and causal online experiments as separate evidence.
M03 Deterministic checks and first human review 3 topics · 13h 30m

Beginner · Core · Techniques

Description
Build the first evaluator from frozen outputs and learn where human judgment adds evidence.
Objectives
Implement stable assertions, validate structured data and explain a disagreement using a rubric.
Labs
Score one fixture containing a correct answer, good paraphrase, incorrect close answer, correct abstention, over-refusal and execution error.
Competencies
Submit a runnable fixture scorer and a reviewed answer sheet that identifies false passes and false fails.
Completion
Submit a Python scorer, frozen known-answer cases and a blind rating sheet. Reproduce the expected pass, fail and unscorable outcomes and explain a substring false pass. Run the answer key and inspect one rubric disagreement. A parser or transport error counted as a model failure fails.
Theory (hrs)
3h
Guided (hrs)
4h 30m
Experiment (hrs)
3h
Exercises (hrs)
3h
Learner total (hrs)
13h 30m
Instructor-led (hrs)
9h
M03-T01 Exact match, normalization and rules Needs Correction · 4h 30m

Beginner · Core

Link to topic
Why it matters
String matches are fast and inspectable but can miss valid aliases and accept wrong claims.
Concepts
Exact match, case and whitespace normalization, Unicode caution, regex boundaries, canonical aliases, numeric checks, precision and recall.
Mandatory prerequisites
M02-T02
Recommended prerequisites
—
Conditional prerequisites
—
Implementation readiness
Use P01–P04 for normalization and assertions in Python. The first truth table can be done on paper.
Demonstration
Implement normalized matches and a typed fee check in Python. Compare with a substring baseline.
Exercise and assessment
Write a truth table for punctuation, negation, an amount mismatch and an acceptable paraphrase.
Theory (hrs)
1h
Guided (hrs)
1h 30m
Experiment (hrs)
1h
Exercises (hrs)
1h
Learner total (hrs)
4h 30m
Instructor-led (hrs)
3h

Repository mapping

Existing files
docs/test0.html, phase-0-foundations/phase_eval.py, docs/test4.html, phase-2-evaluation-techniques/banking-bots/banking_evaluator.py
Audit findings
C05,S02,S09
Reuse strategy
Use the documented false pass and workbook as starter data.
Teaching and experiment gaps
Add fixed expected outputs and error classification.
Technical corrections
Separate 15 automated from 49 manual cases and replace misleading refusal detection in the verified lab.
M03-T02 Structured output and schema validation Needs Improvement · 4h 30m

Beginner · Core

Link to topic
Why it matters
Valid JSON can still contain invalid field values or unsafe business outcomes.
Concepts
Parsing, types, required and nullable fields, enums, ranges, schema versions, cross-field invariants and invalid-output handling.
Mandatory prerequisites
—
Recommended prerequisites
M03-T01
Conditional prerequisites
—
Implementation readiness
Use P01–P04 to parse and check JSON. The lab introduces schema concepts with self-contained examples.
Demonstration
Read mock fee replies using Python and apply explicit field and domain checks after parsing.
Exercise and assessment
Write fixtures for missing, malformed, invalid and business-correct records.
Theory (hrs)
1h
Guided (hrs)
1h 30m
Experiment (hrs)
1h
Exercises (hrs)
1h
Learner total (hrs)
4h 30m
Instructor-led (hrs)
3h

Repository mapping

Existing files
phase-5-production-eval/agent-evaluation/agent.py, docs/test12.html, phase-2-evaluation-techniques/groundedness/groundedness_evaluator.py
Audit findings
S04,S05
Reuse strategy
Reuse action and judge output schemas as later examples.
Teaching and experiment gaps
Add a beginner classifier or extraction schema and trusted outcomes.
Technical corrections
Check top-level shape and field constraints before reading values.
M03-T03 Rubrics and a first human evaluation Needs Correction · 4h 30m

Beginner · Core

Link to topic
Why it matters
Human scores vary when criteria, inputs and evidence are underspecified.
Concepts
Anchored ordinal scales, binary decisions, blind review, N/A, evidence notes and response IDs.
Mandatory prerequisites
M02-T02
Recommended prerequisites
M03-T01
Conditional prerequisites
—
Implementation readiness
Inspect full saved replies and fill a worksheet. Python is optional for summarizing labels.
Demonstration
Blind-score several saved full replies using one criterion at a time and discuss disagreements.
Exercise and assessment
Submit a revised rubric with a low, middle and high anchor and justify one disputed case.
Theory (hrs)
1h
Guided (hrs)
1h 30m
Experiment (hrs)
1h
Exercises (hrs)
1h
Learner total (hrs)
4h 30m
Instructor-led (hrs)
3h

Repository mapping

Existing files
docs/test2.html, phase-1-first-experiments/test2_score_evaluations.py, phase-1-first-experiments/results/scores.json
Audit findings
S01,C01
Reuse strategy
Retain the capture/score split and human-first review.
Teaching and experiment gaps
Supply canonical full responses, annotations and scoring key.
Technical corrections
Blind model labels, show full output and save each decision with its source response ID.
Intermediate4 modules · 66h 30m
M04 Evaluation dataset engineering 3 topics · 16h

Intermediate · Core · Dataset Engineering

Description
Build data with documented coverage, adjudicated labels, and a defensible separation between development and final evaluation.
Objectives
Design a golden dataset schema, curate inputs and labels, and manage sampling and updates.
Labs
Publish a versioned miniature banking dataset with a manifest, slices and isolated holdout.
Competencies
Another learner can validate the schema, reproduce the split and trace each label to evidence.
Completion
Submit a validated JSONL dataset with a manifest, split and adjudication note. Reproduce the split and trace a label to evidence. Check unique IDs and group isolation. Shared source groups across the holdout or missing provenance fail.
Theory (hrs)
4h
Guided (hrs)
5h
Experiment (hrs)
4h
Exercises (hrs)
3h
Learner total (hrs)
16h
Instructor-led (hrs)
10h 30m
M04-T01 Golden schemas, provenance and versioning Needs Improvement · 4h 30m

Intermediate · Core

Link to topic
Why it matters
A score cannot be reproduced if cases, labels and outputs drift independently.
Concepts
Stable case and response IDs, data contracts, task metadata, source references, append-only runs, version manifests and validation.
Mandatory prerequisites
M02-T02, M03-T02
Recommended prerequisites
M03-T01
Conditional prerequisites
—
Implementation readiness
Use P01–P04. The workbook slice is supplied in a format readable without spreadsheet software.
Demonstration
Convert a workbook slice to JSONL and validate field types, unique IDs and references.
Exercise and assessment
Create a revision that changes a label and generate a manifest explaining the effect on previous results.
Theory (hrs)
1h
Guided (hrs)
1h 30m
Experiment (hrs)
1h
Exercises (hrs)
1h
Learner total (hrs)
4h 30m
Instructor-led (hrs)
3h

Repository mapping

Existing files
phase-2-evaluation-techniques/banking-bots/banking_test_cases.xlsx, phase-3-what-you-are-evaluating/rag-evaluation/rag_test_cases.xlsx
Audit findings
S09,S11,S15,S16,V03
Reuse strategy
Borrow realistic categories and documented fields.
Teaching and experiment gaps
Add schema, manifest, converter and duplicate or missing-row fixtures.
Technical corrections
Separate canonical input from run artifacts and catch invalid ranges.
M04-T02 Annotation, adjudication and synthetic data Needs Improvement · 6h

Intermediate · Core

Link to topic
Why it matters
Synthetic breadth is useful when human review and source-backed validation preserve label quality.
Concepts
Annotation instructions, independent labels, disagreement and arbitration, structured generation from policy, review and rejection, provenance and license.
Mandatory prerequisites
M03-T03
Recommended prerequisites
M04-T01
Conditional prerequisites
M04-T01 — required when publishing reviewed rows to the versioned dataset
Implementation readiness
A spreadsheet or text worksheet suffices for manual labels. Live synthetic generation is an optional P05 extension.
Demonstration
Review synthetic policy examples with two readers, adjudicate disagreements and quarantine ambiguous rows.
Exercise and assessment
Submit a label guide, one disagreement record and a synthetic-data acceptance rule.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Existing files
docs/test4.html, phase-5-production-eval/langfuse-observability/create_test_cases.py, phase-2-evaluation-techniques/llm-as-a-judge/banking_bot_test_results.xlsx
Audit findings
C01,V03
Reuse strategy
Reuse the interactive case-writing prompt and fixed workbook generator as simple precursors.
Teaching and experiment gaps
Add double-label and synthetic validation exercises with defective rows.
Technical corrections
Freeze exact outputs before calling verdicts comparable.
M04-T03 Sampling, holdouts, leakage and maintenance New · 5h 30m

Intermediate · Core

Link to topic
Why it matters
Aggregate gains may hide weak slices or reflect tuning on the evaluation set.
Concepts
Representative and stratified sampling, rare-case oversampling, adversarial edge cases, grouped splits, deduplication, train/test contamination, dataset refresh and retirement.
Mandatory prerequisites
M04-T01
Recommended prerequisites
M04-T02
Conditional prerequisites
—
Implementation readiness
Use P02, P04 and P06 to group cases and compute coverage.
Demonstration
Split cases by document family, detect near-duplicates, and compare representative coverage with policy-weighted stress coverage.
Exercise and assessment
Explain how to handle a new policy release without exposing or silently changing the holdout.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
5h 30m
Instructor-led (hrs)
3h 30m

Repository mapping

Existing files
—
Audit findings
V03
Reuse strategy
Build a new split and maintenance lab using the fictional policies as domain context.
Teaching and experiment gaps
Add distribution cards, near-duplicate examples and coverage thresholds.
Technical corrections
—
M05 Reference metrics, embeddings and hybrid scoring 3 topics · 15h 30m

Intermediate · Core · Techniques

Description
Use several fast scorers on the same answers and investigate divergent results before setting a threshold.
Objectives
Compute overlap and semantic similarity, compare error patterns, and design a hybrid decision rule.
Labs
Build a Python scorecard for factual banking answers, paraphrases and unsupported claims.
Competencies
Explain a false positive and false negative for each proxy and defend a held-out decision rule.
Completion
Submit an overlap, embedding and hybrid scorecard with a held-out decision rule. Recompute a hand-scored case and demonstrate a valid paraphrase and a high-overlap wrong answer. An unqualified claim that cosine proves correctness fails.
Theory (hrs)
4h
Guided (hrs)
4h 30m
Experiment (hrs)
4h
Exercises (hrs)
3h
Learner total (hrs)
15h 30m
Instructor-led (hrs)
10h 30m
M05-T01 Token F1, BLEU, ROUGE and reference metrics Needs Correction · 5h

Intermediate · Core

Link to topic
Why it matters
Surface overlap offers a useful baseline for bounded tasks and can penalize correct paraphrases.
Concepts
Token precision/recall/F1, n-grams, BLEU brevity and smoothing, sentence versus corpus BLEU, ROUGE-N and ROUGE-L, multiple references and optional chrF.
Mandatory prerequisites
M02-T02
Recommended prerequisites
M03-T01
Conditional prerequisites
—
Implementation readiness
Use P02 and P06 for counts and fractions. The library extension needs the prepared metric environment.
Demonstration
Count overlap by hand and in Python, then compare with sacreBLEU and rouge-score on frozen summaries.
Exercise and assessment
Rank a paraphrase and a false high-overlap answer, state the metric scale and explain the mismatch.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h
Exercises (hrs)
1h
Learner total (hrs)
5h
Instructor-led (hrs)
3h 30m

Repository mapping

Existing files
docs/test3.html, phase-2-evaluation-techniques/test3_bleu_score_evaluation.py
Audit findings
C10,S06,S08
Reuse strategy
Retain the one-passage example as a simple entry point.
Teaching and experiment gaps
Add token F1, two references, per-case answer key and offline inputs.
Technical corrections
Correct claims equating overlap with quality or removing human bias.
M05-T02 Embeddings and semantic similarity Needs Correction · 5h 30m

Intermediate · Core

Link to topic
Why it matters
Cosine can find paraphrases while giving high scores to changed amounts or opposite meanings.
Concepts
Vectors, cosine, embeddings from pretrained encoders, model/version effects, threshold calibration and triage.
Mandatory prerequisites
M02-T02
Recommended prerequisites
M05-T01, M04-T03
Conditional prerequisites
M04-T03 — required when building your own held-out split for threshold tuning
Implementation readiness
Use P02 and P06 for cosine on supplied vectors. Use cached weights only for the optional live encoder.
Demonstration
Compute cosine on toy vectors, then compare cached embedding results for near-miss answers.
Exercise and assessment
Evaluate two thresholds on held-out data and explain which factual errors go to deterministic checks.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
5h 30m
Instructor-led (hrs)
3h 30m

Repository mapping

Existing files
docs/test6.html, phase-2-evaluation-techniques/semantic-similarity/test6_semantic_similarity.py, phase-2-evaluation-techniques/semantic-similarity/healthcare_test_cases.xlsx
Audit findings
C09,S02,S11,S16
Reuse strategy
Reuse the healthcare near misses and threshold discussion.
Teaching and experiment gaps
Add cached vectors, negations and numeric contrast pairs.
Technical corrections
Separate similarity bands from correctness and exclude transport errors.
M05-T03 Combining evaluators and handling uncertainty Needs Improvement · 5h

Intermediate · Core

Link to topic
Why it matters
An aggregate can hide an unacceptable error or an evaluator failure.
Concepts
Hard gates, rule plus proxy cascades, metric applicability, score vectors, disagreement, human escalation and threshold trade-offs.
Mandatory prerequisites
M03-T01, M03-T03
Recommended prerequisites
M05-T01, M05-T02
Conditional prerequisites
M05-T02 — required when tuning a semantic triage threshold instead of consuming supplied proxy scores
Implementation readiness
Use P02 for ordered checks and P06 for counts and rates.
Demonstration
Apply a numeric gate, semantic triage and human review to one fixture and compare with a single averaged score.
Exercise and assessment
Propose and defend an escalation rule with coverage and error counts.
Theory (hrs)
1h
Guided (hrs)
1h 30m
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
5h
Instructor-led (hrs)
3h 30m

Repository mapping

Audit findings
C02,C05,C09,S02
Reuse strategy
Reuse proxy disagreement and two-layer scoring examples.
Teaching and experiment gaps
Add a decision table, N/A controls and escalation cost.
Technical corrections
Keep hard constraints visible and avoid double-counting correlated evidence.
M06 Model judges and comparative evaluation 3 topics · 17h

Intermediate · Core · Judges

Description
Validate judge evidence with human labels before delegating consequential decisions to an evaluator.
Objectives
Build pointwise and comparative rubrics, measure inter-rater agreement and diagnose judge bias.
Labs
Evaluate identical, frozen responses with human raters, a local or replayed judge and a deterministic baseline.
Competencies
Produce a held-out judge report with valid verdict coverage, per-class errors, agreement and bias checks.
Completion
Submit a frozen-response judge report with coverage, class errors, agreement and pairwise order checks. Match human and model verdicts by response ID and replicate a swapped pair. Treating malformed verdicts as substantive agreement or comparing different responses fails.
Theory (hrs)
4h 30m
Guided (hrs)
5h
Experiment (hrs)
4h 30m
Exercises (hrs)
3h
Learner total (hrs)
17h
Instructor-led (hrs)
11h
M06-T01 Pointwise LLM-as-a-Judge Needs Correction · 5h 30m

Intermediate · Core

Link to topic
Why it matters
A model can scale rubric application while producing confident but unsupported labels.
Concepts
Criterion isolation, pointwise scoring, anchors, reference-guided versus reference-free evaluation, evidence spans, schema validation and fallback.
Mandatory prerequisites
M03-T02, M03-T03
Recommended prerequisites
M04-T02, M05-T03
Conditional prerequisites
—
Implementation readiness
Use P02 and P04 for structured verdict replay. A live judge needs P05 and an approved local or hosted setup.
Demonstration
Compare a structured saved judge verdict with a human adjudication on the exact same response.
Exercise and assessment
Reject malformed, out-of-range and unsupported verdicts and record why each was unscorable.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
5h 30m
Instructor-led (hrs)
3h 30m

Repository mapping

Existing files
docs/test5.html, phase-2-evaluation-techniques/llm-as-a-judge/llm_judge_evaluator.py, phase-2-evaluation-techniques/llm-as-a-judge/banking_bot_test_results.xlsx, config.yaml
Audit findings
C01,S04,S07,S08,S09
Reuse strategy
Keep the human-first disagreement review.
Teaching and experiment gaps
Add frozen responses with history, anchor examples and local replay verdicts.
Technical corrections
Align response IDs, validate schema, correct provider defaults and stop guessing verdicts from text.
M06-T02 Human agreement, calibration and bias Needs Improvement · 6h

Intermediate · Core

Link to topic
Why it matters
Human and model disagreements identify ambiguous rubrics, default preferences and systematic miss rates.
Concepts
Percent agreement, confusion matrices, Cohen's kappa intuition, prevalence effects, repeat judge consistency, verbosity/style and model self-preference, held-out calibration.
Mandatory prerequisites
M06-T01
Recommended prerequisites
M04-T02, M04-T03
Conditional prerequisites
M04-T03 — required when constructing your own calibration and holdout sets
Implementation readiness
Use P02 and P06 to produce a confusion matrix from fixed labels.
Demonstration
Compare independent human and judge labels on a frozen table using Python and inspect disagreement by slice.
Exercise and assessment
Recalibrate on anchors and report held-out error rates, sample sizes and unresolved disagreements.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Audit findings
C01,C07,C12,S09,V03
Reuse strategy
Use documented disagreements and the perfect trajectory result as review prompts.
Teaching and experiment gaps
Add authoritative evidence labels, second-rater labels and an ambiguous example.
Technical corrections
Distinguish uncertain human labels from adjudicated targets and reconcile sample counts.
M06-T03 Pairwise comparisons and ranking New · 5h 30m

Intermediate · Core

Link to topic
Why it matters
Comparative judgments can simplify preference assessment and are sensitive to order and tie policy.
Concepts
Blind randomization, position swaps, tie and abstain policies, win rates, Bradley–Terry intuition, nontransitivity and multi-model uncertainty.
Mandatory prerequisites
M03-T03
Recommended prerequisites
M06-T01, M06-T02
Conditional prerequisites
M06-T01 — required when adding a live model judge to the comparative lab
Implementation readiness
Use P02 and P06 to count wins, ties and order reversals.
Demonstration
Score the same answer pairs in reversed order and construct a win/tie table from saved judgments.
Exercise and assessment
Detect an order-sensitive pair and justify whether the evidence supports a ranking.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
5h 30m
Instructor-led (hrs)
3h 30m

Repository mapping

Existing files
—
Audit findings
—
Reuse strategy
Build on the pointwise frozen fixture with a revised rubric.
Teaching and experiment gaps
Add three-system comparisons, balanced order and tie controls.
Technical corrections
—
M07 Reproducibility, statistics and experiments 3 topics · 18h

Intermediate · Core · Statistics

Description
Start with visual and intuitive examples, then write small Python analyses on frozen cases.
Objectives
Explain variability, quantify paired uncertainty, choose sample sizes and detect regressions.
Labs
Compare two fictional banking configurations using repeated responses, case-level aggregation and Python resampling.
Competencies
Reproduce the report from a manifest and reach a defensible improvement, regression or inconclusive decision.
Completion
Submit a reproducible paired comparison with case counts, effect size, interval and decision. Rerun from the manifest and explain an inconclusive result. Resampling repeated draws as independent cases or reporting only a point estimate fails.
Theory (hrs)
5h
Guided (hrs)
5h 30m
Experiment (hrs)
4h 30m
Exercises (hrs)
3h
Learner total (hrs)
18h
Instructor-led (hrs)
12h
M07-T01 Variance, repetition and score aggregation Needs Correction · 5h 30m

Intermediate · Core

Link to topic
Why it matters
Repeated draws from the same case measure variation within that case and do not add independent case coverage.
Concepts
Analysis units, between/within-case variance, seeds and effective config, repeated measures, macro/micro means, slice weights, denominators and missingness.
Mandatory prerequisites
M02-T01
Recommended prerequisites
M01-T02, M04-T03, M05-T03
Conditional prerequisites
—
Implementation readiness
Use P02 and P06 for grouped counts and aggregation. Supplied observations replace live generation.
Demonstration
Plot saved repeated outcomes in a case-by-setting table, change slice weights and observe a reversed aggregate.
Exercise and assessment
Report attempted, valid and scorable rates separately and explain what would reproduce the result.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
5h 30m
Instructor-led (hrs)
3h 30m

Repository mapping

Existing files
docs/test2.html, phase-1-first-experiments/test2_full_evaluations.py, docs/test8.html, phase-3-what-you-are-evaluating/temperature-sensitivity/temperature_evaluator.py
Audit findings
C08,C11,S02,S08,V04
Reuse strategy
Reuse repeated tasks and the temperature variation as critique material.
Teaching and experiment gaps
Add grouped replay, warm/cold controls and aggregation walkthrough.
Technical corrections
Qualify one-draw claims and preserve run and stage metadata.
M07-T02 Confidence intervals, significance and sample size New · 6h 30m

Intermediate · Core

Link to topic
Why it matters
Small apparent differences can be sampling noise or too small to matter operationally.
Concepts
Effect size, confidence interval intuition, paired case bootstrap, permutation test intuition, practical significance, power/sample planning and multiplicity.
Mandatory prerequisites
M07-T01
Recommended prerequisites
M04-T03
Conditional prerequisites
—
Implementation readiness
Use P02 and P06 to resample paired records. The lesson introduces interval and significance concepts.
Demonstration
Resample whole case pairs in Python and compare the interval with an independent-sample calculation.
Exercise and assessment
Vary sample size or minimum effect and explain why a result is inconclusive without overclaiming equivalence.
Theory (hrs)
2h
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h 30m
Instructor-led (hrs)
4h 30m

Repository mapping

Existing files
—
Audit findings
—
Reuse strategy
Treat existing pass-rate tables as hypotheses for new exercises.
Teaching and experiment gaps
Add hand-resampling diagram, code and answer sheet for clustered outputs.
Technical corrections
—
M07-T03 Comparing models, prompts and A/B variants Needs Improvement · 6h

Intermediate · Core

Link to topic
Why it matters
A regression decision depends on what changed, what was held fixed and which risks matter.
Concepts
Paired offline experiments, locked baselines, key slices, noninferiority margins, guardrails, offline versus randomized online A/B effects and regression reporting.
Mandatory prerequisites
M02-T03, M07-T02
Recommended prerequisites
M04-T03
Conditional prerequisites
—
Implementation readiness
Use P02 and P06 for paired comparisons and simulated exposure logs.
Demonstration
Compare frozen prompt versions on the same case IDs, then simulate a user-randomized A/B with exposure logs.
Exercise and assessment
Write a rollout recommendation with effect size, uncertainty, guardrails, confounds and stopping criteria.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Audit findings
C04,C08,C11,V04
Reuse strategy
Keep the comparative examples as discussion prompts.
Teaching and experiment gaps
Add controlled one-variable tests and simulated online outcomes.
Technical corrections
Match inputs and judge or attribute changes explicitly.
Advanced5 modules · 87h
M08 RAG system evaluation 3 topics · 19h 30m

Advanced · Core · RAG

Description
Evaluate retrieval and generation independently before judging end-to-end answers.
Objectives
Measure ranked retrieval, score claims and citations, and isolate the failing pipeline stage.
Labs
Reuse fictional policy guides and create one labeled replay from query through citations.
Competencies
Diagnose a held-out failure and verify a targeted change against retrieval and answer metrics.
Completion
Submit a labeled retrieval and claim report with a citation check and one diagnosis. Verify a hand-computed rank metric, then repeat a controlled oracle comparison. A report that hides a wrong answer behind an average or omits support for a citation fails.
Theory (hrs)
5h 30m
Guided (hrs)
6h
Experiment (hrs)
5h
Exercises (hrs)
3h
Learner total (hrs)
19h 30m
Instructor-led (hrs)
13h 30m
M08-T01 Ingestion, chunking and ranked retrieval Needs Improvement · 6h 30m

Advanced · Core

Link to topic
Why it matters
A generator cannot reliably cite supporting evidence that the retriever missed.
Concepts
Parsing, chunking and overlap, source IDs, sparse/dense/hybrid retrieval, top-k, gold relevance, Precision@k, Recall@k, MRR, nDCG and unlabeled passages.
Mandatory prerequisites
M02-T02
Recommended prerequisites
M04-T01, M05-T02
Conditional prerequisites
M05-T02 — required when modifying the dense retrieval encoder or interpreting its similarity values
Implementation readiness
Use P02 and P06 for metrics on supplied rankings. Chunking uses local text files.
Demonstration
Hand-score two ranked lists and reproduce metrics in Python before changing chunks in a replay.
Exercise and assessment
Add a graded relevance judgment and explain a case where MRR and Recall@k disagree.
Theory (hrs)
2h
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h 30m
Instructor-led (hrs)
4h 30m

Repository mapping

Existing files
docs/test9.html, phase-3-what-you-are-evaluating/rag-evaluation/chunker.py, phase-3-what-you-are-evaluating/rag-evaluation/retriever.py, phase-3-what-you-are-evaluating/rag-evaluation/chunks.json, phase-3-what-you-are-evaluating/rag-evaluation/rag_test_cases.xlsx
Audit findings
C02
Reuse strategy
Retain the paragraph chunker, retriever and ranked evidence.
Teaching and experiment gaps
Add gold passage IDs, graded gains and hand-computed solutions.
Technical corrections
Keep document origin stable and declare the handling of empty gold sets.
M08-T02 Context sufficiency, answer correctness and citations Needs Correction · 6h 30m

Advanced · Core

Link to topic
Why it matters
Answers may be correct without supporting evidence or faithfully repeat stale information.
Concepts
Context relevance versus sufficiency, claim correctness, groundedness/faithfulness, hallucination, citation validity/entailment/completeness, answerability and abstention.
Mandatory prerequisites
M08-T01
Recommended prerequisites
M03-T03, M06-T01
Conditional prerequisites
M06-T01 — required when replacing manual claim review with a model judge
Implementation readiness
Use a manual claim-to-evidence worksheet. Use P04 to inspect saved retrieved chunks as JSON.
Demonstration
Annotate claims and source IDs for four examples, including a supported but wrong answer and a valid abstention.
Exercise and assessment
Grade a fabricated citation, an unsupported correct answer and an unwarranted refusal separately.
Theory (hrs)
2h
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h 30m
Instructor-led (hrs)
4h 30m

Repository mapping

Existing files
docs/test7.html, phase-2-evaluation-techniques/groundedness/groundedness_evaluator.py, docs/test9.html, phase-3-what-you-are-evaluating/rag-evaluation/rag_evaluator.py
Audit findings
C02,C03,C12,S04,S09
Reuse strategy
Reuse the invented-specifics and misleading-aggregate cases as exercises.
Teaching and experiment gaps
Add authoritative facts, claim-to-span labels and explicit refusal expectations.
Technical corrections
Separate source evidence from behavior policy, and reconcile conflicting sample labels.
M08-T03 Failure attribution, optimization and RAG regression Needs Improvement · 6h 30m

Advanced · Core

Link to topic
Why it matters
A better response score does not show whether retrieval or generation improved.
Concepts
Oracle retrieval, controlled context swaps, distractors, stale and conflicting sources, held-out slices, threshold tuning and regression gates.
Mandatory prerequisites
M08-T02
Recommended prerequisites
M07-T01, M07-T03
Conditional prerequisites
M07-T02 — required when reporting confidence intervals for a comparative result
Implementation readiness
Replay fixed cases using P02 and P04. Live reranking needs the prepared local retrieval stack.
Demonstration
Score one baseline against oracle passage and one retrieval change using the same tasks and answer rubric.
Exercise and assessment
Identify the root cause for a replayed failure and specify a regression test with a failure-stage assertion.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
2h
Exercises (hrs)
1h
Learner total (hrs)
6h 30m
Instructor-led (hrs)
4h 30m

Repository mapping

Existing files
docs/test9.html, phase-3-what-you-are-evaluating/rag-evaluation/rag_test_cases.xlsx, docs/test11.html
Audit findings
C02,C04,S09
Reuse strategy
Keep the TC21 and retrieval-confusion walkthroughs.
Teaching and experiment gaps
Add oracle controls, fixed generator outputs and before/after regression fixtures.
Technical corrections
Use answer/claim outcomes and correct sample denominators.
M09 Framework adapters and tool selection 3 topics · 15h 30m

Advanced · Core · Frameworks

Description
Apply commercial and open tooling through a neutral evaluation contract and compare definitions on matched inputs.
Objectives
Map case objects and metrics, explain assumptions and select an adapter based on evidence and operational needs.
Labs
Use one fixture set to compare a hand-computed scorer and two adapter result shapes.
Competencies
Show per-case parity where metrics match and explain one intentional divergence.
Completion
Submit one neutral case mapped to two adapter outputs and a tool-choice memo. Verify one matched metric against a hand calculation and explain a divergence or N/A. Comparing incompatible scores as parity or discarding adapter errors fails.
Theory (hrs)
4h
Guided (hrs)
5h
Experiment (hrs)
3h 30m
Exercises (hrs)
3h
Learner total (hrs)
15h 30m
Instructor-led (hrs)
10h 30m
M09-T01 Common evaluator contract and framework selection Needs Improvement · 4h 30m

Advanced · Core

Link to topic
Why it matters
Portable contracts keep experiments reproducible when a framework or judge changes.
Concepts
Case/response IDs, metric intent and input schema, results with valid/error/N/A, provenance, offline runner versus hosted service, cost, telemetry and capability fit.
Mandatory prerequisites
M04-T01
Recommended prerequisites
M05-T03, M07-T03
Conditional prerequisites
—
Implementation readiness
Use P04 to inspect case and result schemas. Adapter assessment can run from saved input and output records.
Demonstration
Map one frozen evaluation record to candidate adapter inputs and verify a known score against Python.
Exercise and assessment
Write a tool-choice memo covering local replay, lockfile/platform, data boundary and migration cost.
Theory (hrs)
1h
Guided (hrs)
1h 30m
Experiment (hrs)
1h
Exercises (hrs)
1h
Learner total (hrs)
4h 30m
Instructor-led (hrs)
3h

Repository mapping

Existing files
docs/test11.html, requirements.txt, config.yaml
Audit findings
C04,S06,S08,S15,V02
Reuse strategy
Reuse case-object explanation and framework comparison as a discussion starter.
Teaching and experiment gaps
Add a neutral schema, checked score parity, deterministic preflight and explicit version metadata.
Technical corrections
Distinguish scoring constructs and identify effective settings.
M09-T02 DeepEval and RAGAS on frozen RAG records Needs Correction · 6h

Advanced · Core

Link to topic
Why it matters
Named metrics can answer different questions and require different labels or model calls.
Concepts
DeepEval test cases, contextual relevance, faithfulness and thresholds. RAGAS variants for context precision, faithfulness and response relevance, reference requirements and judge/embedding calls. Choose DeepEval for Python assertions and RAGAS for retrieval/generation metric suites when prerequisites match.
Mandatory prerequisites
M08-T02, M09-T01
Recommended prerequisites
M06-T01
Conditional prerequisites
M06-T01 — required when configuring a custom judge rather than reading recorded adapter results
Implementation readiness
Use P04 for replay. A live run needs the pinned framework, its compatible model or embedding setup, and P05 if hosted.
Demonstration
Walk one saved record through both adapters and compare which passages, claims and answers drive scores. Verify known zero, N/A and timeout cases.
Exercise and assessment
Explain one matched-input metric disagreement, required reference, expected runtime and optional hosted cost.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Existing files
docs/test11.html, phase-4-frameworks-and-tools/deepeval-evaluation/test11_deepeval_evaluator.py, phase-4-frameworks-and-tools/ragas-evaluation/test10_ragas_evaluator.py, phase-4-frameworks-and-tools/ragas-evaluation/ragas_test_cases.xlsx
Audit findings
C03,C04,S03,S08,S15,V01
Reuse strategy
Retain Test 11 explanations and use Test 10 code as an adapter candidate.
Teaching and experiment gaps
Add Test 10 teaching notes, tested variant mapping and saved outputs for replay.
Technical corrections
Pin compatible versions, preflight credentials before calls, join by ID and preserve errors and denominators.
M09-T03 promptfoo and OpenAI Evals New · 5h

Advanced · Core

Link to topic
Why it matters
Declarative and hosted runs can support review workflows once score meaning is understood.
Concepts
promptfoo for declarative prompt/provider matrices and assertions. OpenAI Evals for evaluation definitions, datasets and grader/runs in an OpenAI-centered workflow. Compare portability, supported graders, spend, data visibility and versioning.
Mandatory prerequisites
M09-T01
Recommended prerequisites
M06-T01, M07-T03
Conditional prerequisites
—
Implementation readiness
Use P03 and P04 to inspect declarative specs. The managed variant uses P05 only if explicitly enabled.
Demonstration
Translate two frozen prompts and one deterministic grading rule into declarative and hosted-style evaluation specs. Inspect mock results locally.
Exercise and assessment
Recommend a tool for a CI matrix and another for a team using managed grading, with limits and a fixture fallback.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h
Exercises (hrs)
1h
Learner total (hrs)
5h
Instructor-led (hrs)
3h 30m

Repository mapping

Existing files
—
Audit findings
—
Reuse strategy
Reuse the common contract rather than duplicating metrics.
Teaching and experiment gaps
Add a local matrix lab and an offline mock of the hosted workflow.
Technical corrections
—
M10 Agent and workflow evaluation 3 topics · 18h

Advanced · Core · Agents

Description
Evaluate what a workflow does, what state it leaves behind and how it recovers under controlled failures.
Objectives
Measure calls, arguments, outputs, observable planning, task completion and recovery within budgets.
Labs
Replay frozen traces for fictional account and policy tasks against mock tool results and state.
Competencies
Score alternative valid paths and reject a plausible answer that leaves an incorrect or unauthorized state.
Completion
Submit a trace replay scorecard covering arguments, tool results, termination and final state. Pass two equivalent valid paths and reject a wrong-state trace with a plausible answer. An exact-path check alone or an unexplained duplicate action fails.
Theory (hrs)
4h 30m
Guided (hrs)
6h
Experiment (hrs)
4h 30m
Exercises (hrs)
3h
Learner total (hrs)
18h
Instructor-led (hrs)
12h
M10-T01 Execution model, structured calls and tool contracts Needs Correction · 6h

Advanced · Core

Link to topic
Why it matters
Tool response quality and authorization can fail independently of language quality.
Concepts
Agent loop, action schemas, tool selection, typed arguments, tool outputs, permissions, required/optional calls, refusals and step limits.
Mandatory prerequisites
M03-T02
Recommended prerequisites
M01-T03, M05-T03
Conditional prerequisites
M08-T01 — required only for the document-retrieval tool extension
Implementation readiness
Use P02 and P04 to validate saved calls against a mock tool contract.
Demonstration
Validate two calls to a read-only mock bank tool and compare traces for an unnecessary call and a useful clarification.
Exercise and assessment
Identify an incorrect argument and verify the corresponding result, action and trust boundary.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Existing files
docs/test12.html, phase-5-production-eval/agent-evaluation/agent.py, phase-5-production-eval/agent-evaluation/tools.py, phase-5-production-eval/agent-evaluation/test12_agent_evaluator.py, phase-5-production-eval/agent-evaluation/agent_test_cases.xlsx
Audit findings
C06,S05,S13
Reuse strategy
Reuse local mock data, tools and visible schemas.
Teaching and experiment gaps
Add typed validators, authorization fixtures and expected tool outcomes.
Technical corrections
Replace unbounded arithmetic evaluation and carry typed tool errors through replay.
M10-T02 Trajectories, planning outcomes and final state Needs Correction · 6h

Advanced · Core

Link to topic
Why it matters
A path may look plausible while the task remains incomplete, and multiple paths may satisfy the same goal.
Concepts
State invariants, event/order constraints, task completion, alternative plans, trace-grounding, stop reasons, partial success and judgment N/A.
Mandatory prerequisites
M02-T01, M10-T01
Recommended prerequisites
M03-T03, M06-T01
Conditional prerequisites
M06-T01 — required when adding judge-scored trajectories
Implementation readiness
Use P02 and P04 to inspect saved events and verify state predicates.
Demonstration
Replay a trace with a missing second step and another with a different valid path. Compare state checks and qualitative judgments.
Exercise and assessment
Explain why a perfect trajectory score can hide missed behavior and defend an outcome-first rubric.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Existing files
docs/test12.html, docs/test13.html, phase-5-production-eval/trajectory-evaluation/test13_trajectory_evaluator.py, phase-5-production-eval/trajectory-evaluation/sample_agent_evaluation_results.json, phase-5-production-eval/trajectory-evaluation/results/sample_trajectory_evaluation_results.json
Audit findings
C06,C07,S05
Reuse strategy
Retain the Test 12 to Test 13 replay and compare both scoring layers.
Teaching and experiment gaps
Add hard negatives, goal-state predicates, annotated alternative paths and scoring denominators.
Technical corrections
Validate judged dimensions and compute verdicts independently.
M10-T03 Recovery, reliability and efficiency Needs Improvement · 6h

Advanced · Core

Link to topic
Why it matters
Retries may improve success while increasing cost, delay or duplicated side effects.
Concepts
Scripted faults, timeouts, retry budgets, idempotency, interrupted state, recoverability, success per cost and latency versus task completion.
Mandatory prerequisites
M10-T02
Recommended prerequisites
M07-T01
Conditional prerequisites
—
Implementation readiness
Use P02 and P06 to compare retries, budgets and final states with synthetic timings.
Demonstration
Replay a bounded transient failure and compare retry policy, end state and cost from simulated timings.
Exercise and assessment
Demonstrate a safe retry, then choose a threshold for recovery within budget.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Existing files
docs/test12.html, phase-5-production-eval/agent-evaluation/agent.py, phase-5-production-eval/agent-evaluation/test12_agent_evaluator.py
Audit findings
S02,S05,S13
Reuse strategy
Reuse the practical timeout discussion with mock faults.
Teaching and experiment gaps
Add state snapshots, failure injection, latency and token trace fields.
Technical corrections
Separate infrastructure errors and test idempotency before discussing writes.
M11 Safety and responsible AI evaluation 3 topics · 17h

Advanced · Core · Safety

Description
Establish a safety measurement plan that covers risky outcomes and permitted use.
Objectives
Classify risk, evaluate refusal and over-refusal, audit fairness and privacy, and verify safety regressions.
Labs
Score a synthetic safety set against a fictional bank policy and document uncertain labels.
Competencies
Produce a risk matrix, slice report and explicit release criteria with review paths for severe or ambiguous cases.
Completion
Submit a risk matrix and scorecard with labeled appropriate refusals, over-refusals and one matched subgroup check. Recompute counts and assess a severe case. An undisaggregated mean or unspecified policy boundary fails.
Theory (hrs)
4h 30m
Guided (hrs)
5h
Experiment (hrs)
4h 30m
Exercises (hrs)
3h
Learner total (hrs)
17h
Instructor-led (hrs)
11h
M11-T01 Risk taxonomy, harmful content and refusal Needs Correction · 5h 30m

Advanced · Core

Link to topic
Why it matters
Safety depends on intended policy and on continuing to answer permissible requests.
Concepts
Severity, toxic or harmful output categories, policy versus factual correctness, appropriate refusal, over-refusal and safe alternatives.
Mandatory prerequisites
M03-T03
Recommended prerequisites
M04-T02, M05-T03, M06-T02
Conditional prerequisites
—
Implementation readiness
Use the supplied fictional policy and label worksheet. No live model is needed.
Demonstration
Label fixed benign assessment fixtures as answer, abstain, refuse or escalate with expected behavior and reason.
Exercise and assessment
Report correct refusals and false refusals with their own denominators and adjudicate a borderline case.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
5h 30m
Instructor-led (hrs)
3h 30m

Repository mapping

Audit findings
C03,C05,C12
Reuse strategy
Reuse examples of scope, uncertainty and high-stakes policies with fictionalized answers.
Teaching and experiment gaps
Add a severity rubric, acceptable alternatives and balanced refusal fixtures.
Technical corrections
Validate behavior beyond keywords and keep abstention separate.
M11-T02 Bias, fairness and privacy controls New · 5h 30m

Advanced · Core

Link to topic
Why it matters
A global metric can hide subgroup harms, and observability can increase unnecessary data exposure.
Concepts
Counterfactual and matched-slice tests, label and population bias, disparate error rates, consent, data minimization and synthetic sensitive-data checks.
Mandatory prerequisites
M11-T01
Recommended prerequisites
M04-T03, M07-T02
Conditional prerequisites
M07-T02 — required when quantifying uncertainty for a subgroup estimate
Implementation readiness
Use P06 for matched rates and synthetic privacy-control records.
Demonstration
Compare matched fictional customers and inspect privacy controls with dummy records.
Exercise and assessment
Explain confounds and identify what evidence and review are needed before a fairness or privacy claim.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
5h 30m
Instructor-led (hrs)
3h 30m

Repository mapping

Existing files
—
Audit findings
S12
Reuse strategy
Use the fictional banking context without importing personal data.
Teaching and experiment gaps
Add paired cases, synthetic trace receipts and human review protocol.
Technical corrections
—
M11-T03 Robustness, safety benchmarks and regression Needs Improvement · 6h

Advanced · Core

Link to topic
Why it matters
A high benchmark average can hide severe edge cases, instability or a new false refusal.
Concepts
Perturbation controls, held-out safety sets, stress versus representative evaluation, repeated runs, false-pass severity, escalation and lockboxes.
Mandatory prerequisites
M11-T01
Recommended prerequisites
M07-T03, M11-T02
Conditional prerequisites
M11-T02 — required for fairness and privacy slices in an expanded safety suite
Implementation readiness
Use P02 and P06 to replay a fixed rubric and compare severity slices.
Demonstration
Apply harmless formatting and paraphrase variants to a frozen safety set, then compare a policy update by severity.
Exercise and assessment
Design a slice-sensitive safety gate and state what the benchmark cannot establish.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Audit findings
C08,C13
Reuse strategy
Adapt existing scope and robustness discussions.
Teaching and experiment gaps
Add fixed controls, independent labels and reproducible safety regression cases.
Technical corrections
Report independent case counts and preserve critical slices.
M12 Defensive red-team evaluation 3 topics · 17h

Advanced · Core · Red Teaming

Description
Plan and assess authorized defensive tests of isolated fictional systems using fixed benign scenarios, synthetic records and mock tools.
Objectives
Identify trust boundaries, measure observed policy violations and verify mitigations on matched evidence.
Labs
Evaluate canned outcomes from a fictional bank with document and tool trust boundaries, then write a mitigation regression report.
Competencies
Explain what was measured, what was in scope, which fixes held and what evidence remains uncertain.
Completion
Submit a defensive assessment memo scoped to the fictional system. Verify control and mitigation outcomes on fixed, harmless fixtures and report match counts and false refusals. Unscoped testing or an unverified mitigation claim fails.
Theory (hrs)
4h 30m
Guided (hrs)
5h
Experiment (hrs)
4h 30m
Exercises (hrs)
3h
Learner total (hrs)
17h
Instructor-led (hrs)
11h
M12-T01 Threat modeling, scope and test design Needs Improvement · 5h 30m

Advanced · Core

Link to topic
Why it matters
A clear threat model links input channels and assets to the defensive behavior expected of the system.
Concepts
Direct and indirect prompt injection as threat categories, jailbreaks, protected assets, trust boundaries, authorized test scope, containment and harmless fixtures.
Mandatory prerequisites
M11-T01
Recommended prerequisites
M08-T01, M10-T01
Conditional prerequisites
—
Implementation readiness
Inspect a provided trust-boundary diagram and benign fixtures. No scanner or live target is used.
Demonstration
Diagram user, document and tool trust boundaries. Inspect inert labels for prepared policy-conflict cases.
Exercise and assessment
Specify assets, failure outcomes, negative controls, test boundaries and responsible reporting.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
5h 30m
Instructor-led (hrs)
3h 30m

Repository mapping

Audit findings
C13
Reuse strategy
Reuse the glossary and fixed scope examples for discussion.
Teaching and experiment gaps
Add threat-model worksheet and prewritten benign outcome fixtures.
Technical corrections
Treat system controls and authorization as independently testable.
M12-T02 Defensive checks for RAG, agents and data exposure New · 6h

Advanced · Core

Link to topic
Why it matters
Untrusted content, permission mistakes and exposure can occur even when an assistant response sounds appropriate.
Concepts
Indirect document trust, RAG provenance, mock-tool authorization, least privilege, synthetic exposure markers, attack-success denominators and matched normal cases.
Mandatory prerequisites
M12-T01
Recommended prerequisites
M08-T02, M10-T02, M11-T02
Conditional prerequisites
M10-T01 — required when independently interpreting raw mock-tool arguments in the agent extension
Implementation readiness
Use P04 and P06 to read stored synthetic outcomes and calculate rates on supplied controls.
Demonstration
Compare saved control-off and control-on records from an isolated mock system. Check the simulated authorization decision and whether a dummy marker reached the output.
Exercise and assessment
Calculate observed violation rate by scenario and distinguish authorization failures from correct refusals and unavailable evidence.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Existing files
—
Audit findings
C13
Reuse strategy
Carry forward mock states and separate answer/authorization labels from M10.
Teaching and experiment gaps
Add canned before/after traces, benign marker receipts and a scoring key.
Technical corrections
—
M12-T03 Mitigation verification, reporting and defensive tooling New · 5h 30m

Advanced · Core

Link to topic
Why it matters
A change needs independent evidence on fixed cases, unseen controls and ordinary permitted behavior.
Concepts
Re-test protocols, severity and reproducible evidence, reporting and ownership. Garak and PyRIT report vocabulary and scope constraints. Static contract and replay checks for targeted controls.
Mandatory prerequisites
M12-T02
Recommended prerequisites
M11-T03
Conditional prerequisites
—
Implementation readiness
Review fixed offline report excerpts. The lab uses synthetic evidence and does not run scanners.
Demonstration
Read prepared Garak and PyRIT report excerpts and reconcile their labels with a manually scored, held-out synthetic fixture.
Exercise and assessment
Write a red-team finding and verification memo with coverage, confirmed failures, false refusals, remediation and remaining risk.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
5h 30m
Instructor-led (hrs)
3h 30m

Repository mapping

Existing files
—
Audit findings
C13
Reuse strategy
Use offline report interpretation alongside the neutral contract.
Teaching and experiment gaps
Add fictional report exports, outcome labels and mitigation regression worksheet.
Technical corrections
—
Production3 modules · 48h
M13 Continuous evaluation and releases 3 topics · 17h 30m

Advanced · Core · Production Engineering

Description
Turn evaluation evidence into reliable checks and release decisions while preserving diagnostic artifacts.
Objectives
Design a versioned pipeline, define CI/CD quality gates, and decide when to pause or roll back.
Labs
Turn the bank fixture suite into an offline CI run with immutable candidate artifacts and a release decision.
Competencies
Demonstrate pass, fail and insufficient evidence paths and replay the report from its manifest.
Completion
Submit a fixture-driven gate, a reproducible manifest and a release decision. Replay pass, fail and insufficient-evidence outcomes and explain a rollback trigger. Treating infrastructure errors as passes or allowing a critical failure through an aggregate fails.
Theory (hrs)
4h 30m
Guided (hrs)
5h 30m
Experiment (hrs)
4h 30m
Exercises (hrs)
3h
Learner total (hrs)
17h 30m
Instructor-led (hrs)
11h 30m
M13-T01 Evaluation pipelines and versioned run contracts Needs Correction · 6h

Advanced · Core

Link to topic
Why it matters
A run must be attributable to one prompt, model, dataset, scorer and configuration.
Concepts
Build and score separation, immutable run metadata, fixtures and live tiers, parallelism, caching, secure config, evaluator error taxonomy and artifact integrity.
Mandatory prerequisites
M03-T01, M04-T01
Recommended prerequisites
M07-T01, M09-T01
Conditional prerequisites
M09-T01 — required when adding an external framework adapter
Implementation readiness
Use P01–P04 to replay a local pipeline and inspect its run manifest.
Demonstration
Build a fixture-driven pipeline design with preflight, case selection, outputs, version tags and safe report rendering.
Exercise and assessment
Reconstruct a run manifest and classify invalid input, evaluator timeout and valid failure.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Existing files
config.yaml, model_client.py, requirements.txt, phase-3-what-you-are-evaluating/rag-evaluation/report_generator.py
Audit findings
S02,S06,S07,S08,S10,S11,S14,S15
Reuse strategy
Reuse config, adapters and report layouts as examples to review.
Teaching and experiment gaps
Add dry-run, version contract, harmless markup fixture and reproducible environment profile.
Technical corrections
Separate errors from data, escape reports, verify links and remove tracked secret guidance.
M13-T02 CI/CD quality gates and regression evidence New · 6h

Advanced · Core

Link to topic
Why it matters
A flaky or underpowered gate can block safe changes or let critical regressions through.
Concepts
Stable baselines, fast and slow tiers, absolute and relative thresholds, uncertainty, critical-slice gates, retry policy, artifact review and exceptions.
Mandatory prerequisites
M07-T02, M13-T01
Recommended prerequisites
M07-T03, M11-T03
Conditional prerequisites
M11-T03 — required when defining a separate safety-benchmark gate
Implementation readiness
Use P02, P03 and P06 for a simulated CI gate. A hosted CI service is optional.
Demonstration
Simulate candidate releases with performance regressions and metric failures, using the same locked fixtures.
Exercise and assessment
Implement a three-way decision and justify how a critical regression overrides aggregate improvement.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Existing files
—
Audit findings
V04
Reuse strategy
Build on the fixture scorecard and the neutral contract.
Teaching and experiment gaps
Add local CI simulation, audit log, uncertainty and infrastructure-failure tests.
Technical corrections
—
M13-T03 Release decisions, rollback and governance New · 5h 30m

Advanced · Core

Link to topic
Why it matters
A release requires ownership, explicit risk acceptance and a reversible decision backed by evidence.
Concepts
Version sign-off, model and prompt change management, canary/shadow comparison, kill criteria, rollback, approval trail and evidence retention.
Mandatory prerequisites
M02-T03, M13-T02
Recommended prerequisites
M12-T03
Conditional prerequisites
—
Implementation readiness
Use the release worksheet and simulated rollout. No service credentials are needed.
Demonstration
Write a release packet and run a tabletop rollback after a synthetic severe-slice regression.
Exercise and assessment
Present the owner, decision, rollback trigger, evidence link and follow-up dataset change.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
5h 30m
Instructor-led (hrs)
3h 30m

Repository mapping

Existing files
—
Audit findings
C13,V04
Reuse strategy
Reuse the training traces and scores as mock release evidence.
Teaching and experiment gaps
Add policy, sign-off and rollback templates plus decision scenarios.
Technical corrections
—
M14 Observability, operations and review loops 3 topics · 17h 30m

Advanced · Core · Production Engineering

Description
Follow live-style evidence from a trace into triage, labeling, drift detection and regression checks.
Objectives
Instrument traces, sample responsibly, define alerts and understand cost, latency and operational reliability.
Labs
Simulate a bank support incident using synthetic traces, a dashboard export and an offline score pipeline.
Competencies
Explain which signal moved, identify the cause and turn it into a versioned regression case with a traceable owner.
Completion
Submit a trace-based incident brief and new regression case. Correlate a synthetic event, justify a sampling or alert decision and recompute a tail latency or cost measure. Confusing telemetry loss with behavior drift or retaining unnecessary payloads fails.
Theory (hrs)
4h 30m
Guided (hrs)
6h
Experiment (hrs)
4h
Exercises (hrs)
3h
Learner total (hrs)
17h 30m
Instructor-led (hrs)
12h
M14-T01 Traces, monitoring and Langfuse Needs Correction · 5h 30m

Advanced · Core

Link to topic
Why it matters
Observability makes eval context inspectable across stages and repeated incidents.
Concepts
Trace/span/generation concepts, correlation IDs, tags, scored observations, telemetry loss and redaction. Choose Langfuse for inspectable LLM traces and scores with suitable deployment/data controls. Consider OpenTelemetry for cross-service context, with application-specific quality scoring elsewhere.
Mandatory prerequisites
M02-T03
Recommended prerequisites
M04-T01, M11-T02, M13-T01
Conditional prerequisites
M11-T02 — required when designing privacy controls for your own trace pipeline
Implementation readiness
Use P04 to read saved traces. The optional dashboard needs local Docker and Langfuse configured.
Demonstration
Follow a bank request through a saved trace and associate outcome, case ID, model and latency. Optionally reproduce with local Langfuse.
Exercise and assessment
Annotate a trace and show which fields to retain, redact or drop before collection.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h
Exercises (hrs)
1h
Learner total (hrs)
5h 30m
Instructor-led (hrs)
4h

Repository mapping

Existing files
docs/test14.html, phase-5-production-eval/langfuse-observability/test14_langfuse_evaluator.py, phase-5-production-eval/langfuse-observability/docker-compose.yml
Audit findings
C05,S07,S08,S12,S16,V02
Reuse strategy
Preserve the walkthrough and dashboard exploration.
Teaching and experiment gaps
Add de-identified export/replay, telemetry failure and trace coverage example.
Technical corrections
Correct provider/range claims and use loopback, safe credentials and payload controls.
M14-T02 Production sampling, human feedback, drift and alerts New · 6h

Advanced · Core

Link to topic
Why it matters
Trace populations, user feedback and evaluator behavior can change independently.
Concepts
Representative trace sampling, consent and retention, label delays, human-in-the-loop triage, selection bias, input/answer/evaluator drift, control charts and alert triage.
Mandatory prerequisites
M04-T03, M07-T02, M14-T01
Recommended prerequisites
M11-T02, M13-T03
Conditional prerequisites
—
Implementation readiness
Use P02, P04 and P06 to inspect sampled synthetic traces and delayed labels.
Demonstration
Compare synthetic weeks with changing traffic mix and delayed labels. Produce an alert and review queue.
Exercise and assessment
Distinguish drift from instrumentation loss and promote reviewed cases into an offline suite.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Existing files
—
Audit findings
S12,V04
Reuse strategy
Use Test 14 categories as context for a synthetic monitoring dataset.
Teaching and experiment gaps
Add trace sampling weights, dashboards, feedback routing and alert scenarios.
Technical corrections
—
M14-T03 Cost, tail latency and operational reliability Needs Improvement · 6h

Advanced · Core

Link to topic
Why it matters
Per-call averages can hide long waits, retry spend and the cost of an evaluation pipeline itself.
Concepts
TTFT, TPOT, duration, latency percentiles, tokens, cost per valid case, concurrency, error budget, SLOs and degradation modes.
Mandatory prerequisites
M07-T01, M14-T01
Recommended prerequisites
M10-T03
Conditional prerequisites
—
Implementation readiness
Use P02 and P06 for percentiles and per-outcome costs. All timings and rates are supplied.
Demonstration
Compute p50/p95 and cost per valid outcome from a trace export that includes warm-ups, retries and judge calls.
Exercise and assessment
Defend a cost and latency budget and diagnose an incident where quality improved but reliability worsened.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Audit findings
C08,S03,V04
Reuse strategy
Reuse the performance definitions and real latency anomalies as prompts.
Teaching and experiment gaps
Add token and percentile calculations, load profiles and cost model.
Technical corrections
Separate inference, evaluation and end-to-end timing and record failed attempts.
M15 End-to-end capstone 2 topics · 13h

Advanced · Core · Production Engineering

Description
Demonstrate ownership of evaluation from goals and annotated data to release evidence and incident follow-up.
Objectives
Integrate a release-quality suite, explain decision trade-offs and defend results in a teaching demonstration.
Labs
Submit a replayable evaluation package for one fictional feature spanning chatbot, RAG or agent behavior and production evidence.
Competencies
A peer can reproduce the result, challenge the decision and rerun a regression after a simulated incident.
Completion
Submit a replayable evaluation package, incident diagnosis and a release defense for the chosen feature. A peer must reproduce scores, reproduce the regression and observe an evidence-backed decision. An unreproducible report or no clear response to risk fails.
Theory (hrs)
2h
Guided (hrs)
3h 30m
Experiment (hrs)
4h
Exercises (hrs)
3h 30m
Learner total (hrs)
13h
Instructor-led (hrs)
8h
M15-T01 Design review and reproducible suite New · 6h 30m

Advanced · Core

Link to topic
Why it matters
A clear evidence plan lets teammates challenge a metric before the implementation hides its assumptions.
Concepts
Acceptance criteria, risk assessment, golden set versions, independent rubric review, paired baselines and environment constraints.
Mandatory prerequisites
M13-T02
Recommended prerequisites
M08-T03, M09-T01, M10-T03, M11-T03
Conditional prerequisites
M08-T03 — required for the RAG capstone variant / M10-T03 — required for the agent capstone variant
Implementation readiness
Use the offline harness and one assessed track. The FAQ capstone uses the mandatory route.
Demonstration
Draft the contract, choose a track and have a peer run your frozen evaluation plan.
Exercise and assessment
Submit a verified manifest and a suite containing good, incorrect, ambiguous and unscorable cases.
Theory (hrs)
1h
Guided (hrs)
2h
Experiment (hrs)
2h
Exercises (hrs)
1h 30m
Learner total (hrs)
6h 30m
Instructor-led (hrs)
4h

Repository mapping

Existing files
—
Audit findings
V03,V04
Reuse strategy
Draw on the banking fixtures and course scorecards.
Teaching and experiment gaps
Add briefs for FAQ, support RAG and account workflow variants and a grading rubric.
Technical corrections
—
M15-T02 Decision defense, incident response and teaching New · 6h 30m

Advanced · Core

Link to topic
Why it matters
Releasing and teaching an evaluation requires explaining what evidence can support under uncertainty.
Concepts
Decision memo, incident diagnosis, drift or regression, ownership, release/hold/rollback, credible evidence and peer feedback.
Mandatory prerequisites
M13-T03, M14-T02, M15-T01
Recommended prerequisites
M12-T03, M14-T03
Conditional prerequisites
M12-T03 — required for the defensive-assessment incident variant
Implementation readiness
Use the synthetic incident packet, report template and peer verification.
Demonstration
Diagnose a scripted incident, add a regression test and teach one evaluation failure with a reproducible demonstration.
Exercise and assessment
Defend the release decision and show a peer can verify the corrected outcome from the submitted artifacts.
Theory (hrs)
1h
Guided (hrs)
1h 30m
Experiment (hrs)
2h
Exercises (hrs)
2h
Learner total (hrs)
6h 30m
Instructor-led (hrs)
4h

Repository mapping

Existing files
—
Audit findings
V04
Reuse strategy
Use selected historical examples as prompts for the oral defense.
Teaching and experiment gaps
Add instructor scoring rubric, incident variants and model answers.
Technical corrections
—
Specialized / Expert3 modules · 54h 30m
M16 Evaluation across languages and modalities 3 topics · 17h 30m

Expert · Elective · Specialization

Description
Adapt data and rubrics when text quality measures fail to capture language, visual or long-context requirements.
Objectives
Design slice-aware metrics and evidence for three settings with different failure modes.
Labs
Build a small, domain-specific slice on top of the bank case contract and calibrate it.
Competencies
Demonstrate why a baseline score changes meaning and validate a revised rubric on held-out cases.
Completion
Submit a modality or locale-specific slice for each studied topic and a revised rubric. Verify an evidence label and one held-out counterexample per slice. Score-only comparisons with no domain evidence or incomparable samples fail.
Theory (hrs)
4h 30m
Guided (hrs)
5h 30m
Experiment (hrs)
4h 30m
Exercises (hrs)
3h
Learner total (hrs)
17h 30m
Instructor-led (hrs)
11h 30m
M16-T01 Multilingual and localized evaluation New · 5h 30m

Expert · Elective

Link to topic
Why it matters
Translation coverage and native-language correctness can diverge, especially for domain terms.
Concepts
Native annotations, locale and register, code-switching, script normalization, multilingual embeddings and stratified comparisons.
Mandatory prerequisites
M04-T03, M07-T02
Recommended prerequisites
M05-T02
Conditional prerequisites
M05-T02 — required when interpreting multilingual embedding scores
Implementation readiness
Use P02 and P06 and provide reviewed language labels for the selected locale.
Demonstration
Compare translated and native fictional FAQ cases against language-specific policies and evaluate normalization choices.
Exercise and assessment
Produce a per-locale scorecard and identify translation leakage and subgroup uncertainty.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
5h 30m
Instructor-led (hrs)
3h 30m

Repository mapping

Existing files
—
Audit findings
—
Reuse strategy
Retain the case schema and threshold lesson.
Teaching and experiment gaps
Add native reviewer guidance, parallel fixtures and locale-specific test controls.
Technical corrections
—
M16-T02 Multimodal evaluation New · 6h

Expert · Elective

Link to topic
Why it matters
A fluent answer can hallucinate a visual detail or attach the right explanation to the wrong region.
Concepts
OCR/chart and image QA, visual evidence, modality-specific metrics, region grounding, abstention and accessibility.
Mandatory prerequisites
M02-T02, M03-T03
Recommended prerequisites
M04-T01, M08-T02
Conditional prerequisites
M08-T02 — required for the retrieval-backed visual citation extension
Implementation readiness
Use supplied accessible charts and evidence regions. No vision model is needed for replay.
Demonstration
Score answers to fictional fee charts using annotated cells, regions and text references.
Exercise and assessment
Identify unsupported claims and report errors by modality and extraction stage.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Existing files
—
Audit findings
—
Reuse strategy
Adapt claim and citation scoring to image regions.
Teaching and experiment gaps
Add accessible synthetic charts, labeled regions and verification examples.
Technical corrections
—
M16-T03 Long-context evidence and memory New · 6h

Expert · Elective

Link to topic
Why it matters
More context can degrade evidence use or change which policy applies.
Concepts
Position sensitivity, interference, distractors, multi-document synthesis, recency, context budget and memory consistency.
Mandatory prerequisites
M01-T01, M07-T01
Recommended prerequisites
M07-T03, M08-T03
Conditional prerequisites
M08-T03 — required when experimenting with reranked retrieval context
Implementation readiness
Use P02 and P04 to inspect saved context variants.
Demonstration
Use saved prompts with the same evidence in different positions and label unsupported conclusions.
Exercise and assessment
Report controlled position and context-length effects across slices.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Existing files
—
Audit findings
—
Reuse strategy
Reuse the RAG evidence and oracle-control structure.
Teaching and experiment gaps
Add controlled contexts, staged difficulty and held-out document families.
Technical corrections
—
M17 Specialized outputs and coordinated systems 3 topics · 18h

Expert · Elective · Specialization

Description
Verify outcomes for typed extraction, generated code and collaboration between agents.
Objectives
Choose precise task outcomes and analyze errors hidden by generic text scores.
Labs
Apply the case contract to one specialization and verify outcomes with deterministic fixtures.
Competencies
Submit a domain-specific failure taxonomy and reproducible scorecard against examples that defeat a generic judge.
Completion
Submit outcome fixtures and a failure taxonomy for the specialized output or workflow. Replay at least one hard negative per topic and explain why a generic judge misses it. Missing field checks, unisolated code execution or unverifiable final state fails.
Theory (hrs)
4h 30m
Guided (hrs)
6h
Experiment (hrs)
4h 30m
Exercises (hrs)
3h
Learner total (hrs)
18h
Instructor-led (hrs)
12h
M17-T01 Structured extraction and classification New · 6h

Expert · Elective

Link to topic
Why it matters
A document can produce correct fields with wrong entities or invalid records.
Concepts
Field and record scoring, strict/normalized matches, null/unknown, span alignment, type and value checks, confidence and per-class error.
Mandatory prerequisites
M03-T02, M05-T01
Recommended prerequisites
M04-T02, M07-T01
Conditional prerequisites
—
Implementation readiness
Use P02, P04 and P06 to compute field and record scores.
Demonstration
Score synthetic form extraction against typed fields and entity IDs with Python.
Exercise and assessment
Explain how micro F1 hides a critical rare-field miss and add a matching gate.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Existing files
—
Audit findings
—
Reuse strategy
Adapt beginner schema checks and proposed annotation contract.
Teaching and experiment gaps
Add forms, field-level answer key and negative controls.
Technical corrections
—
M17-T02 Code-generation evaluation New · 6h

Expert · Elective

Link to topic
Why it matters
Generated code needs evidence from compilation, behavior, safety and resource bounds.
Concepts
Functional tests, cases and hidden tests, pass@k intuition, runtime/error categorization, resource controls, offline isolation and static review.
Mandatory prerequisites
M03-T02, M07-T01
Recommended prerequisites
M11-T03
Conditional prerequisites
—
Implementation readiness
Use P02, P03 and P06 to inspect fixed outcomes for benign pure functions. Use a prepared isolated runner for optional execution.
Demonstration
Evaluate canned benign pure functions against fixed inputs and captured results in a restricted teaching runner.
Exercise and assessment
Distinguish a compilation failure, wrong answer and timeout and interpret pass@k assumptions.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Existing files
—
Audit findings
—
Reuse strategy
Reuse typed outcomes and locked holdout principles.
Teaching and experiment gaps
Add harmless functions, run-result fixtures and isolated test harness design.
Technical corrections
—
M17-T03 Multi-agent coordination New · 6h

Expert · Elective

Link to topic
Why it matters
A successful individual reply can hide lost handoffs or inconsistent shared state.
Concepts
Role boundaries, coordination and handoff metrics, shared state, conflict resolution, redundancy, team-level success and coordination overhead.
Mandatory prerequisites
M10-T02
Recommended prerequisites
M10-T03, M11-T03
Conditional prerequisites
—
Implementation readiness
Use P04 to trace fictional handoff events and verify shared state.
Demonstration
Replay a mock support handoff where one agent omits a required state field. Compare with a valid delegated workflow.
Exercise and assessment
Verify global invariants, handoff quality and team cost from saved events.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Existing files
—
Audit findings
C06
Reuse strategy
Extend the single-agent trace schema with actor and handoff IDs.
Teaching and experiment gaps
Add coordination and conflicting-state fixtures with outcome labels.
Technical corrections
—
M18 Evaluator robustness and evaluation research 3 topics · 19h

Expert · Elective · Specialization

Description
Keep scores meaningful as evaluators, datasets and deployed models change.
Objectives
Stress-test evaluators, manage model migration, identify contamination risk and assess emerging methods.
Labs
Audit an evaluator and recommend a replacement strategy using anchors, holdout tests and monitored deployment evidence.
Competencies
Deliver a reproducible audit with known limits and an evidence-based tool or research recommendation.
Completion
Submit a method audit and migration or replacement memo. Verify one controlled anchor result, one uncertainty claim and a preserved holdout. Unchecked judge agreement or repeated tuning on the evaluation holdout fails.
Theory (hrs)
5h 30m
Guided (hrs)
5h 30m
Experiment (hrs)
5h
Exercises (hrs)
3h
Learner total (hrs)
19h
Instructor-led (hrs)
12h 30m
M18-T01 Advanced judge design and evaluator robustness Needs Improvement · 6h 30m

Expert · Elective

Link to topic
Why it matters
Judge ensembles and better prompts remain vulnerable to common correlated errors.
Concepts
Evidence-conditioned judgments, multi-judge agreement versus accuracy, selective review, style perturbations, evaluator drift and defensive handling of untrusted inputs.
Mandatory prerequisites
M06-T02
Recommended prerequisites
M06-T03, M12-T03
Conditional prerequisites
M06-T03 — required for the pairwise ensemble variant
Implementation readiness
Use P02 and P06 to analyze stored judge outputs.
Demonstration
Compare judge configurations on fixed ambiguous and transformed answers, then estimate reliability and review rates.
Exercise and assessment
Justify a human escalation boundary and explain one shared judge blind spot.
Theory (hrs)
2h
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h 30m
Instructor-led (hrs)
4h 30m

Repository mapping

Existing files
docs/test7.html, docs/test13.html, phase-5-production-eval/trajectory-evaluation/results/sample_trajectory_evaluation_results.json
Audit findings
C07,C12,S04,S05
Reuse strategy
Retain the false-confidence examples as hard negatives.
Teaching and experiment gaps
Add independent anchors, repeat tests and disagreement/error review.
Technical corrections
Validate score consistency and delimit evidence and input roles.
M18-T02 Benchmark contamination and model migration New · 6h

Expert · Elective

Link to topic
Why it matters
Benchmarks can overstate capability and model swaps can change outputs, costs and judge behavior simultaneously.
Concepts
Time/source holdouts, contamination and benchmark overfitting, matched migration suites, rating comparability, shadowing and rollback.
Mandatory prerequisites
M04-T03, M07-T03
Recommended prerequisites
M13-T03
Conditional prerequisites
M13-T03 — required for the live rollout governance extension
Implementation readiness
Use P04 and P06 and compare fixed public-style and private fictional data.
Demonstration
Compare public-style versus private synthetic holdouts, then simulate migration with old/new model and fixed/judge variants.
Exercise and assessment
Identify confounds and defend a migration plan for rare and severe slices.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h

Repository mapping

Existing files
—
Audit findings
C11,V03
Reuse strategy
Reuse the benchmark example and release packet structure.
Teaching and experiment gaps
Add sealed slices, contamination indicators and migration fixtures.
Technical corrections
—
M18-T03 Emerging evaluation methods and research literacy New · 6h 30m

Expert · Elective

Link to topic
Why it matters
Evaluation methods need a testable hypothesis and comparisons against simpler baselines.
Concepts
Active evaluation and adaptive sampling, preference modeling, redacted behavioral simulations, living benchmarks, measure validity and reproducible method reviews.
Mandatory prerequisites
M04-T03, M07-T02
Recommended prerequisites
M06-T03, M14-T02
Conditional prerequisites
M06-T03 — required for a preference-ranking method review
Implementation readiness
Use P02 and P06 for a simulated adaptive sampling comparison.
Demonstration
Simulate risk-aware active case selection, then compare estimates and label costs with uniform sampling.
Exercise and assessment
Write a method card with construct, assumptions, evidence limits and a replication plan.
Theory (hrs)
2h
Guided (hrs)
1h 30m
Experiment (hrs)
2h
Exercises (hrs)
1h
Learner total (hrs)
6h 30m
Instructor-led (hrs)
4h

Repository mapping

Existing files
—
Audit findings
V04
Reuse strategy
Reuse the evaluation contract and comparison harness.
Teaching and experiment gaps
Add method review template, simulation data and a replication exercise.
Technical corrections
—

Learning Roadmap

About the Course

Vision and audience

A long-term reference for learners entering LLM evaluation, engineers owning evaluation decisions, and instructors teaching or adapting practical labs. By the end of the core path, a learner should be able to design a versioned evaluation, build and calibrate scorers, quantify uncertainty, diagnose RAG and agent failures, and defend a production release decision. Expertise includes knowing when the evidence warrants another experiment or human review.

Start with concepts and explicit decision criteria, then build a small Python scorer. Use frameworks once learners can explain the metric and verify a result by hand. Study RAG and agents through observable inputs, outputs and states. Finish by connecting offline evaluations to release decisions, traces and monitored outcomes. The plan describes target competencies and future labs. The coverage statuses describe the current repository.

Sequence and time estimates

Core progression spans Beginner, Intermediate, Advanced and Production. Specialized / Expert modules are electives that can be studied independently after their listed prerequisites. Topic records give the minimum preparation for focused study. The recommended module order provides a complete beginner route. Required study for an individual module is computed from the prerequisites of its topics. Each stage ends with a reviewable artifact, and the capstone brings those artifacts together.

Times are planning estimates in hours. Each topic lists theory, guided hands-on implementation, independent experiments and interpretation, exercises and assessment, total learner time, and instructor-led classroom time. Instructor-led hours are a separate delivery estimate for explanation, demonstration and supervised practice. Preparation, breaks, and optional provider work are outside that estimate. Plan extra preparation for a first teaching run.

Estimates assume a learner who can use a terminal and read simple Python, coaching available for the first lab, and prepared fixtures with expected outputs. Guided times include routine setup and troubleshooting for the activity. Learners new to Python should practice reading a JSON file, looping over records, and running a test before starting guided implementation. Large downloads, hardware delays, provider wait times, deep remediation and elective live extensions vary by environment. Pilot time estimates with both a beginner and an experienced engineer.

One evolving application

  • Beginner — Evaluate a fictional bank support chatbot from six saved responses. Build exact/semantic boundary examples, a deterministic scorer and a blind human review sheet. Explain failure, uncertainty and execution error.
  • Intermediate — Publish a versioned gold dataset, calibrate a judge on frozen examples and compare prompt versions with paired Python analysis. Verify labels, denominators and uncertainty before drawing a conclusion.
  • Advanced RAG — Give the chatbot a fictional policy corpus. Evaluate ranked passages, context sufficiency, claims and citations, then compare retrieval changes against an oracle reference.
  • Advanced agents and safety — Add a mock account lookup and a simulated workflow. Verify actions, arguments, state, recovery and safe refusals. Assess defensive controls from fixed fictional traces and inert records.
  • Production — Add fixture-driven gates, trace metrics and a simulated incident. Convert a reviewed failure into a regression fixture and defend a release, hold or rollback decision.
  • Electives — Change the task when the method needs different evidence. Use synthetic fee charts for multimodal evaluation, public-style and private fictional benchmarks for contamination lessons, and benign generated functions for test-driven code evaluation. Use the healthcare examples for transfer lessons on score validity and source grounding.

Execution modes and tool choices

Every essential lesson starts with a fixture or simulated run that needs no model, network or credentials. Live Ollama inference and locally cached embeddings are optional hands-on extensions. The existing Test 5 configuration points to OpenAI and the existing Test 10 RAGAS integration requires OpenAI calls. These examples are references until local or fixture alternatives and effective settings are verified. Hosted graders and paid comparisons are optional, explicitly budgeted extensions.

Choose DeepEval for Python-oriented assertions and thresholds when its chosen metric fits the construct. Choose RAGAS for a RAG metric suite when the record carries its required references or evidence. Both may call judges or embeddings and need versioned adapters, explicit failure states and checked denominator behavior. Choose promptfoo for a declarative prompt and provider matrix, and OpenAI Evals for an OpenAI-centered managed grading workflow when supported graders, data handling and cost fit. Compare both on saved cases before adding a managed run.

Choose Langfuse for linked LLM traces, observations and scores. It depends on the quality of instrumented events and attached scores. OpenTelemetry is useful when correlation across services is the objective, with separate application quality measures. In defensive evaluation, Garak illustrates scanner-style coverage and PyRIT illustrates organized assessment records. Use prepared offline reports to compare their evidence formats and coverage. Tool output depends on scope, fixtures and labels, and requires review before it informs a decision.

Course time breakdown

Course hours by track
StrandModulesTopicsTheory (hrs)Guided (hrs)Experiment (hrs)Exercises (hrs)Learner total (hrs)Instructor-led (hrs)
Core154460h 30m74h60h43h 30m238h158h
Elective3914h 30m17h14h9h54h 30m36h
All185375h91h74h52h 30m292h 30m194h

Resources

Published examples and downloads · Concepts reference

Instructor Resources

Existing Repository Coverage

Reading the mapping

The course audit in docs/COURSE_AUDIT_REPORT.md records evidence and impact. Each topic lists its current status, source files, relevant findings, suggested reuse, missing teaching and experiments, and corrections for a future implementation phase. Reusable means a substantial part already supports the topic. Needs Improvement indicates useful but incomplete material. Needs Correction identifies a defect affecting the intended lesson. New indicates absent substantive coverage. Not Assessed is reserved for insufficient evidence. Validation items beginning with V require further investigation. Current HTML/JSON reports illustrate past runs, and the teaching plan calls for separate answer keys.

Existing lesson crosswalk

  • Reusable 2
  • Needs Improvement 17
  • Needs Correction 14
  • New 20
  • Not Assessed 0
Coverage by topic
TopicStatusExisting filesAudit findings
M01-T01 — Tokens, messages and context windowsNeeds Improvementdocs/test1.html, model_client.pyS08
M01-T02 — Inference, sampling and repeatabilityNeeds Correctiondocs/test8.html, phase-3-what-you-are-evaluating/temperature-sensitivity/temperature_evaluator.pyC08,V02
M01-T03 — Application architecture and a first smoke testReusabledocs/test1.html, phase-1-first-experiments/test1_basic_test.py, config.yamlC11,S06,S08,V02
M02-T01 — Objectives, dimensions and failure taxonomyNeeds Improvementdocs/concepts.html, docs/test0.htmlC05,S02
M02-T02 — Ground truth, references and benchmarksReusabledocs/test0.html, phase-0-foundations/explore_squad.py, docs/concepts.htmlS06,V03
M02-T03 — Offline, online and the evaluation lifecycleNeeds Improvementdocs/test14.html, README.mdV04
M03-T01 — Exact match, normalization and rulesNeeds Correctiondocs/test0.html, phase-0-foundations/phase_eval.py, docs/test4.html, phase-2-evaluation-techniques/banking-bots/banking_evaluator.pyC05,S02,S09
M03-T02 — Structured output and schema validationNeeds Improvementphase-5-production-eval/agent-evaluation/agent.py, docs/test12.html, phase-2-evaluation-techniques/groundedness/groundedness_evaluator.pyS04,S05
M03-T03 — Rubrics and a first human evaluationNeeds Correctiondocs/test2.html, phase-1-first-experiments/test2_score_evaluations.py, phase-1-first-experiments/results/scores.jsonS01,C01
M04-T01 — Golden schemas, provenance and versioningNeeds Improvementphase-2-evaluation-techniques/banking-bots/banking_test_cases.xlsx, phase-3-what-you-are-evaluating/rag-evaluation/rag_test_cases.xlsxS09,S11,S15,S16,V03
M04-T02 — Annotation, adjudication and synthetic dataNeeds Improvementdocs/test4.html, phase-5-production-eval/langfuse-observability/create_test_cases.py, phase-2-evaluation-techniques/llm-as-a-judge/banking_bot_test_results.xlsxC01,V03
M04-T03 — Sampling, holdouts, leakage and maintenanceNew—V03
M05-T01 — Token F1, BLEU, ROUGE and reference metricsNeeds Correctiondocs/test3.html, phase-2-evaluation-techniques/test3_bleu_score_evaluation.pyC10,S06,S08
M05-T02 — Embeddings and semantic similarityNeeds Correctiondocs/test6.html, phase-2-evaluation-techniques/semantic-similarity/test6_semantic_similarity.py, phase-2-evaluation-techniques/semantic-similarity/healthcare_test_cases.xlsxC09,S02,S11,S16
M05-T03 — Combining evaluators and handling uncertaintyNeeds Improvementdocs/test6.html, docs/test9.html, docs/test13.htmlC02,C05,C09,S02
M06-T01 — Pointwise LLM-as-a-JudgeNeeds Correctiondocs/test5.html, phase-2-evaluation-techniques/llm-as-a-judge/llm_judge_evaluator.py, phase-2-evaluation-techniques/llm-as-a-judge/banking_bot_test_results.xlsx, config.yamlC01,S04,S07,S08,S09
M06-T02 — Human agreement, calibration and biasNeeds Improvementdocs/test5.html, docs/test7.html, docs/test13.htmlC01,C07,C12,S09,V03
M06-T03 — Pairwise comparisons and rankingNew——
M07-T01 — Variance, repetition and score aggregationNeeds Correctiondocs/test2.html, phase-1-first-experiments/test2_full_evaluations.py, docs/test8.html, phase-3-what-you-are-evaluating/temperature-sensitivity/temperature_evaluator.pyC08,C11,S02,S08,V04
M07-T02 — Confidence intervals, significance and sample sizeNew——
M07-T03 — Comparing models, prompts and A/B variantsNeeds Improvementdocs/test8.html, docs/test11.htmlC04,C08,C11,V04
M08-T01 — Ingestion, chunking and ranked retrievalNeeds Improvementdocs/test9.html, phase-3-what-you-are-evaluating/rag-evaluation/chunker.py, phase-3-what-you-are-evaluating/rag-evaluation/retriever.py, phase-3-what-you-are-evaluating/rag-evaluation/chunks.json, phase-3-what-you-are-evaluating/rag-evaluation/rag_test_cases.xlsxC02
M08-T02 — Context sufficiency, answer correctness and citationsNeeds Correctiondocs/test7.html, phase-2-evaluation-techniques/groundedness/groundedness_evaluator.py, docs/test9.html, phase-3-what-you-are-evaluating/rag-evaluation/rag_evaluator.pyC02,C03,C12,S04,S09
M08-T03 — Failure attribution, optimization and RAG regressionNeeds Improvementdocs/test9.html, phase-3-what-you-are-evaluating/rag-evaluation/rag_test_cases.xlsx, docs/test11.htmlC02,C04,S09
M09-T01 — Common evaluator contract and framework selectionNeeds Improvementdocs/test11.html, requirements.txt, config.yamlC04,S06,S08,S15,V02
M09-T02 — DeepEval and RAGAS on frozen RAG recordsNeeds Correctiondocs/test11.html, phase-4-frameworks-and-tools/deepeval-evaluation/test11_deepeval_evaluator.py, phase-4-frameworks-and-tools/ragas-evaluation/test10_ragas_evaluator.py, phase-4-frameworks-and-tools/ragas-evaluation/ragas_test_cases.xlsxC03,C04,S03,S08,S15,V01
M09-T03 — promptfoo and OpenAI EvalsNew——
M10-T01 — Execution model, structured calls and tool contractsNeeds Correctiondocs/test12.html, phase-5-production-eval/agent-evaluation/agent.py, phase-5-production-eval/agent-evaluation/tools.py, phase-5-production-eval/agent-evaluation/test12_agent_evaluator.py, phase-5-production-eval/agent-evaluation/agent_test_cases.xlsxC06,S05,S13
M10-T02 — Trajectories, planning outcomes and final stateNeeds Correctiondocs/test12.html, docs/test13.html, phase-5-production-eval/trajectory-evaluation/test13_trajectory_evaluator.py, phase-5-production-eval/trajectory-evaluation/sample_agent_evaluation_results.json, phase-5-production-eval/trajectory-evaluation/results/sample_trajectory_evaluation_results.jsonC06,C07,S05
M10-T03 — Recovery, reliability and efficiencyNeeds Improvementdocs/test12.html, phase-5-production-eval/agent-evaluation/agent.py, phase-5-production-eval/agent-evaluation/test12_agent_evaluator.pyS02,S05,S13
M11-T01 — Risk taxonomy, harmful content and refusalNeeds Correctiondocs/test4.html, docs/test5.html, docs/test7.html, docs/test11.htmlC03,C05,C12
M11-T02 — Bias, fairness and privacy controlsNew—S12
M11-T03 — Robustness, safety benchmarks and regressionNeeds Improvementdocs/test4.html, docs/test8.html, docs/test9.htmlC08,C13
M12-T01 — Threat modeling, scope and test designNeeds Improvementdocs/concepts.html#prompt-injection, docs/test4.html, docs/test9.htmlC13
M12-T02 — Defensive checks for RAG, agents and data exposureNew—C13
M12-T03 — Mitigation verification, reporting and defensive toolingNew—C13
M13-T01 — Evaluation pipelines and versioned run contractsNeeds Correctionconfig.yaml, model_client.py, requirements.txt, phase-3-what-you-are-evaluating/rag-evaluation/report_generator.pyS02,S06,S07,S08,S10,S11,S14,S15
M13-T02 — CI/CD quality gates and regression evidenceNew—V04
M13-T03 — Release decisions, rollback and governanceNew—C13,V04
M14-T01 — Traces, monitoring and LangfuseNeeds Correctiondocs/test14.html, phase-5-production-eval/langfuse-observability/test14_langfuse_evaluator.py, phase-5-production-eval/langfuse-observability/docker-compose.ymlC05,S07,S08,S12,S16,V02
M14-T02 — Production sampling, human feedback, drift and alertsNew—S12,V04
M14-T03 — Cost, tail latency and operational reliabilityNeeds Improvementdocs/concepts.html, docs/test8.html, docs/test11.html, docs/test14.htmlC08,S03,V04
M15-T01 — Design review and reproducible suiteNew—V03,V04
M15-T02 — Decision defense, incident response and teachingNew—V04
M16-T01 — Multilingual and localized evaluationNew——
M16-T02 — Multimodal evaluationNew——
M16-T03 — Long-context evidence and memoryNew——
M17-T01 — Structured extraction and classificationNew——
M17-T02 — Code-generation evaluationNew——
M17-T03 — Multi-agent coordinationNew—C06
M18-T01 — Advanced judge design and evaluator robustnessNeeds Improvementdocs/test7.html, docs/test13.html, phase-5-production-eval/trajectory-evaluation/results/sample_trajectory_evaluation_results.jsonC07,C12,S04,S05
M18-T02 — Benchmark contamination and model migrationNew—C11,V03
M18-T03 — Emerging evaluation methods and research literacyNew—V04
Teaching Plan

Mandatory lesson standard

  1. State observable learning objectives.
  2. Give prerequisites, setup, input artifacts and an entry checkpoint.
  3. Explain the concept in plain language.
  4. Explain why it exists.
  5. State the problem it solves and constraints on use.
  6. Give realistic engineering use cases.
  7. Work a small example by hand with a known answer.
  8. Increase complexity gradually, including a conflicting or ambiguous case.
  9. Give a step-by-step Python implementation or reproducible Python replay when the topic concerns a managed workflow.
  10. Explain code sections that change behavior, scores or denominators.
  11. Show expected artifacts and key execution results.
  12. Interpret results and what decisions they can support.
  13. Demonstrate common mistakes.
  14. Explain limitations and failure cases, including evaluator failure where relevant.
  15. Include a guided experiment with one controlled variable.
  16. Assign independent exercises with measurable outputs.
  17. Provide solutions or verification criteria, including accepted alternative solutions.
  18. Include instructor notes with timing, prediction prompts, common learner errors and offline fallback.
  19. Document troubleshooting and how to identify failure stage.
  20. Include knowledge checks and observable completion criteria.
  21. Connect the current topic to earlier and later evaluation decisions.

Classroom preparation and assessment

Each session should use the topic's prerequisite check and demonstrations, then ask learners to predict a result before running it. Prepare a small known-answer fixture, one common misconception and a harder case. Grade claims against evidence and review inclusion, decision policy and exceptions. Where output is stochastic, grade expected invariants and justified decisions. For the capstone, require an annotated dataset, manifest, replayable report, failure diagnosis and release recommendation. Ask a peer to reproduce the result.

At least one teaching variant for core labs should use standard-library Python and prepared data. Where a framework is the objective, provide a fixture containing its inputs and results and prepare a pinned optional environment. Instructors should try the first run on target classroom hardware and define bounds for compute, data handling and spend before optional live sessions.

Implementation sequence

  1. First milestone — Deliver the M01–M03 fixture-first slice, the starter M04-T01 schema and the first M06-T01 judge replay. Use exactly shared response IDs across the human, rule and judge evidence. Supply a handout, annotated answer key, instructor demonstration and local fixture verification before updating requirements for optional runs.
  2. Build the dataset and measurement spine in M04–M07. Establish reliable denominators, anchors, held-out cases, paired comparisons and uncertainty exercises before publishing example baselines.
  3. Add M08–M12 around the same contract. Reuse the banking corpus and traces. Verify gold passage and final-state controls before framework demos. Prepare safety and defensive-review fixtures for bounded, fictional labs.
  4. Add M13–M15 after the offline gates are stable. Introduce sampled synthetic traffic, redacted tracing and incident-to-regression review.
  5. Build electives once the relevant core artifacts are stable. Add tracks based on learner need and verify each against a separate domain-specific rubric.

Maintenance, versioning and regeneration

Curriculum revision 1.1.

Use stable module and topic IDs. Edit titles and passages in place. Allocate new IDs for new learning objectives and preserve old IDs in a deprecation record when retiring a topic. Increment the curriculum revision when changing expected competencies, prerequisites or coverage. Version datasets, prompts, model snapshots and evaluator definitions independently and link those versions from lesson materials.

Update the per-topic mapping whenever a lesson or experiment changes. Review all dependent topics when a prerequisite changes. A topic is ready for implementation sign-off when the lesson follows the standard, an instructor has replayed it, a learner can verify its output and the audit-linked corrections are resolved or documented. Re-estimate time after actual pilots, and record scope changes with the relevant IDs.

The Markdown is authoritative. Run python3 docs/generate_master_curriculum.py from the repository root with Python 3.10 or newer to regenerate the standalone HTML. Use python3 docs/generate_master_curriculum.py --check to verify the checked-in artifact matches exactly. Edit the structured Markdown fields and rerun the generator. Use explicit anchors for IDs, comma-separated mandatory and recommended topic links, and the fixed time column order. Use a link followed by an em dash and a named exercise for each conditional prerequisite, with slash-separated entries when needed. Update the Paths A–G table when a target or required dependency changes. The generator validates finding IDs against the audit when that file is present. The HTML is a portable view of the plan. Repository lesson links require the companion lesson files. Progress is stored in the browser for the current device and may depend on file URL and privacy settings.