Complete the core and extend evaluation to the expert domains.
For Learners seeking the whole curriculum plus specialized evaluation practice
What you’ll build Produce the core capstone and specialized evidence such as a field-level scorecard, coordinated workflow audit or migration assessment.
Prerequisites and checkpoint
Required foundations included See the ordered plan for the required topics.
Final checkpoint Reproduce the core capstone and defend the evidence and limits of the expert module submissions.
Quick Learning sequences the required topics for a specific goal. Complete Course follows the full core in order and offers elective topics separately. Both use the same topic material and completion checkpoints. A learning plan describes the intended exercises and links to the existing material where it is available.
Preparation / Getting Started
Allow 7 learner hours if all six readiness skills are new. Skip any row once you can demonstrate its checkpoint. Practice locally with synthetic files and saved responses. No model downloads or API credentials are needed. Preparation hours are separate from all path and curriculum totals.
Preparation skills
ID
Readiness skill
Learning objective and practice
Readiness checkpoint
Learner total (hrs)
P01
Files, folders and the terminal
Navigate a sample project with pwd, cd and a directory listing. Recognize absolute and relative paths and save into a new output file.
From the project root, locate a fixture and explain where a relative output path will be written.
1h
P02
Basic Python for evaluation
Read strings, numbers, lists and dictionaries. Use a loop, a conditional, a small function and an assertion to count outcomes.
Given three saved outcomes, return the number that pass and verify an empty input.
1h 30m
P03
Run Python scripts
Confirm the Python version, run a local script from the right directory, pass it a file path and read standard output and a traceback.
Run a supplied script and fix a deliberately mistyped input path.
1h
P04
JSON and JSONL
Read and write JSON objects and arrays. Read a JSONL file one record at a time with Python and distinguish syntax errors from missing fields.
Convert three sample cases to JSONL and identify the invalid line in a second fixture.
1h 30m
P05
API requests and responses
Read a saved HTTP request and response. Identify method, endpoint, headers, body, status and timeout. Keep sample credentials out of fixtures.
Explain the difference between a successful request with a wrong answer and an unsuccessful request.
1h
P06
Numerical reasoning
Calculate rates, percentages, means and percentage-point changes on a small score sheet. Identify the denominator and how an excluded row changes it.
Explain why 8 of 10 scored cases and 8 of 12 attempted cases make different claims.
A — Complete Beginner. Use the preparation checkpoints first, then follow the full core in curriculum order. Start at M01. The capstone verifies a reproducible release decision.
B — LLM Evaluation Fundamentals. Start with the objective and references, then learn deterministic and structured checks, human review and overlap metrics. Exit by scoring a small frozen set and explaining a false pass.
C — LLM-as-a-Judge Specialist. Complete the required reference, schema and rubric topics, then pointwise scoring, calibration and comparisons in M06. Extend with advanced judge design if needed. Embeddings and RAG can be skipped.
D — RAG Evaluation Specialist. Start with objectives and references, then work through M08. The ranked-list and claim exercises use supplied fixtures. Take embeddings or judge setup only for the extensions that require them.
E — Agent Evaluation Specialist. Learn JSON schema and evaluation objectives, then study M10. The required labs use mock tools and saved traces. Study retrieval only for the document-tool extension.
F — Production Evaluation Engineer. Follow the minimum route to release gates and production monitoring. The route includes versioned cases, sample design and uncertainty. RAG, agent and defensive assessment topics are optional for this path unless they apply to the release under review.
G — Complete Expert. Complete core and the electives. Choose the preparation rows you need and add their hours if applicable.
Path goals and checkpoints
Learning path goals
Path
Audience
Learning goal
Practical outcome
Final completion checkpoint
Suggested extensions
A
Learners seeking a structured foundation and a production portfolio piece
Design, run and defend an end-to-end evaluation.
Produce a versioned evaluator, replayable run, incident diagnosis and release recommendation.
Have a peer reproduce the scores and the incident regression, then defend the release decision with evidence.
—
B
New evaluators, QA engineers and developers adding a first evaluation
Translate an evaluation goal into checks, references and human review.
Build a small deterministic scorer and compare its results with human labels and overlap metrics.
Score a frozen set, distinguish execution errors from wrong answers and explain a false pass and a false fail.
Learners seeking the whole curriculum plus specialized evaluation practice
Complete the core and extend evaluation to the expert domains.
Produce the core capstone and specialized evidence such as a field-level scorecard, coordinated workflow audit or migration assessment.
Reproduce the core capstone and defend the evidence and limits of the expert module submissions.
—
Using the dependency roadmap
Open Learning Roadmap and select a module or topic by name or ID. The required route includes the target and every transitive mandatory topic, each counted once, in suggested study order. Selecting a module uses every topic in that module as a target. Recommended preparation adds context and may be skipped. A conditional link applies only to the exercise named beside it. Implementation readiness describes skills or tool setup you can demonstrate independently, so those notes and preparation hours sit outside the academic dependency graph. A dash means that category has no prerequisite.
M01-T01, M02-T01 and M03-T02 have no mandatory course prerequisites. Start with M01-T01 for a first look at model behavior, M02-T01 for evaluation design, or M03-T02 if you already know basic Python and JSON and want to validate structured outputs.
I want to learn RAG evaluation. Select M08 or Path D. Check the reference-answer prerequisites first. For an interpretation-only refresher, select M08-T02. Use the conditional notes before changing a dense encoder or adding a judge.
I already know Python and deterministic evaluation. Verify the preparation checkpoints and use Path B's fixture as a diagnostic. Mark familiar topics completed only when you can demonstrate their stated result. For structured data and provenance, enter at M04-T01 once its prerequisites are met.
I only want to learn LLM-as-a-Judge. Select M06-T01 for pointwise scoring, or Path C to add calibration and pairwise ranking. Use optional extensions as needed.
I want to become a complete evaluation engineer. Use Path A through the production capstone, then choose specialization topics or Path G. A short route establishes the advertised skills for its target. Course-wide competency requires the broader assessments and capstone.
Use the module overview to orient yourself and the topic view to inspect exact links. Arrows point to the topic or module that uses the prerequisite. The unlocked list uses your saved completion status and identifies topics that would become available by completing the current target. Use Export graph to share the full module or topic diagram, or choose a target or path and export its required route. Image downloads use the full graph for that scope regardless of pan and zoom. The standalone HTML download opens on the roadmap and supports offline interaction.
In Quick Learning, use Download Learning Plan to share an offline copy that opens on the selected path. Hosted path links use fragments such as #quick-d. For a local copy, share the downloaded HTML file. A local file address only works on the machine where that file is stored. Completion marks belong to the browser and are excluded from the shared plan.
Instructor preparation
Replay the chosen lab offline and grade a good submission and a borderline one. Verify the topic's requirements, exact fixtures, code and expected outcomes before teaching. Prepare a demonstration fallback for any live extension. Pilot the preparation checkpoints and required route timings with your class before planning further sessions.
Exercise and checkpoint Calculate observed violation rate by scenario and distinguish authorization failures from correct refusals and unavailable evidence.
Exercise and checkpoint Write a red-team finding and verification memo with coverage, confirmed failures, false refusals, remediation and remaining risk.
Exercise and checkpoint Calculate observed violation rate by scenario and distinguish authorization failures from correct refusals and unavailable evidence.
Exercise and checkpoint Write a red-team finding and verification memo with coverage, confirmed failures, false refusals, remediation and remaining risk.
Follow a request from user input to model output. Learn which settings and components can change the evidence an evaluator sees.
Objectives
Trace a generation request, explain output variability, and locate failure boundaries in a model-backed application.
Labs
Replay a fictional bank help request as plain text, then read its saved request and result envelopes.
Competencies
Label the request, model configuration, response and failure stage in a fresh replay.
Completion
Submit a labeled request/response trace. Mark message roles, missing context and one failure boundary. Compare your labels to the supplied replay key. A submission fails if a transport error is labeled a wrong model answer.
Theory (hrs)
3h
Guided (hrs)
4h
Experiment (hrs)
2h 30m
Exercises (hrs)
1h 30m
Learner total (hrs)
11h
Instructor-led (hrs)
7h 30m
M01-T01 Tokens, messages and context windowsNeeds Improvement · 3h 30m
Retain the smoke-test idea and worked terminal output.
Teaching and experiment gaps
Add setup preflight and a one-model replay option.
Technical corrections
Qualify timing and creativity observations and verify direct requirements before teaching live setup.
M02 Design the evaluation3 topics · 12h
Beginner · Core · Foundations
Description
Translate a user goal and potential harm into an evaluation contract with cases, acceptance criteria and decisions.
Objectives
Select an evaluation unit, define outcomes, and explain when to use offline and online evidence.
Labs
Draft a one-page evaluation contract for fictional fee and account-support questions.
Competencies
Given an ambiguous feature, specify intended behavior, cases, labels, failure categories and a decision owner.
Completion
Submit an evaluation contract with a unit, two quality dimensions, expected behavior and a decision owner. Match one reference and one unanswerable case to the contract in a peer walkthrough. A score with no decision or acceptance criterion fails.
Theory (hrs)
3h
Guided (hrs)
3h 30m
Experiment (hrs)
3h
Exercises (hrs)
2h 30m
Learner total (hrs)
12h
Instructor-led (hrs)
8h
M02-T01 Objectives, dimensions and failure taxonomyNeeds Improvement · 4h 30m
Add a lifecycle diagram, evidence worksheet and decision scenarios.
Technical corrections
Describe offline traces and causal online experiments as separate evidence.
M03 Deterministic checks and first human review3 topics · 13h 30m
Beginner · Core · Techniques
Description
Build the first evaluator from frozen outputs and learn where human judgment adds evidence.
Objectives
Implement stable assertions, validate structured data and explain a disagreement using a rubric.
Labs
Score one fixture containing a correct answer, good paraphrase, incorrect close answer, correct abstention, over-refusal and execution error.
Competencies
Submit a runnable fixture scorer and a reviewed answer sheet that identifies false passes and false fails.
Completion
Submit a Python scorer, frozen known-answer cases and a blind rating sheet. Reproduce the expected pass, fail and unscorable outcomes and explain a substring false pass. Run the answer key and inspect one rubric disagreement. A parser or transport error counted as a model failure fails.
Theory (hrs)
3h
Guided (hrs)
4h 30m
Experiment (hrs)
3h
Exercises (hrs)
3h
Learner total (hrs)
13h 30m
Instructor-led (hrs)
9h
M03-T01 Exact match, normalization and rulesNeeds Correction · 4h 30m
Retain the capture/score split and human-first review.
Teaching and experiment gaps
Supply canonical full responses, annotations and scoring key.
Technical corrections
Blind model labels, show full output and save each decision with its source response ID.
Intermediate4 modules · 66h 30m
M04 Evaluation dataset engineering3 topics · 16h
Intermediate · Core · Dataset Engineering
Description
Build data with documented coverage, adjudicated labels, and a defensible separation between development and final evaluation.
Objectives
Design a golden dataset schema, curate inputs and labels, and manage sampling and updates.
Labs
Publish a versioned miniature banking dataset with a manifest, slices and isolated holdout.
Competencies
Another learner can validate the schema, reproduce the split and trace each label to evidence.
Completion
Submit a validated JSONL dataset with a manifest, split and adjudication note. Reproduce the split and trace a label to evidence. Check unique IDs and group isolation. Shared source groups across the holdout or missing provenance fail.
Theory (hrs)
4h
Guided (hrs)
5h
Experiment (hrs)
4h
Exercises (hrs)
3h
Learner total (hrs)
16h
Instructor-led (hrs)
10h 30m
M04-T01 Golden schemas, provenance and versioningNeeds Improvement · 4h 30m
Synthetic breadth is useful when human review and source-backed validation preserve label quality.
Concepts
Annotation instructions, independent labels, disagreement and arbitration, structured generation from policy, review and rejection, provenance and license.
Use several fast scorers on the same answers and investigate divergent results before setting a threshold.
Objectives
Compute overlap and semantic similarity, compare error patterns, and design a hybrid decision rule.
Labs
Build a Python scorecard for factual banking answers, paraphrases and unsupported claims.
Competencies
Explain a false positive and false negative for each proxy and defend a held-out decision rule.
Completion
Submit an overlap, embedding and hybrid scorecard with a held-out decision rule. Recompute a hand-scored case and demonstrate a valid paraphrase and a high-overlap wrong answer. An unqualified claim that cosine proves correctness fails.
Surface overlap offers a useful baseline for bounded tasks and can penalize correct paraphrases.
Concepts
Token precision/recall/F1, n-grams, BLEU brevity and smoothing, sentence versus corpus BLEU, ROUGE-N and ROUGE-L, multiple references and optional chrF.
Reuse proxy disagreement and two-layer scoring examples.
Teaching and experiment gaps
Add a decision table, N/A controls and escalation cost.
Technical corrections
Keep hard constraints visible and avoid double-counting correlated evidence.
M06 Model judges and comparative evaluation3 topics · 17h
Intermediate · Core · Judges
Description
Validate judge evidence with human labels before delegating consequential decisions to an evaluator.
Objectives
Build pointwise and comparative rubrics, measure inter-rater agreement and diagnose judge bias.
Labs
Evaluate identical, frozen responses with human raters, a local or replayed judge and a deterministic baseline.
Competencies
Produce a held-out judge report with valid verdict coverage, per-class errors, agreement and bias checks.
Completion
Submit a frozen-response judge report with coverage, class errors, agreement and pairwise order checks. Match human and model verdicts by response ID and replicate a swapped pair. Treating malformed verdicts as substantive agreement or comparing different responses fails.
Compare two fictional banking configurations using repeated responses, case-level aggregation and Python resampling.
Competencies
Reproduce the report from a manifest and reach a defensible improvement, regression or inconclusive decision.
Completion
Submit a reproducible paired comparison with case counts, effect size, interval and decision. Rerun from the manifest and explain an inconclusive result. Resampling repeated draws as independent cases or reporting only a point estimate fails.
Theory (hrs)
5h
Guided (hrs)
5h 30m
Experiment (hrs)
4h 30m
Exercises (hrs)
3h
Learner total (hrs)
18h
Instructor-led (hrs)
12h
M07-T01 Variance, repetition and score aggregationNeeds Correction · 5h 30m
Keep the comparative examples as discussion prompts.
Teaching and experiment gaps
Add controlled one-variable tests and simulated online outcomes.
Technical corrections
Match inputs and judge or attribute changes explicitly.
Advanced5 modules · 87h
M08 RAG system evaluation3 topics · 19h 30m
Advanced · Core · RAG
Description
Evaluate retrieval and generation independently before judging end-to-end answers.
Objectives
Measure ranked retrieval, score claims and citations, and isolate the failing pipeline stage.
Labs
Reuse fictional policy guides and create one labeled replay from query through citations.
Competencies
Diagnose a held-out failure and verify a targeted change against retrieval and answer metrics.
Completion
Submit a labeled retrieval and claim report with a citation check and one diagnosis. Verify a hand-computed rank metric, then repeat a controlled oracle comparison. A report that hides a wrong answer behind an average or omits support for a citation fails.
Theory (hrs)
5h 30m
Guided (hrs)
6h
Experiment (hrs)
5h
Exercises (hrs)
3h
Learner total (hrs)
19h 30m
Instructor-led (hrs)
13h 30m
M08-T01 Ingestion, chunking and ranked retrievalNeeds Improvement · 6h 30m
Keep the TC21 and retrieval-confusion walkthroughs.
Teaching and experiment gaps
Add oracle controls, fixed generator outputs and before/after regression fixtures.
Technical corrections
Use answer/claim outcomes and correct sample denominators.
M09 Framework adapters and tool selection3 topics · 15h 30m
Advanced · Core · Frameworks
Description
Apply commercial and open tooling through a neutral evaluation contract and compare definitions on matched inputs.
Objectives
Map case objects and metrics, explain assumptions and select an adapter based on evidence and operational needs.
Labs
Use one fixture set to compare a hand-computed scorer and two adapter result shapes.
Competencies
Show per-case parity where metrics match and explain one intentional divergence.
Completion
Submit one neutral case mapped to two adapter outputs and a tool-choice memo. Verify one matched metric against a hand calculation and explain a divergence or N/A. Comparing incompatible scores as parity or discarding adapter errors fails.
Theory (hrs)
4h
Guided (hrs)
5h
Experiment (hrs)
3h 30m
Exercises (hrs)
3h
Learner total (hrs)
15h 30m
Instructor-led (hrs)
10h 30m
M09-T01 Common evaluator contract and framework selectionNeeds Improvement · 4h 30m
Portable contracts keep experiments reproducible when a framework or judge changes.
Concepts
Case/response IDs, metric intent and input schema, results with valid/error/N/A, provenance, offline runner versus hosted service, cost, telemetry and capability fit.
Named metrics can answer different questions and require different labels or model calls.
Concepts
DeepEval test cases, contextual relevance, faithfulness and thresholds. RAGAS variants for context precision, faithfulness and response relevance, reference requirements and judge/embedding calls. Choose DeepEval for Python assertions and RAGAS for retrieval/generation metric suites when prerequisites match.
Declarative and hosted runs can support review workflows once score meaning is understood.
Concepts
promptfoo for declarative prompt/provider matrices and assertions. OpenAI Evals for evaluation definitions, datasets and grader/runs in an OpenAI-centered workflow. Compare portability, supported graders, spend, data visibility and versioning.
Use P03 and P04 to inspect declarative specs. The managed variant uses P05 only if explicitly enabled.
Demonstration
Translate two frozen prompts and one deterministic grading rule into declarative and hosted-style evaluation specs. Inspect mock results locally.
Exercise and assessment
Recommend a tool for a CI matrix and another for a team using managed grading, with limits and a fixture fallback.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h
Exercises (hrs)
1h
Learner total (hrs)
5h
Instructor-led (hrs)
3h 30m
Repository mapping
Existing files
—
Audit findings
—
Reuse strategy
Reuse the common contract rather than duplicating metrics.
Teaching and experiment gaps
Add a local matrix lab and an offline mock of the hosted workflow.
Technical corrections
—
M10 Agent and workflow evaluation3 topics · 18h
Advanced · Core · Agents
Description
Evaluate what a workflow does, what state it leaves behind and how it recovers under controlled failures.
Objectives
Measure calls, arguments, outputs, observable planning, task completion and recovery within budgets.
Labs
Replay frozen traces for fictional account and policy tasks against mock tool results and state.
Competencies
Score alternative valid paths and reject a plausible answer that leaves an incorrect or unauthorized state.
Completion
Submit a trace replay scorecard covering arguments, tool results, termination and final state. Pass two equivalent valid paths and reject a wrong-state trace with a plausible answer. An exact-path check alone or an unexplained duplicate action fails.
Reuse the practical timeout discussion with mock faults.
Teaching and experiment gaps
Add state snapshots, failure injection, latency and token trace fields.
Technical corrections
Separate infrastructure errors and test idempotency before discussing writes.
M11 Safety and responsible AI evaluation3 topics · 17h
Advanced · Core · Safety
Description
Establish a safety measurement plan that covers risky outcomes and permitted use.
Objectives
Classify risk, evaluate refusal and over-refusal, audit fairness and privacy, and verify safety regressions.
Labs
Score a synthetic safety set against a fictional bank policy and document uncertain labels.
Competencies
Produce a risk matrix, slice report and explicit release criteria with review paths for severe or ambiguous cases.
Completion
Submit a risk matrix and scorecard with labeled appropriate refusals, over-refusals and one matched subgroup check. Recompute counts and assess a severe case. An undisaggregated mean or unspecified policy boundary fails.
A global metric can hide subgroup harms, and observability can increase unnecessary data exposure.
Concepts
Counterfactual and matched-slice tests, label and population bias, disparate error rates, consent, data minimization and synthetic sensitive-data checks.
Add fixed controls, independent labels and reproducible safety regression cases.
Technical corrections
Report independent case counts and preserve critical slices.
M12 Defensive red-team evaluation3 topics · 17h
Advanced · Core · Red Teaming
Description
Plan and assess authorized defensive tests of isolated fictional systems using fixed benign scenarios, synthetic records and mock tools.
Objectives
Identify trust boundaries, measure observed policy violations and verify mitigations on matched evidence.
Labs
Evaluate canned outcomes from a fictional bank with document and tool trust boundaries, then write a mitigation regression report.
Competencies
Explain what was measured, what was in scope, which fixes held and what evidence remains uncertain.
Completion
Submit a defensive assessment memo scoped to the fictional system. Verify control and mitigation outcomes on fixed, harmless fixtures and report match counts and false refusals. Unscoped testing or an unverified mitigation claim fails.
Theory (hrs)
4h 30m
Guided (hrs)
5h
Experiment (hrs)
4h 30m
Exercises (hrs)
3h
Learner total (hrs)
17h
Instructor-led (hrs)
11h
M12-T01 Threat modeling, scope and test designNeeds Improvement · 5h 30m
A clear threat model links input channels and assets to the defensive behavior expected of the system.
Concepts
Direct and indirect prompt injection as threat categories, jailbreaks, protected assets, trust boundaries, authorized test scope, containment and harmless fixtures.
M10-T01 — required when independently interpreting raw mock-tool arguments in the agent extension
Implementation readiness
Use P04 and P06 to read stored synthetic outcomes and calculate rates on supplied controls.
Demonstration
Compare saved control-off and control-on records from an isolated mock system. Check the simulated authorization decision and whether a dummy marker reached the output.
Exercise and assessment
Calculate observed violation rate by scenario and distinguish authorization failures from correct refusals and unavailable evidence.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h
Repository mapping
Existing files
—
Audit findings
C13
Reuse strategy
Carry forward mock states and separate answer/authorization labels from M10.
Teaching and experiment gaps
Add canned before/after traces, benign marker receipts and a scoring key.
Technical corrections
—
M12-T03 Mitigation verification, reporting and defensive toolingNew · 5h 30m
A change needs independent evidence on fixed cases, unseen controls and ordinary permitted behavior.
Concepts
Re-test protocols, severity and reproducible evidence, reporting and ownership. Garak and PyRIT report vocabulary and scope constraints. Static contract and replay checks for targeted controls.
Review fixed offline report excerpts. The lab uses synthetic evidence and does not run scanners.
Demonstration
Read prepared Garak and PyRIT report excerpts and reconcile their labels with a manually scored, held-out synthetic fixture.
Exercise and assessment
Write a red-team finding and verification memo with coverage, confirmed failures, false refusals, remediation and remaining risk.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
5h 30m
Instructor-led (hrs)
3h 30m
Repository mapping
Existing files
—
Audit findings
C13
Reuse strategy
Use offline report interpretation alongside the neutral contract.
Teaching and experiment gaps
Add fictional report exports, outcome labels and mitigation regression worksheet.
Technical corrections
—
Production3 modules · 48h
M13 Continuous evaluation and releases3 topics · 17h 30m
Advanced · Core · Production Engineering
Description
Turn evaluation evidence into reliable checks and release decisions while preserving diagnostic artifacts.
Objectives
Design a versioned pipeline, define CI/CD quality gates, and decide when to pause or roll back.
Labs
Turn the bank fixture suite into an offline CI run with immutable candidate artifacts and a release decision.
Competencies
Demonstrate pass, fail and insufficient evidence paths and replay the report from its manifest.
Completion
Submit a fixture-driven gate, a reproducible manifest and a release decision. Replay pass, fail and insufficient-evidence outcomes and explain a rollback trigger. Treating infrastructure errors as passes or allowing a critical failure through an aggregate fails.
Theory (hrs)
4h 30m
Guided (hrs)
5h 30m
Experiment (hrs)
4h 30m
Exercises (hrs)
3h
Learner total (hrs)
17h 30m
Instructor-led (hrs)
11h 30m
M13-T01 Evaluation pipelines and versioned run contractsNeeds Correction · 6h
A run must be attributable to one prompt, model, dataset, scorer and configuration.
Concepts
Build and score separation, immutable run metadata, fixtures and live tiers, parallelism, caching, secure config, evaluator error taxonomy and artifact integrity.
A flaky or underpowered gate can block safe changes or let critical regressions through.
Concepts
Stable baselines, fast and slow tiers, absolute and relative thresholds, uncertainty, critical-slice gates, retry policy, artifact review and exceptions.
Use the release worksheet and simulated rollout. No service credentials are needed.
Demonstration
Write a release packet and run a tabletop rollback after a synthetic severe-slice regression.
Exercise and assessment
Present the owner, decision, rollback trigger, evidence link and follow-up dataset change.
Theory (hrs)
1h 30m
Guided (hrs)
1h 30m
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
5h 30m
Instructor-led (hrs)
3h 30m
Repository mapping
Existing files
—
Audit findings
C13,V04
Reuse strategy
Reuse the training traces and scores as mock release evidence.
Teaching and experiment gaps
Add policy, sign-off and rollback templates plus decision scenarios.
Technical corrections
—
M14 Observability, operations and review loops3 topics · 17h 30m
Advanced · Core · Production Engineering
Description
Follow live-style evidence from a trace into triage, labeling, drift detection and regression checks.
Objectives
Instrument traces, sample responsibly, define alerts and understand cost, latency and operational reliability.
Labs
Simulate a bank support incident using synthetic traces, a dashboard export and an offline score pipeline.
Competencies
Explain which signal moved, identify the cause and turn it into a versioned regression case with a traceable owner.
Completion
Submit a trace-based incident brief and new regression case. Correlate a synthetic event, justify a sampling or alert decision and recompute a tail latency or cost measure. Confusing telemetry loss with behavior drift or retaining unnecessary payloads fails.
Theory (hrs)
4h 30m
Guided (hrs)
6h
Experiment (hrs)
4h
Exercises (hrs)
3h
Learner total (hrs)
17h 30m
Instructor-led (hrs)
12h
M14-T01 Traces, monitoring and LangfuseNeeds Correction · 5h 30m
Observability makes eval context inspectable across stages and repeated incidents.
Concepts
Trace/span/generation concepts, correlation IDs, tags, scored observations, telemetry loss and redaction. Choose Langfuse for inspectable LLM traces and scores with suitable deployment/data controls. Consider OpenTelemetry for cross-service context, with application-specific quality scoring elsewhere.
Reuse the performance definitions and real latency anomalies as prompts.
Teaching and experiment gaps
Add token and percentile calculations, load profiles and cost model.
Technical corrections
Separate inference, evaluation and end-to-end timing and record failed attempts.
M15 End-to-end capstone2 topics · 13h
Advanced · Core · Production Engineering
Description
Demonstrate ownership of evaluation from goals and annotated data to release evidence and incident follow-up.
Objectives
Integrate a release-quality suite, explain decision trade-offs and defend results in a teaching demonstration.
Labs
Submit a replayable evaluation package for one fictional feature spanning chatbot, RAG or agent behavior and production evidence.
Competencies
A peer can reproduce the result, challenge the decision and rerun a regression after a simulated incident.
Completion
Submit a replayable evaluation package, incident diagnosis and a release defense for the chosen feature. A peer must reproduce scores, reproduce the regression and observe an evidence-backed decision. An unreproducible report or no clear response to risk fails.
Theory (hrs)
2h
Guided (hrs)
3h 30m
Experiment (hrs)
4h
Exercises (hrs)
3h 30m
Learner total (hrs)
13h
Instructor-led (hrs)
8h
M15-T01 Design review and reproducible suiteNew · 6h 30m
M12-T03 — required for the defensive-assessment incident variant
Implementation readiness
Use the synthetic incident packet, report template and peer verification.
Demonstration
Diagnose a scripted incident, add a regression test and teach one evaluation failure with a reproducible demonstration.
Exercise and assessment
Defend the release decision and show a peer can verify the corrected outcome from the submitted artifacts.
Theory (hrs)
1h
Guided (hrs)
1h 30m
Experiment (hrs)
2h
Exercises (hrs)
2h
Learner total (hrs)
6h 30m
Instructor-led (hrs)
4h
Repository mapping
Existing files
—
Audit findings
V04
Reuse strategy
Use selected historical examples as prompts for the oral defense.
Teaching and experiment gaps
Add instructor scoring rubric, incident variants and model answers.
Technical corrections
—
Specialized / Expert3 modules · 54h 30m
M16 Evaluation across languages and modalities3 topics · 17h 30m
Expert · Elective · Specialization
Description
Adapt data and rubrics when text quality measures fail to capture language, visual or long-context requirements.
Objectives
Design slice-aware metrics and evidence for three settings with different failure modes.
Labs
Build a small, domain-specific slice on top of the bank case contract and calibrate it.
Competencies
Demonstrate why a baseline score changes meaning and validate a revised rubric on held-out cases.
Completion
Submit a modality or locale-specific slice for each studied topic and a revised rubric. Verify an evidence label and one held-out counterexample per slice. Score-only comparisons with no domain evidence or incomparable samples fail.
Theory (hrs)
4h 30m
Guided (hrs)
5h 30m
Experiment (hrs)
4h 30m
Exercises (hrs)
3h
Learner total (hrs)
17h 30m
Instructor-led (hrs)
11h 30m
M16-T01 Multilingual and localized evaluationNew · 5h 30m
M08-T03 — required when experimenting with reranked retrieval context
Implementation readiness
Use P02 and P04 to inspect saved context variants.
Demonstration
Use saved prompts with the same evidence in different positions and label unsupported conclusions.
Exercise and assessment
Report controlled position and context-length effects across slices.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h
Repository mapping
Existing files
—
Audit findings
—
Reuse strategy
Reuse the RAG evidence and oracle-control structure.
Teaching and experiment gaps
Add controlled contexts, staged difficulty and held-out document families.
Technical corrections
—
M17 Specialized outputs and coordinated systems3 topics · 18h
Expert · Elective · Specialization
Description
Verify outcomes for typed extraction, generated code and collaboration between agents.
Objectives
Choose precise task outcomes and analyze errors hidden by generic text scores.
Labs
Apply the case contract to one specialization and verify outcomes with deterministic fixtures.
Competencies
Submit a domain-specific failure taxonomy and reproducible scorecard against examples that defeat a generic judge.
Completion
Submit outcome fixtures and a failure taxonomy for the specialized output or workflow. Replay at least one hard negative per topic and explain why a generic judge misses it. Missing field checks, unisolated code execution or unverifiable final state fails.
Theory (hrs)
4h 30m
Guided (hrs)
6h
Experiment (hrs)
4h 30m
Exercises (hrs)
3h
Learner total (hrs)
18h
Instructor-led (hrs)
12h
M17-T01 Structured extraction and classificationNew · 6h
Use P04 to trace fictional handoff events and verify shared state.
Demonstration
Replay a mock support handoff where one agent omits a required state field. Compare with a valid delegated workflow.
Exercise and assessment
Verify global invariants, handoff quality and team cost from saved events.
Theory (hrs)
1h 30m
Guided (hrs)
2h
Experiment (hrs)
1h 30m
Exercises (hrs)
1h
Learner total (hrs)
6h
Instructor-led (hrs)
4h
Repository mapping
Existing files
—
Audit findings
C06
Reuse strategy
Extend the single-agent trace schema with actor and handoff IDs.
Teaching and experiment gaps
Add coordination and conflicting-state fixtures with outcome labels.
Technical corrections
—
M18 Evaluator robustness and evaluation research3 topics · 19h
Expert · Elective · Specialization
Description
Keep scores meaningful as evaluators, datasets and deployed models change.
Objectives
Stress-test evaluators, manage model migration, identify contamination risk and assess emerging methods.
Labs
Audit an evaluator and recommend a replacement strategy using anchors, holdout tests and monitored deployment evidence.
Competencies
Deliver a reproducible audit with known limits and an evidence-based tool or research recommendation.
Completion
Submit a method audit and migration or replacement memo. Verify one controlled anchor result, one uncertainty claim and a preserved holdout. Unchecked judge agreement or repeated tuning on the evaluation holdout fails.
Evaluation methods need a testable hypothesis and comparisons against simpler baselines.
Concepts
Active evaluation and adaptive sampling, preference modeling, redacted behavioral simulations, living benchmarks, measure validity and reproducible method reviews.
Drag the background or use arrow keys while the graph has focus. Use the zoom buttons or + and −. Search can take you directly to a node.
About the Course
Vision and audience
A long-term reference for learners entering LLM evaluation, engineers owning evaluation decisions, and instructors teaching or adapting practical labs. By the end of the core path, a learner should be able to design a versioned evaluation, build and calibrate scorers, quantify uncertainty, diagnose RAG and agent failures, and defend a production release decision. Expertise includes knowing when the evidence warrants another experiment or human review.
Start with concepts and explicit decision criteria, then build a small Python scorer. Use frameworks once learners can explain the metric and verify a result by hand. Study RAG and agents through observable inputs, outputs and states. Finish by connecting offline evaluations to release decisions, traces and monitored outcomes. The plan describes target competencies and future labs. The coverage statuses describe the current repository.
Sequence and time estimates
Core progression spans Beginner, Intermediate, Advanced and Production. Specialized / Expert modules are electives that can be studied independently after their listed prerequisites. Topic records give the minimum preparation for focused study. The recommended module order provides a complete beginner route. Required study for an individual module is computed from the prerequisites of its topics. Each stage ends with a reviewable artifact, and the capstone brings those artifacts together.
Times are planning estimates in hours. Each topic lists theory, guided hands-on implementation, independent experiments and interpretation, exercises and assessment, total learner time, and instructor-led classroom time. Instructor-led hours are a separate delivery estimate for explanation, demonstration and supervised practice. Preparation, breaks, and optional provider work are outside that estimate. Plan extra preparation for a first teaching run.
Estimates assume a learner who can use a terminal and read simple Python, coaching available for the first lab, and prepared fixtures with expected outputs. Guided times include routine setup and troubleshooting for the activity. Learners new to Python should practice reading a JSON file, looping over records, and running a test before starting guided implementation. Large downloads, hardware delays, provider wait times, deep remediation and elective live extensions vary by environment. Pilot time estimates with both a beginner and an experienced engineer.
One evolving application
Beginner — Evaluate a fictional bank support chatbot from six saved responses. Build exact/semantic boundary examples, a deterministic scorer and a blind human review sheet. Explain failure, uncertainty and execution error.
Intermediate — Publish a versioned gold dataset, calibrate a judge on frozen examples and compare prompt versions with paired Python analysis. Verify labels, denominators and uncertainty before drawing a conclusion.
Advanced RAG — Give the chatbot a fictional policy corpus. Evaluate ranked passages, context sufficiency, claims and citations, then compare retrieval changes against an oracle reference.
Advanced agents and safety — Add a mock account lookup and a simulated workflow. Verify actions, arguments, state, recovery and safe refusals. Assess defensive controls from fixed fictional traces and inert records.
Production — Add fixture-driven gates, trace metrics and a simulated incident. Convert a reviewed failure into a regression fixture and defend a release, hold or rollback decision.
Electives — Change the task when the method needs different evidence. Use synthetic fee charts for multimodal evaluation, public-style and private fictional benchmarks for contamination lessons, and benign generated functions for test-driven code evaluation. Use the healthcare examples for transfer lessons on score validity and source grounding.
Execution modes and tool choices
Every essential lesson starts with a fixture or simulated run that needs no model, network or credentials. Live Ollama inference and locally cached embeddings are optional hands-on extensions. The existing Test 5 configuration points to OpenAI and the existing Test 10 RAGAS integration requires OpenAI calls. These examples are references until local or fixture alternatives and effective settings are verified. Hosted graders and paid comparisons are optional, explicitly budgeted extensions.
Choose DeepEval for Python-oriented assertions and thresholds when its chosen metric fits the construct. Choose RAGAS for a RAG metric suite when the record carries its required references or evidence. Both may call judges or embeddings and need versioned adapters, explicit failure states and checked denominator behavior. Choose promptfoo for a declarative prompt and provider matrix, and OpenAI Evals for an OpenAI-centered managed grading workflow when supported graders, data handling and cost fit. Compare both on saved cases before adding a managed run.
Choose Langfuse for linked LLM traces, observations and scores. It depends on the quality of instrumented events and attached scores. OpenTelemetry is useful when correlation across services is the objective, with separate application quality measures. In defensive evaluation, Garak illustrates scanner-style coverage and PyRIT illustrates organized assessment records. Use prepared offline reports to compare their evidence formats and coverage. Tool output depends on scope, fixtures and labels, and requires review before it informs a decision.
The course audit in docs/COURSE_AUDIT_REPORT.md records evidence and impact. Each topic lists its current status, source files, relevant findings, suggested reuse, missing teaching and experiments, and corrections for a future implementation phase. Reusable means a substantial part already supports the topic. Needs Improvement indicates useful but incomplete material. Needs Correction identifies a defect affecting the intended lesson. New indicates absent substantive coverage. Not Assessed is reserved for insufficient evidence. Validation items beginning with V require further investigation. Current HTML/JSON reports illustrate past runs, and the teaching plan calls for separate answer keys.
Test 10 contributes RAGAS code to the framework comparison. Its script and dataset paths are in that topic's mapping. Test 11 contributes DeepEval setup, metrics and pitfalls to the same topic.
The repository README, course hub, home page, and concepts reference support navigation, setup and terminology. Use the Master Curriculum for the recommended learning order.
Give prerequisites, setup, input artifacts and an entry checkpoint.
Explain the concept in plain language.
Explain why it exists.
State the problem it solves and constraints on use.
Give realistic engineering use cases.
Work a small example by hand with a known answer.
Increase complexity gradually, including a conflicting or ambiguous case.
Give a step-by-step Python implementation or reproducible Python replay when the topic concerns a managed workflow.
Explain code sections that change behavior, scores or denominators.
Show expected artifacts and key execution results.
Interpret results and what decisions they can support.
Demonstrate common mistakes.
Explain limitations and failure cases, including evaluator failure where relevant.
Include a guided experiment with one controlled variable.
Assign independent exercises with measurable outputs.
Provide solutions or verification criteria, including accepted alternative solutions.
Include instructor notes with timing, prediction prompts, common learner errors and offline fallback.
Document troubleshooting and how to identify failure stage.
Include knowledge checks and observable completion criteria.
Connect the current topic to earlier and later evaluation decisions.
Classroom preparation and assessment
Each session should use the topic's prerequisite check and demonstrations, then ask learners to predict a result before running it. Prepare a small known-answer fixture, one common misconception and a harder case. Grade claims against evidence and review inclusion, decision policy and exceptions. Where output is stochastic, grade expected invariants and justified decisions. For the capstone, require an annotated dataset, manifest, replayable report, failure diagnosis and release recommendation. Ask a peer to reproduce the result.
At least one teaching variant for core labs should use standard-library Python and prepared data. Where a framework is the objective, provide a fixture containing its inputs and results and prepare a pinned optional environment. Instructors should try the first run on target classroom hardware and define bounds for compute, data handling and spend before optional live sessions.
Implementation sequence
First milestone — Deliver the M01–M03 fixture-first slice, the starter M04-T01 schema and the first M06-T01 judge replay. Use exactly shared response IDs across the human, rule and judge evidence. Supply a handout, annotated answer key, instructor demonstration and local fixture verification before updating requirements for optional runs.
Build the dataset and measurement spine in M04–M07. Establish reliable denominators, anchors, held-out cases, paired comparisons and uncertainty exercises before publishing example baselines.
Add M08–M12 around the same contract. Reuse the banking corpus and traces. Verify gold passage and final-state controls before framework demos. Prepare safety and defensive-review fixtures for bounded, fictional labs.
Add M13–M15 after the offline gates are stable. Introduce sampled synthetic traffic, redacted tracing and incident-to-regression review.
Build electives once the relevant core artifacts are stable. Add tracks based on learner need and verify each against a separate domain-specific rubric.
Maintenance, versioning and regeneration
Curriculum revision 1.1.
Use stable module and topic IDs. Edit titles and passages in place. Allocate new IDs for new learning objectives and preserve old IDs in a deprecation record when retiring a topic. Increment the curriculum revision when changing expected competencies, prerequisites or coverage. Version datasets, prompts, model snapshots and evaluator definitions independently and link those versions from lesson materials.
Update the per-topic mapping whenever a lesson or experiment changes. Review all dependent topics when a prerequisite changes. A topic is ready for implementation sign-off when the lesson follows the standard, an instructor has replayed it, a learner can verify its output and the audit-linked corrections are resolved or documented. Re-estimate time after actual pilots, and record scope changes with the relevant IDs.
The Markdown is authoritative. Run python3 docs/generate_master_curriculum.py from the repository root with Python 3.10 or newer to regenerate the standalone HTML. Use python3 docs/generate_master_curriculum.py --check to verify the checked-in artifact matches exactly. Edit the structured Markdown fields and rerun the generator. Use explicit anchors for IDs, comma-separated mandatory and recommended topic links, and the fixed time column order. Use a link followed by an em dash and a named exercise for each conditional prerequisite, with slash-separated entries when needed. Update the Paths A–G table when a target or required dependency changes. The generator validates finding IDs against the audit when that file is present. The HTML is a portable view of the plan. Repository lesson links require the companion lesson files. Progress is stored in the browser for the current device and may depend on file URL and privacy settings.