The Complete AI Evaluation Curriculum

Why AI benchmarks don't predict real-world performance. The gap between demo and production.

Real-world failures: hallucinating legal citations, biased hiring tools, wrong medical advice.

Why evaluation isn't just for ML engineers. How PMs, marketers, ops, and execs all play a role.

Define → Measure → Test → Interpret → Decide. The mental model that structures all evaluation.

Why automation can't replace human evaluation - and what happens when orgs try.

Adoption rate is vanity. Resolution rate is outcome. The single most important distinction.

What to measure for chatbots, RAG pipelines, code assistants, agents, and content generators.

A RAG answer can be faithful to stale docs and still wrong. Why this distinction is critical.

Why tracking only accuracy misses the picture. Build a measurement system, not a single number.

A practical framework for choosing which metrics to track. Covers relevance, measurability, actionability, and cost.

Automated metrics, human evaluation, LLM-as-judge, and hybrid. When to use each and why.

Verbosity bias, position bias, self-preference, authority bias. How LLM judges fail and how to calibrate.

Tasks where automated evals fail: nuance, safety, cultural sensitivity, open-ended generation.

A 3-tier content labeling system showing how much human judgment backs each eval result.

Blended Average, Aggregate Confidence, Infrastructure-as-Quality, Faithfulness-as-Correctness, Threshold-in-Isolation.

Why a green dashboard doesn't mean your AI is working. Aggregation hides failure modes.

Slice by user type, query type, time window. Aggregate scores hide who's failing.

The 6 questions to ask before trusting any eval number: sample size, distribution, recency, segmentation, baseline, trend.

Activity → Output → Outcome → Portfolio. Where most orgs are stuck and how to advance.

"We deployed 12 AI tools" is activity. "Support resolution time dropped 40%" is value.

How to justify eval investment to leadership. ROI framing, risk reduction, competitive positioning.

Human eval at $60/task vs. LLM-as-judge at $0.02. When cheap eval costs more in the long run.

Clinical accuracy, patient safety, regulatory requirements (FDA, HIPAA). Eval stakes are life-or-death.

Risk assessment accuracy, regulatory compliance, audit trails. Where one wrong number can move millions.

Citation accuracy, precedent matching, hallucination detection in legal research tools.

Resolution rate, customer satisfaction, escalation accuracy. Measuring the full support journey.

Originality, brand alignment, tone consistency. How do you evaluate what's inherently subjective?

Choosing between automated, human, LLM-as-judge, and hybrid based on task characteristics.

Unit testing for LLMs: DeepEval, pytest-style assertions, CI/CD integration, regression suites.

Rubric construction, rater recruitment, calibration sessions, and compensation models.

Prompt engineering for judges, multi-judge panels, and bias mitigation strategies.

AI screens + human confirms. Confidence-based routing. When to escalate from auto to human.

Faithfulness, answer relevancy, contextual precision, contextual recall. The RAGAS framework.

Tool correctness, step efficiency, goal completion. Evaluating multi-step autonomous agents.

Task completion, conversation flow, user satisfaction, escalation rate. End-to-end evaluation.

Functional correctness, security, test coverage, code quality. Evaluating generated code beyond "does it run."

Step-by-step process for creating domain-specific evaluation metrics. From definition to implementation.

Low inter-rater consistency corrupts training data, misleads benchmarks, makes results meaningless.

Calculating κ and κw. Interpreting: poor (<0.20) to near-perfect (0.81+). Python implementation.

Intraclass Correlation Coefficient for 3+ raters. ICC(2,1) vs ICC(3,1). When to use which.

Structure, frequency, gold standard items, adjudication process. The operational playbook.

Using QWK to validate when an LLM judge agrees with human evaluators. Trust threshold: 0.70.

Golden datasets, synthetic generation, stratified sampling, coverage analysis.

Automated quality thresholds in your deployment pipeline. Fail a PR if eval scores regress.

DeepEval, RAGAS, TruLens, MLflow, LangSmith, Arize, Confident AI compared.

Turning production failures into test cases automatically. The closed-loop eval system.

Linking every eval score to the exact prompt, model, and dataset version. Reproducibility as practice.

The artifact that translates eval results into a go/no-go deployment recommendation.

Executives want ROI. Engineers want methodology. PMs want feature decisions. Tailor the message.

What to show, what to hide. Segmented views, trend lines, confidence intervals.

How to handle "but our accuracy is 95%!" pushback. Methodology defense, limitation disclosure.

The bridge between "here are the numbers" and "here's what we should do."

Define metrics, run evaluation, identify failure modes, and recommend improvements. Full walkthrough.

Multi-step agent handling refunds and escalations. Trajectory analysis, tool use correctness.

Does it catch real bugs? False positive rate. Security review quality assessment.

Brand voice consistency, factual accuracy, originality scoring. Evaluating creative output systematically.

Draft a complete Deployment Clearance Report with methodology, findings, and recommendation.

Separating retrieval failures from generation failures. Context precision, recall, noise sensitivity in isolation.

Testing generation when context is contradictory, partial, or stale. Robustness testing.

Evaluating systems that synthesize across multiple documents. Reasoning chain validation.

12 failure modes: missed retrieval, wrong chunk, stale context, hallucinated source, and more.

Real-time faithfulness tracking, drift detection, embedding degradation, alerting thresholds.

Structured adversarial testing: threat models, attack surfaces, scenario design. NIST AI RMF aligned.

Direct injection, indirect injection, jailbreaks, system prompt extraction. Testing boundaries.

40+ vulnerability categories. DeepEval's red-teaming suite. Building automated safety gates.

Testing whether your AI system leaks training data, user data, or personally identifiable information.

How evaluation maps to regulatory frameworks. ARIA, CoRIx, compliance evidence generation.

6-criteria rubric with exact weights. Writing behavioral anchors on a 1 - 4 scale.

The method for determining cut-scores. Mean score of borderline-competent candidates.

Publishing κw, ICC with 95% CI, score distributions. Methodology page as a trust signal.

Discrimination index, difficulty level, distractor analysis. Ensuring each question does its job.

DIF analysis, accommodation policies, cultural sensitivity review. Test measures skill, not background.

The flagship CAEE scenario. DocAssist retrieval system with medical documentation. Full walkthrough.

10 stress test scenarios embedded in the lab. Contradictory evidence, missing context, adversarial inputs.

Time management for the 4-hour lab. What to include when you have 90 minutes left.

Dual-grading process, AI-screened → human-confirmed pipeline, rater calibration, appeals.

Skipping segmentation, confusing faithfulness with correctness, ignoring edge cases, weak recommendations.

Phase 1: full human dual-grading ($120). Phase 2: AI-screened ($70). Phase 3: AI primary ($35).

5-hour onboarding, gold standard scoring, monthly calibration, quarterly re-calibration.

Monitoring rater agreement over time. Every 25th submission triggers dual-grading.

Gold standard datasets, multi-layered QA, Krippendorff's alpha. Lessons from Apple and Google.

$60/submission at CAEE. Industry benchmarks. Balancing quality incentives with cost control.

Traces, spans, and eval hooks. LangSmith, Arize, Langfuse. Monitoring AI in the wild.

When performance degrades silently. Embedding drift, distribution shift, concept drift.

Prompt variants, model swaps, retrieval strategies. Statistical significance in AI context.

Thumbs up/down, explicit corrections, implicit signals. Turning user behavior into eval data.

Are your metrics measuring what you think? Correlation analysis, counterfactual testing.

Ad-hoc → Structured → Production → Continuous → Portfolio. Gap analysis and advancement roadmap.

Policies, standards (CS-001 - CS-004), advisory boards, review cycles. Institutional backbone.

Eval Engineer, Reliability Engineer, Quality Manager, ROI Analyst, Trust & Safety Lead. Hiring and structuring.

Evaluate with integrity, serve the stakeholder, disclose conflicts, protect data. The 8 principles.

Cost modeling: human eval, compute, tooling, training. Making the financial case to leadership.

When you have 20+ AI systems: prioritization, shared infrastructure, cross-system metrics, risk ranking.

Task routing, evaluator matching, result storage, API design. Build vs. buy decision framework.

Comparing AI systems against each other and baselines. Normalization, fairness, leaderboard design.

Smart routing, confidence-based escalation, parallel evaluation, queue management.

Comparing DeepEval, RAGAS, Arize, Confident AI, Galileo, LangSmith against your needs.

ML/AI engineers: code eval, model benchmarking, pipeline testing, CI/CD integration.

PMs and analysts: feature impact, user experience evaluation, A/B testing for AI features.

Marketing, sales, customer support. Content quality, lead scoring, resolution rate tracking.

Legal, finance, operations. Regulatory compliance evaluation, audit trail, risk scoring.

CTOs, VPs, board members: portfolio ROI dashboards, risk heat maps, strategic recommendations.

Scenario design, hidden challenges, time constraints, variant rotation. Measuring judgment.

Scenario variants, similarity detection, integrity statements, LLM-output detection.

Skill trees, progression paths, qualification gates, performance analytics, leaderboards.

What it takes to get your certification accredited. Standards, evidence, timelines, cost.

The EHCS system: Eval Craft (40%), Eval Impact (25%), Eval Leadership (15%). Earning paths A - D.

Bronze → Silver → Gold → Platinum tiers. Discount structures, adoption incentives.

Gate 1: 30+ deposits. Gate 2: 5 employer LOIs. Gate 3: κ ≥ 0.60 + target pass rate.

Lab difficulty, published methodology, portfolio artifact, tool-agnostic positioning.

Individual vs. corporate vs. enterprise pricing. Volume discounts, team packages, subscriptions.

Getting academic recognition. CE credits, university partnerships, academic advisory boards.

Fortune 500 evaluating 15 RAG systems. Architecture, tooling, governance, results.

Building an eval program for clinical AI. FDA alignment, patient safety, continuous monitoring.

Audit-ready evaluation for banking AI. Regulatory compliance, explainability, bias testing.

Capstone: given a fictional company with 8 AI systems, design a complete eval architecture.

The L4 peer defense format. Presenting your portfolio to a panel, handling questions.

The emerging C-suite role ($200K - $350K+). Scope, responsibilities, reporting, and the case.

How organizations that evaluate well outperform. Evaluation as decision infrastructure, not overhead.

Translating eval data into board decks. Risk heat maps, portfolio health, investment justification.

Revenue follows authority, not the reverse. Building a hard certification creates brand power.

Agent evaluation, multimodal eval, real-time eval, eval-as-a-service. Where the market is heading.

Govern, Map, Measure, Manage. How the risk management framework structures evaluation.

Articles 9 - 72: testing, evaluation, documentation. How eval certification provides compliance evidence.

Creating and publishing evaluation standards. Governance, public comment, revision cycles.

Partnering with PMI, IAPP, IEEE. Cross-recognition frameworks and mutual benefit structures.

NIST's Assessing Risks and Impacts of AI. Contextual Robustness Index. How to participate.

From "nobody evaluates" to "evaluation is how we make decisions." 5 stages with diagnostic indicators.

Overcoming resistance: "our AI is fine" syndrome. Stakeholder mapping, quick wins, champions.

EAC mentorship model: pairing Commanders with Specialist candidates. Structured programs.

Monthly community challenges on real AI systems. Leaderboards, shared playbooks, frameworks.

Writing for the eval community: case studies, methodology papers, open-source contributions.

Text, images, audio, video. Cross-modal faithfulness, grounding, and coherence evaluation.

Multi-step planning, tool use chains, error recovery, goal completion. Independent systems.

Evaluating AI responses as they happen. Streaming eval, latency constraints, online learning.

The recursive challenge. When your eval pipeline uses the same technology it's evaluating.

"Would the outcome have been different without AI?" Causal inference applied to evaluation.

Safety evaluation specialization: red teaming, vulnerability assessment, safety culture, incident response.

Policy, standards, compliance, audit. The governance layer above technical evaluation.

Cognitive bias, calibration, structured analytical techniques, expert elicitation.

Open Badges 3.0. Core + specialty badges. Stacking rules, display, and verification.

Community-contributed frameworks, evaluator placements, custom eval services. Platform economy.

3 artifacts required: eval program design, published contribution, mentorship evidence.

Publish methodology, open-source a framework, contribute to standards. Making eval better.

60-minute panel: 20 min presentation, 25 min Q&A, 15 min deliberation. Scoring criteria.

Anonymized examples of outstanding L5 portfolios. What excellence looks like at the top.

Advisory board membership, industry consulting, standards committee leadership. Where Commanders go next.