Chapters 16 + 18 · Evals Evals Track Eval Harness

Stop eyeballing AI answers: make an LLM judge them in pytest.

Three graded exercises that turn "the chatbot seems fine" into an assertable test suite. DeepEval's AnswerRelevancy and Hallucination metrics score a real model under test (Groq Llama-4 Scout), with GPT-4.1 as the judge, all inside plain pytest functions.

01 What students build

The core pattern of AI quality engineering: model under test vs judge model. Exercise 1 asserts a hardcoded answer with AnswerRelevancyMetric and HallucinationMetric. Exercise 2 calls Groq Llama-4 Scout live, captures the raw answer, and lets GPT-4.1 score it. Exercise 3 swaps the provider to OpenRouter with the same code shape, proving the harness is model-agnostic.

Metrics as Assertions

A LLMTestCase bundles input, actual output, expected output, and grounding context. assert_test() fails the build when a metric score crosses its threshold, just like any pytest assert.

Separate Judge Model

The model being graded never grades itself. Llama-4 Scout answers, GPT-4.1 judges relevancy and hallucination and writes a reason you can read in the report.

Provider Swapping

Groq and OpenRouter are both OpenAI-compatible: only the base URL, model id, and key change. One harness evaluates any of hundreds of models.

02 Architecture

Promptpytest test fn
──▶
Model under testLlama-4 Scout · Groq
──▶
LLMTestCaseinput · output · context
──▶
DeepEval metricsrelevancy · hallucination
──▶
Judge LLMGPT-4.1
──▶
assert_test()PASS / FAIL + reason
deepeval test run exercises/ · thresholds gate the build · swap groq for openrouter freely

03 Live demo (simulated with sample data)

This is a UI walkthrough with pre-baked sample data. In the course you run the real harness: deepeval test run exercises/ -d all -v with your own Groq, OpenAI, and OpenRouter keys in .env.local.

deepeval test run exercises/ · venv demo · sample data

04 Inputs & outputs

What goes in

  • A question, the model's actual answer, and an expected answer
  • Grounding context the answer must not contradict
  • Metric thresholds (relevancy ≥ 0.8, hallucination ≤ 0.3)
  • API keys: Groq or OpenRouter for the model, OpenAI for the judge

What comes out

  • A 0-1 score per metric with the judge's written reasoning
  • Hard PASS / FAIL verdicts that gate CI like any pytest suite
  • Per-test-case breakdown in the deepeval run report
  • Repeatable evals you can point at any OpenAI-compatible model

05 How it's built

Install DeepEval in a venv and export keys: GROQ_API_KEY for the model under test, OPENAI_API_KEY for the judge.
Exercise 1: build a minimal LLMTestCase with grounding context and assert AnswerRelevancy (≥ 0.8) plus Hallucination (≤ 0.1).
Exercise 2: write ask_groq() that hits Llama-4 Scout at temperature 0 and feeds the raw answer into the test case.
Pass model="gpt-4.1" to each metric so a different, stronger model does the judging.
Exercise 3: swap the client to OpenRouter (new base URL, model id, key) and re-run the identical assertions.
Run everything with deepeval test run exercises/ -d all -v and read scores, verdicts, and judge reasons in the report.

06 Tech stack

DeepEvalpytestGroq Llama-4 ScoutGPT-4.1 judgeOpenRouter gatewayAnswerRelevancyMetricHallucinationMetricpython-dotenv