01 What students build
Three subsystems that build on each other. A: ShopSphere, a real customer-support chatbot to point the judge at. B: RAG Explorer, a pipeline that exposes every stage (ingest, chunk, embed, store, retrieve, answer) so retrieval itself can be graded. C: the DeepEval framework, a single metric registry of 29 metric × target rows that drives pytest runs and a live dashboard on :8203, scoring relevancy, faithfulness, grounding, hallucination, bias, toxicity, correctness and PII leakage.
Switchable Judge
One env var, JUDGE_PROVIDER=openai | groq | ollama. The same CompatibleJudge works for all three because each exposes an OpenAI-compatible endpoint.
29-Row Metric Registry
registry.py is the single source of truth: 10 chatbot, 11 RAG, 7 BrowserBash and 1 synthetic row, each with metric, target, threshold and scoring direction.
Black-Box Target
BrowserBash on aleeup.com replies in plain text, takes a visitorId, and hides its model. Grading a vendor bot you don't control is the realistic production case.
02 Architecture
03 Live demo (simulated with sample data)
This is a UI walkthrough with pre-baked sample data so you can feel the dashboard without booting three services. In the course you run the real thing: FastAPI apps on :8201 and :8202, the judge harness on :8203, and deepeval pinned to 3.9.9.
04 Inputs & outputs
What goes in
- Golden datasets per target (question, expected answer, context)
- registry.py rows: metric, target, threshold, scoring direction
- JUDGE_PROVIDER env var to pick the judge LLM
- Live HTTP replies from :8201, :8202 and the black-box bot
What comes out
- Score + pass/fail + judge reasoning for every metric row
- pytest / deepeval test run results, filterable by marker
- Dashboard run history with editable goldens
- Optional push to Confident AI for team-visible reports
05 How it's built
POST /chat on :8201 for the judge (mock mode without a Groq key).dashboard/registry.py: 10 chatbot, 11 RAG, 7 BrowserBash, 1 synthetic.