Grading

This page covers grading: recording pass/fail verdicts on your real traffic, the ground truth every judge is measured against. Grade before you trust any judged number: an LLM judge is a noisy instrument, and used raw it agrees with a human expert far less often than two humans agree with each other, so a judged “80% pass rate” tells you nothing until you know how often that judge agrees with you. Validated evals close that gap: you grade a slice of real traffic, Omnia measures every judge against your grades, and every number downstream carries that measurement. Throughout these pages, the dashboard words and the API words: judges are criteria in the API, calibration is align/alignment, grades are labels, and a comparison is an eval run. The prose uses the dashboard words; every API literal keeps its real name. The workflow, end to end:

1. Grade real traffic pass/fail (this page, the Review page, or POST /v1/labels).
2. Create judges (binary quality questions) and calibrate each one against your grades to measure its TPR/TNR/κ. See Judges & calibration.
3. Run judge-scored comparisons whose pass rates are corrected for the judge’s measured error, and gate deploys in CI on assertions + interval bounds. See Corrected pass rates.
4. Turn it online: score a sample of live traffic continuously, alert when quality drops, and let a quality-gated canary promote or roll back a model change on statistical evidence. See Online monitoring.

In the dashboard, all of this lives on Evals, which has three tabs: Comparisons (bake-offs; see Evals), Judges (everything in this group of pages), and Live switches (the gate board: every quality-gated canary, its arms, and its latest verdict in one place).

​

Grade real traffic

Dashboard → Review is a keyboard-driven queue over your logged traffic: P pass, F fail (with a critique: say why), S skip. Grades are binary, one per request, and owner/admin-only: they define your quality bar. The queue sorts the highest-signal traces first: responses the user abandoned mid-stream (aborted), outputs cut off by the token limit (cut off at token limit, finish_reason: length), and requests whose opening message was re-sent within minutes (user retried): implicit feedback the gateway captures with zero integration on your side. Multi-step agent runs (and multi-turn conversations you group the same way) are first-class: send an X-Omnia-Trace-Id header and every step is grouped. Traced traffic appears in the queue as one item per run, its steps, tool calls, and failure flags one toggle away, and your verdict is a whole-run verdict (scope: "trace" on the label). You can’t honestly judge step 7 without seeing what step 3 did, so the queue shows you the trajectory before you press P or F. The same trace view opens from any traced row in Logs, and GET /v1/traces/{trace_id} returns the steps in execution order. Aim for ~100 grades; stop when 20 in a row teach you nothing new. Critiques on failures matter: they become the judge’s few-shot examples and the raw material for suggested judges. Programmatic grading (POST /v1/labels) lets systems that already know an outcome push verdicts: a support tool marking an escalation, a pipeline that caught a bad extraction.

​

Grade retention

The Review page’s Your grades list makes every grade you’ve ever recorded browsable: verdict, critique, and when. Grades are permanent; the logged conversations they graded are not, because logs age out on your retention window (7, 30, or 90 days). A grade whose conversation has expired still shows honestly: the grade record is there, the content is gone. This is by design: your quality history is an asset; retention is a promise. Datasets sit between the two: a dataset curated from logs is a frozen snapshot that does not age out with the logs it came from.

​

Next

With grades recorded, create a judge and calibrate it against them. That measurement is what every downstream number stands on.