> ## Documentation Index
> Fetch the complete documentation index at: https://docs.omnia-voice.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Grading

> Ground truth for every judge: grade a slice of real traffic pass/fail, and every number downstream carries that measurement.

This page covers grading: recording pass/fail verdicts on your real traffic,
the ground truth every judge is measured against. Grade before you trust any
judged number: an LLM judge is a noisy instrument, and used raw it agrees
with a human expert far less often than two humans agree with each other, so
a judged "80% pass rate" tells you nothing until you know how often that
judge agrees with **you**. Validated evals close that gap: you grade a slice
of real traffic, Omnia measures every judge against your grades, and every
number downstream carries that measurement.

Throughout these pages, the dashboard words and the API words: **judges** are
`criteria` in the API, **calibration** is `align`/`alignment`, **grades** are
`labels`, and a **comparison** is an eval run. The prose uses the dashboard
words; every API literal keeps its real name.

The workflow, end to end:

1. **Grade** real traffic pass/fail (this page, the Review page, or
   `POST /v1/labels`).
2. **Create judges** (binary quality questions) and **calibrate** each one
   against your grades to measure its TPR/TNR/κ. See
   [Judges & calibration](/judges/calibration).
3. Run **judge-scored comparisons** whose pass rates are corrected for the
   judge's measured error, and gate deploys in CI on **assertions + interval
   bounds**. See [Corrected pass rates](/evaluate/corrected-rates).
4. Turn it **online**: score a sample of live traffic continuously, alert
   when quality drops, and let a **quality-gated canary** promote or roll
   back a model change on statistical evidence. See
   [Online monitoring](/judges/online-monitoring).

In the dashboard, all of this lives on **Evals**, which has three tabs:
**Comparisons** (bake-offs; see [Evals](/reference/evals)), **Judges**
(everything in this group of pages), and **Live switches** (the gate board:
every quality-gated canary, its arms, and its latest verdict in one place).

<Note>
  One base URL, one key: your **traffic** (chat, embeddings, responses) and the
  **management API** on these pages (labels, criteria, evals, aliases) are all
  served from `https://gateway.omnia-voice.com/v1` with the same workspace API
  key.
</Note>

## Grade real traffic

**Dashboard → Review** is a keyboard-driven queue over your logged traffic:
`P` pass, `F` fail (with a critique: say *why*), `S` skip. Grades are
binary, one per request, and owner/admin-only: they define your quality bar.

The queue sorts the **highest-signal traces first**: responses the user
abandoned mid-stream (`aborted`), outputs cut off by the token limit
(`cut off at token limit`, `finish_reason: length`), and requests whose
opening message was re-sent within minutes (`user retried`): implicit
feedback the gateway captures with zero integration on your side.

Multi-step agent runs (and multi-turn conversations you group the same way)
are first-class: send an `X-Omnia-Trace-Id` header and every step is grouped.
Traced traffic appears in the queue as **one item per run**, its steps, tool
calls, and failure flags one toggle away, and your verdict is a
**whole-run verdict** (`scope: "trace"` on the label). You can't honestly
judge step 7 without seeing what step 3 did, so the queue shows you the
trajectory before you press P or F. The same trace view opens from any traced
row in Logs, and `GET /v1/traces/{trace_id}` returns the steps in execution
order.

Aim for \~100 grades; stop when 20 in a row teach you nothing new. Critiques
on failures matter: they become the judge's few-shot examples and the raw
material for suggested judges.

Programmatic grading (`POST /v1/labels`) lets systems that already know an
outcome push verdicts: a support tool marking an escalation, a pipeline that
caught a bad extraction.

## Grade retention

The Review page's **Your grades** list makes every grade you've ever recorded
browsable: verdict, critique, and when. Grades are **permanent**; the logged
conversations they graded are not, because logs age out on your
[retention window](/reference/request-logging) (7, 30, or 90 days). A grade
whose conversation has expired still shows honestly: the grade record is
there, the content is gone. This is by design: your quality history is an
asset; retention is a promise. Datasets sit between the two: a
[dataset curated from logs](/reference/request-logging#browsing-and-exporting)
is a **frozen snapshot** that does not age out with the logs it came from.

## Next

With grades recorded, [create a judge and calibrate it](/judges/calibration)
against them. That measurement is what every downstream number stands on.