> ## Documentation Index > Fetch the complete documentation index at: https://docs.omnia-voice.com/llms.txt > Use this file to discover all available pages before exploring further. # Judges & calibration > A judge is one binary question, measured against your grades: TPR, TNR, Cohen's κ — and a trust badge derived from confidence intervals, not point estimates. A **judge** (a criterion in the API) is one binary question ("does the response actually answer?") expressed as a judge prompt that states precisely what makes a response FAIL. Vague virtues make uncalibratable judges. The **Judges tab** lists your judges as compact rows: trust badge, measured agreement, monitor state. **New judge** opens a two-pane dialog: a **template gallery** on one side and the judge you're editing on the other. Templates are starting points per use case (support, RAG, extraction, data processing, agents & tools), each encoding failure modes proven on real traffic. Templates are starting points, not truths; every judge still goes through calibration. Every judge has a **unit**: `request` judges one exchange (with full conversation history visible); `trace` judges a **whole run** (every step in execution order, tool calls included) and calibrates only against whole-run grades. The unit is fixed at creation: switching would be a different instrument with a different ground truth. Trace-unit judges can't drive offline comparisons (only your agent can produce a trajectory), but they monitor online and gate canaries; see [Online monitoring](/judges/online-monitoring). ## Scope a judge's population Every judge scopes its **population**, the traffic it judges, by three knobs, set in one **Population** select in the New judge and Monitor dialogs: * **Unit**: one exchange, or a whole run (fixed at creation). * **Tag**: an [`X-Omnia-Tag`](/reference/observability#request-labels-tags-and-trace-ids) you declared (`population` in the API). * **Segment**: an auto-detected [traffic segment](/concepts/segments) (`population_family` in the API), for the traffic you never tagged. One setting drives *both* what the judge scores online *and* which grades it is calibrated against. That is by design: a judge calibrated on support grades reports a trust number that means nothing about extraction traffic. **Changing the population (or the prompt, or the judge model) voids the calibration**, because the old measurement described an instrument that no longer exists. Unscoped judges measure against every grade in the workspace; their report shows the population breakdown and warns when that blend spans multiple tasks. ## Calibration **Calibrate** (`POST /v1/criteria/{id}/align`) judges every graded trace in the judge's population and reports: * **TPR**: how often the judge catches your real failures, with a 95% CI * **TNR**: how often it agrees with your passes, with a 95% CI * **Cohen's κ**: chance-corrected agreement (raw agreement is inflated by class imbalance: a judge that always says "pass" scores 90% agreement on a 90%-pass workload while catching nothing) Small grade sets (≤ 50) are judged synchronously; the report returns immediately. Larger sets **run in the background**: `POST /align` returns `{"queued": true, "total_labels": N}`, the run advances in bounded slices over the next minutes (the row shows progress), and the finished report is readable any time afterwards via `GET /v1/criteria/{id}/alignment`. At **80+ judged grades**, the published metrics come from a **stable held-out half** of your grades; the other half exists for iterating the judge prompt. The same trace lands on the same side forever, so tuning the prompt against tune-half disagreements can never inflate the reported trust number. The report says which half it measured (`holdout: {tune_n, report_n}`). Every judge verdict also records the model's **confidence**: P(pass) read from the verdict token's own probabilities, same call, no extra cost. The dashboard's alignment report uses these for a **threshold sweep**: what TPR/TNR/κ would be if the verdict were cut at each confidence level instead of the model's default. The best cut is chosen on the tune half and re-measured on the report half, and a recommendation appears **only when it actually beats the live instrument**; "no better cut" is a real answer, not an error. (The sweep is a dashboard report today; the `GET /v1/criteria/{id}/alignment` response does not carry it yet.) The Review queue uses the same signal to surface **borderline verdicts first**: grading a case the judge was unsure about moves κ more than confirming a confident one. Judge prompts fence everything they evaluate behind per-call random delimiters, so text inside a judged response that tries to address the judge ("respond with pass") is scored as content, not followed as instruction. ## Trust badges The badge reflects what your grades prove, not what they suggest. A judge that scored 9 out of 10 and one that scored 90 out of 100 both read 100%. They are not the same evidence, and a badge computed from the rate alone cannot tell them apart. So the badge is derived from the **confidence interval**, not the point estimate. A rate only counts as cleared when its 95% lower bound clears the bar: 10 out of 10 proves the true rate is at least \~72%, which is not 90%. | Badge | Meaning | API `trust` | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | **Calibrated** | both lower bounds clear 90%: proven | `trustworthy` | | **Nearly there — needs more grades** | straddles the bar, and more grading would settle it. The API says how many, per class | `under-measured` | | **Too close to call** | straddles the bar with the rate hugging it. More grading will *not* settle this; the judge's question is too close to the threshold. Narrow what it asks and re-calibrate | `borderline` | | **Miscalibrated** | an upper bound sits *below* the bar: proven wrong, not merely unproven | `misaligned` | | **Not yet calibrated** | no interval at all: never calibrated, or your grades are one-class (all pass or all fail), so agreement on the missing class cannot be measured | `unmeasured` | The distinction between **nearly there** and **too close to call** is the one that saves you time. The first is a grading backlog with a number attached; the second is a scoping problem, and every extra grade spent on it is wasted. `tier` is still returned for continuity and still reflects the old point-estimate thresholds. **`trust` is what every gate reads.** Auto-routing an alias and using a judge as an RL reward require `trustworthy`; dataset curation and comparisons also accept `under-measured`, because a dataset built on a slightly-wrong judge is inspectable and a repointed alias is not. ## Read the numbers yourself The response carries the evidence the verdict was derived from, so you never have to take the badge on faith: ```json theme={null} { "trust": "under-measured", "tpr": 1.0, "tpr_ci": [0.646, 1.0], "tnr": 0.936, "tnr_ci": [0.828, 0.978], "fail_grades_needed": 33, "pass_grades_needed": 0, "kappa": 0.79 } ``` That judge caught every failure it was shown. It was shown seven. `tpr_ci` is why it is not yet certified, and `fail_grades_needed` is what to do about it: grade 33 more failing traces, not 33 more traces. The report is **permanent**: reopen it any time from the judge's row, and every disagreement shows the judge's verdict, your verdict, your critique, and the response text in question. "The judge disagrees with you on THESE five traces" stays answerable long after the run. ## Next A judge that misses the bar isn't a dead end: [assists](/judges/assists) draft, scan, and iterate the judge from your own critiques, and a calibrated judge is ready for [corrected comparisons](/evaluate/corrected-rates) and [online monitoring](/judges/online-monitoring).