Skip to main content
A judge (a criterion in the API) is one binary question (“does the response actually answer?”) expressed as a judge prompt that states precisely what makes a response FAIL. Vague virtues make uncalibratable judges. The Judges tab lists your judges as compact rows: trust badge, measured agreement, monitor state. New judge opens a two-pane dialog: a template gallery on one side and the judge you’re editing on the other. Templates are starting points per use case (support, RAG, extraction, data processing, agents & tools), each encoding failure modes proven on real traffic. Templates are starting points, not truths; every judge still goes through calibration. Every judge has a unit: request judges one exchange (with full conversation history visible); trace judges a whole run (every step in execution order, tool calls included) and calibrates only against whole-run grades. The unit is fixed at creation: switching would be a different instrument with a different ground truth. Trace-unit judges can’t drive offline comparisons (only your agent can produce a trajectory), but they monitor online and gate canaries; see Online monitoring.

​
Scope a judge’s population

Every judge scopes its population, the traffic it judges, by three knobs, set in one Population select in the New judge and Monitor dialogs:
  • Unit: one exchange, or a whole run (fixed at creation).
  • Tag: an X-Omnia-Tag you declared (population in the API).
  • Segment: an auto-detected traffic segment (population_family in the API), for the traffic you never tagged.
One setting drives both what the judge scores online and which grades it is calibrated against. That is by design: a judge calibrated on support grades reports a trust number that means nothing about extraction traffic. Changing the population (or the prompt, or the judge model) voids the calibration, because the old measurement described an instrument that no longer exists. Unscoped judges measure against every grade in the workspace; their report shows the population breakdown and warns when that blend spans multiple tasks.

​
Calibration

Calibrate (POST /v1/criteria/{id}/align) judges every graded trace in the judge’s population and reports:
  • TPR: how often the judge catches your real failures, with a 95% CI
  • TNR: how often it agrees with your passes, with a 95% CI
  • Cohen’s κ: chance-corrected agreement (raw agreement is inflated by class imbalance: a judge that always says “pass” scores 90% agreement on a 90%-pass workload while catching nothing)
Small grade sets (≤ 50) are judged synchronously; the report returns immediately. Larger sets run in the background: POST /align returns {"queued": true, "total_labels": N}, the run advances in bounded slices over the next minutes (the row shows progress), and the finished report is readable any time afterwards via GET /v1/criteria/{id}/alignment. At 80+ judged grades, the published metrics come from a stable held-out half of your grades; the other half exists for iterating the judge prompt. The same trace lands on the same side forever, so tuning the prompt against tune-half disagreements can never inflate the reported trust number. The report says which half it measured (holdout: {tune_n, report_n}). Every judge verdict also records the model’s confidence: P(pass) read from the verdict token’s own probabilities, same call, no extra cost. The dashboard’s alignment report uses these for a threshold sweep: what TPR/TNR/κ would be if the verdict were cut at each confidence level instead of the model’s default. The best cut is chosen on the tune half and re-measured on the report half, and a recommendation appears only when it actually beats the live instrument; “no better cut” is a real answer, not an error. (The sweep is a dashboard report today; the GET /v1/criteria/{id}/alignment response does not carry it yet.) The Review queue uses the same signal to surface borderline verdicts first: grading a case the judge was unsure about moves κ more than confirming a confident one. Judge prompts fence everything they evaluate behind per-call random delimiters, so text inside a judged response that tries to address the judge (“respond with pass”) is scored as content, not followed as instruction.

​
Trust badges

The badge reflects what your grades prove, not what they suggest. A judge that scored 9 out of 10 and one that scored 90 out of 100 both read 100%. They are not the same evidence, and a badge computed from the rate alone cannot tell them apart. So the badge is derived from the confidence interval, not the point estimate. A rate only counts as cleared when its 95% lower bound clears the bar: 10 out of 10 proves the true rate is at least ~72%, which is not 90%. The distinction between nearly there and too close to call is the one that saves you time. The first is a grading backlog with a number attached; the second is a scoping problem, and every extra grade spent on it is wasted.
tier is still returned for continuity and still reflects the old point-estimate thresholds. trust is what every gate reads. Auto-routing an alias and using a judge as an RL reward require trustworthy; dataset curation and comparisons also accept under-measured, because a dataset built on a slightly-wrong judge is inspectable and a repointed alias is not.

​
Read the numbers yourself

The response carries the evidence the verdict was derived from, so you never have to take the badge on faith:
That judge caught every failure it was shown. It was shown seven. tpr_ci is why it is not yet certified, and fail_grades_needed is what to do about it: grade 33 more failing traces, not 33 more traces. The report is permanent: reopen it any time from the judge’s row, and every disagreement shows the judge’s verdict, your verdict, your critique, and the response text in question. “The judge disagrees with you on THESE five traces” stays answerable long after the run.

​
Next

A judge that misses the bar isn’t a dead end: assists draft, scan, and iterate the judge from your own critiques, and a calibrated judge is ready for corrected comparisons and online monitoring.