> ## Documentation Index
> Fetch the complete documentation index at: https://docs.omnia-voice.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Corrected pass rates & the CI gate

> Judge-scored comparisons corrected for the judge's measured error, deterministic assertions, a deploy gate on interval bounds — and the certified switch test.

A **judge-scored comparison** (`POST /v1/evals` with `eval_kind:
"criterion"`) grades each model's fresh answers pass/fail and reports two
numbers per model: the judge's observed pass rate, and the **Rogan–Gladen
corrected** rate, what a human would most likely have scored given the
judge's [measured error rates](/judges/calibration), each with a 95%
interval. When a judge has no usable signal (TPR + TNR ≈ 1), Omnia refuses
to correct rather than amplify noise.

**Assertions** are the deterministic half: `json_valid`, `json_schema`
(full JSON Schema: `minItems`, `pattern`, `additionalProperties` and the
rest are enforced), `regex_match`, `contains`, `not_contains`, `max_length`,
`min_length`, and `completed`. Free, exact, validated at creation, and run
against every generated response.

`completed` fails any output cut off by the token limit
(`finish_reason: length`) or a reasoning trace that never produced a final
answer. This is the failure every text-reading judge misses (a truncated
answer is usually fluent right up to the cliff), so results also report
a mechanical **cut-off count per model** whether or not you add the
assertion.

## The CI gate

**`GET /v1/evals/{id}/gate`** turns any finished run into a deploy decision:

```bash theme={null}
curl -f "https://gateway.omnia-voice.com/v1/evals/$RUN/gate?min_assertion_pass_rate=1&min_win_rate=0.5" \
  -H "Authorization: Bearer $OMNIA_API_KEY" || exit 1
```

200 when every check passes, 412 otherwise. Statistical thresholds gate on
**confidence-interval lower bounds**, never point estimates: a 62% win rate
whose interval reaches down to 51% does not clear a 55% bar. Unfinished runs
fail closed.

## The certified switch test

When the decision is "replace the model behind this traffic", run the
comparison in its strictest shape: set the baseline to the literal
`__stored__`. Your logged answers become the incumbent arm; the judge grades
what you actually shipped against fresh candidate answers on the same prompts,
and three extra rules apply:

* **A calibrated judge is required.** No trusted instrument, no run.
* **Population matching is strict.** Where an ordinary run warns when the
  judge's calibration population and the sampled traffic don't match, a
  certified run refuses: a switch decision must be measured on the population
  it will affect, not a blend.
* **The gate is a noninferiority test, not a raw comparison.** The margin is scaled by
  the judge's own measured error (its Youden index), so a noisier judge must
  clear a wider bar. If the sample is too small to conclude at that bar, the
  gate answers `412` (*underpowered, refused*) instead of pretending.

A certified pass is the strongest statement the platform makes: this candidate
is not worse than what you're shipping, on your traffic, by your standards,
measured with an instrument whose error rate is known.