Skip to main content
A judge-scored comparison (POST /v1/evals with eval_kind: "criterion") grades each model’s fresh answers pass/fail and reports two numbers per model: the judge’s observed pass rate, and the Rogan–Gladen corrected rate, what a human would most likely have scored given the judge’s measured error rates, each with a 95% interval. When a judge has no usable signal (TPR + TNR ≈ 1), Omnia refuses to correct rather than amplify noise. Assertions are the deterministic half: json_valid, json_schema (full JSON Schema: minItems, pattern, additionalProperties and the rest are enforced), regex_match, contains, not_contains, max_length, min_length, and completed. Free, exact, validated at creation, and run against every generated response. completed fails any output cut off by the token limit (finish_reason: length) or a reasoning trace that never produced a final answer. This is the failure every text-reading judge misses (a truncated answer is usually fluent right up to the cliff), so results also report a mechanical cut-off count per model whether or not you add the assertion.

​
The CI gate

GET /v1/evals/{id}/gate turns any finished run into a deploy decision:
200 when every check passes, 412 otherwise. Statistical thresholds gate on confidence-interval lower bounds, never point estimates: a 62% win rate whose interval reaches down to 51% does not clear a 55% bar. Unfinished runs fail closed.

​
The certified switch test

When the decision is “replace the model behind this traffic”, run the comparison in its strictest shape: set the baseline to the literal __stored__. Your logged answers become the incumbent arm; the judge grades what you actually shipped against fresh candidate answers on the same prompts, and three extra rules apply:
  • A calibrated judge is required. No trusted instrument, no run.
  • Population matching is strict. Where an ordinary run warns when the judge’s calibration population and the sampled traffic don’t match, a certified run refuses: a switch decision must be measured on the population it will affect, not a blend.
  • The gate is a noninferiority test, not a raw comparison. The margin is scaled by the judge’s own measured error (its Youden index), so a noisier judge must clear a wider bar. If the sample is too small to conclude at that bar, the gate answers 412 (underpowered, refused) instead of pretending.
A certified pass is the strongest statement the platform makes: this candidate is not worse than what you’re shipping, on your traffic, by your standards, measured with an instrument whose error rate is known.