Skip to main content
Monitor online (per judge) scores a deterministic sample of fresh logged traffic continuously: capped per tick, billed as normal usage, and never double-judging a request. Trace-unit judges monitor completed runs: a trace quiet for 10 minutes is judged once, as a whole trajectory, keyed to the trace so a straggler step never causes a re-judge. The rolling corrected pass rate feeds:
  • quality_low alerts: fired when the rate’s CI lower bound crosses your floor with enough judgments to mean it. A bad hour on 8 samples is noise, not an incident.
  • The quality-gated canary: set gate_* fields on a model alias canary split and the online scorer judges both arms of the live split separately. The gate:
    • holds until both arms have gate_min_samples scored requests,
    • promotes only when the canary’s corrected-rate CI lower bound is at least the incumbent’s: demonstrably not-worse, not point-estimate lucky,
    • rolls back only when even the canary’s optimistic bound sits below your rollback threshold: confidently bad, not unluckily sampled.
Every gate lives on the Live switches tab (arms, sample counts, and the latest verdict) and the same gate config is editable in the alias dialog on the Models page, not only via the API. Trace-quality gating: point gate_criterion_id at a trace-unit judge and the gate scores completed runs of the alias’s traffic: a model swap evaluated on whole agent runs or conversations. Arm attribution is honest or absent: a run whose steps touched both models counts toward neither. Monitor config: everything in the Monitor dialog (online_enabled, online_percent, the population (tag and segment), and the weekly spend ceiling online_cap_usd, 0 = uncapped) is equally settable via PATCH /v1/criteria/{id}, same validation, no forked rules. When a judge’s online judging bill reaches its cap, that judge pauses until the week rolls over, and everything else keeps running. Every billed micro-cent counts, judge retries included. Changing the cap never voids calibration (money config is not instrument config); changing the population does. gate_mode: "recommend" (default) surfaces verdicts on GET /v1/aliases and via your alert email/webhook; you click the button. "auto" lets the gate repoint the alias itself, always audited with the numbers, and requires a calibrated judge: a judge with a weaker signal may recommend, but it cannot repoint traffic itself.

​
Drift detection

Corrected pass rates rely on the judge’s TPR/TNR staying true on live traffic. When your traffic shifts (a release, a new failure mode, a change in what users ask), a judge that was calibrated last month may quietly drift, and a stale correction would report confident numbers that are wrong. Omnia watches for this. A background sweep flags a judge whose calibration has gone stale or whose live corrected-quality has dropped anomalously; the flagged judge shows drift_status: "flagged", and the review queue prompts you to re-calibrate it on a fresh slice of grades, recomputing a current, localized TPR/TNR. The instrument that catches drift in your models is held to the same standard itself: a validated number that goes stale gets caught and re-measured, not trusted indefinitely.

​
Design rules

  • Binary verdicts everywhere. Pass/fail aligns with humans; 1–5 scores don’t.
  • Judges rank and monitor in aggregate; they never gate a single request. Inline guardrails are deterministic; evaluators are sampled and statistical.
  • Alert and promote on interval bounds, never point estimates.
  • Partial responses are review signals, never data. Aborted streams are visible in the review queue and Logs (opt-in view) but structurally excluded from datasets, exports, comparison sampling, and online scoring.
  • The human stays the judge. Suggestions, scans, and auto-improve amplify your grades; nothing auto-grades, and only a calibrated judge in explicit auto mode may touch routing.