1. Capture
Every measurement runs on your real traffic, so capture is the one required integration step. There are four integration paths, sized to what your codebase already does: route through the gateway, keep your provider and bring your own key, point existing OpenTelemetry at Omnia, or add the tracing SDK. Routing through the gateway also gets you one API across many open models on multiple clouds, per-token or per-GPU-hour billing from one wallet, caching, fallbacks, and observability; the other paths capture traffic without moving your inference at all. Capture is content-free by default; full content is opt-in and secret-scrubbed.2. Measure
An LLM judge is a noisy instrument. Used raw, an “80% pass rate” tells you little, because you don’t know how often that judge agrees with you. Omnia’s validated evals close that gap: you grade a slice of real traffic, Omnia calibrates every judge against your grades (TPR, TNR, Cohen’s κ), and corrects every downstream number for the judge’s measured error. That correction is what makes the numbers trustworthy. Instead of the judge’s raw guess, Omnia subtracts the judge’s known error and reports the corrected rate with a confidence interval, and it watches the judge for drift so the correction stays valid as your traffic changes. With a calibrated judge you can answer the question teams usually guess at: would a model 20× cheaper hold up on my traffic? Run a comparison against your incumbent model’s own logged answers and read the corrected result.Silent failures
The most expensive failures never throw: the trace says200 OK, and the
user got nothing useful. Dashboards built on status codes can’t see them.
Omnia catches silent failures three ways: judges catch them semantically
(a fluent answer that fails your rubric), finish-reason tracking
catches truncation (length means the answer was cut off, however confident
it sounds), and abort and abandonment signals catch the user walking away
mid-stream or re-asking the same question. The review queue boosts exactly
these rows, so the grades that calibrate your judge come from where the
product actually hurt.
3. Improve
When a cheaper model doesn’t quite hold up, you don’t have to stay on the expensive one. The calibrated judge becomes a reward: Omnia samples answers on your traffic, keeps only the ones the judge passes, and trains on the winners. See self-improvement, the offline form of reinforcement learning. The result is a model measurably better at your bar, proven on the same held-out grades that calibrated the judge. The switch is safe to ship: a quality-gated canary sends a slice of live traffic to the new model and promotes it only when it holds up statistically, never on a lucky sample.4. Own
The model trained on your traffic is yours. Run it on Omnia, deploy it to another provider, or take the weights and run them on your own infrastructure, air-gapped if you need to. Closed fine-tuning platforms lock your model inside their serving stack; Omnia hands you the weights and the validation report that proves they work. See Owning your model.You can enter the workflow at any step and stop wherever it pays off. Route traffic and never grade a thing. Compare a cheaper model and switch: that’s the common win. Train your own and take the weights: that’s the ceiling. Each step makes the next one possible, and none of them requires the ones after it.
Start with grading
The measurement layer everything else is built on.