> ## Documentation Index
> Fetch the complete documentation index at: https://docs.omnia-voice.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals

> Judged model comparisons on your own prompts: test whether a cheaper model holds up on your traffic, with a savings report.

Evals answer the question every team eventually asks: **"could a smaller,
cheaper model handle our traffic?"**, with evidence from your own prompts
instead of generic benchmarks. In the dashboard these are **comparisons**
(eval runs in the API), and they live on the **Comparisons** tab of the Evals
page.

<Note>
  Pairwise comparisons are half the story. For judges calibrated against **your
  own grades**, corrected pass rates, CI deploy gates, and the quality-gated
  canary, see [Judges & calibration](/judges/calibration); the Evals page's
  **Judges** and **Live switches** tabs.
</Note>

A comparison samples real requests from your [logged traffic](/reference/request-logging),
generates fresh answers on your current model (the *baseline*) and one or more
*candidate* models under identical conditions, and has an LLM judge score each
pair against a rubric you write in plain English. The result is a savings
report: quality delta, latency delta, and cost delta, side by side.

## Requirements

* [Request logging](/reference/request-logging) enabled, with at least 5
  distinct prompts captured (comparisons sample from your logs).
* Wallet balance: eval generations and judge calls **bill as normal usage**,
  disclosed before you start a run.

## Running a comparison

**Dashboard → Evals → Comparisons → Run a comparison** (owner/admin). The tab
is table-first: every past comparison is a row with its result inline, and
the guided flow walks you through:

<Steps>
  <Step title="Write a rubric">
    Describe what "good" means for your traffic, in plain English, e.g.
    "Answers must be factually correct, concise, and follow the system
    prompt's tone. Penalize hallucinated product features."
  </Step>

  <Step title="Pick models">
    Your baseline (what you run today) and up to 3 candidates. 5–500 samples.
    Optionally choose the judge (a strong instruct model is pre-selected).
  </Step>

  <Step title="Scope the prompts (optional)">
    Sample only one task's traffic by [tag](/reference/request-logging), by
    auto-detected [segment](/concepts/segments), by logged model, or from a
    saved dataset. Unscoped, a comparison mixes every kind of request you
    serve, which measures an average nobody actually runs.
  </Step>

  <Step title="Wait a few minutes">
    Runs process in the background; progress is visible live. Then open the
    report.
  </Step>
</Steps>

When a candidate **wins with a clear interval**, the result row offers
**Switch safely**: point a [model alias](/reference/model-aliases) at the
winner, behind a canary split and a
[quality gate](/judges/online-monitoring)
if you want the rollout supervised, without leaving the page.

## How many samples?

Enough that the answer means something. A win rate is a **statistic**, and a
small one carries a margin of error wide enough to hide the truth:

| Samples | A 65% win rate really means | Verdict                            |
| ------- | --------------------------- | ---------------------------------- |
| 20      | somewhere in **43–82%**     | inconclusive: could be a coin flip |
| 50      | 51–77%                      | leaning, but thin                  |
| 200     | 58–71%                      | a real result                      |
| 500     | 61–69%                      | decisive                           |

Every report shows the **95% confidence interval** next to the win rate, and
marks a run **inconclusive** when that interval straddles 50%, because a run
that can't tell the models apart should say so, not hand you a number that
*looks* like a decision.

Use 30+ to point a direction. Use **100–500 when you intend to actually switch
models** on the result. The dialog shows the estimated cost in dollars before
you commit, so you can trade confidence against spend with your eyes open.

## Judging modes

**Direct quality** (default): the judge picks whichever answer better
satisfies your rubric. Use it to ask *"is the cheap model good enough?"*

**Match reference**: the judge compares each answer against a stored
reference and picks whichever is *closer to it*. The reference is the answer
you already trust: the reply your current model actually gave (when sampling
logged traffic), or the assistant turn in a curated dataset. Use it to ask
*"does this model reproduce what we already ship?"*, the natural check for a
cheaper model, or for a fine-tune against the model it was distilled from.

## How scoring works

* **Fresh generations**: every model answers the same sampled prompts anew;
  logged responses are never reused, so the comparison is same-conditions.
* **Position-bias control**: the judge sees each pair **twice with the
  answers swapped**. A model must win *both* orderings to score a win;
  anything inconsistent counts as a tie.
* **Final answers only**: reasoning traces (`<think>` blocks) are stripped
  before judging, so thinking models are scored on what they actually answer,
  not their scratchpads.
* **Win rate** = wins + half of ties, over all judged samples, reported with a
  95% confidence interval, and flagged **inconclusive** when that interval
  straddles even odds.
* **Cost & savings**: the actual billed cost of running the *same prompts*
  on each model; savings is relative to the baseline. Judge cost is shown
  separately.

Every report includes **"View the test cases"**: each sampled prompt, both
models' answers, and the per-sample outcome, so you can see exactly what the
numbers are made of.

## From the API

Comparisons run from CI too: queue one after a deploy, poll it, and gate on
the result. `sample_filters` takes `tag`, `segment` (an auto-detected
[traffic segment](/concepts/segments)), `model`, `dataset_id`, or
`trace_replay`:

```bash theme={null}
# Queue a run scoped to one task's traffic
curl -X POST https://gateway.omnia-voice.com/v1/evals \
  -H "Authorization: Bearer $OMNIA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "support-bot: can we drop to 30B?",
    "rubric": "Accurate, concise, follows the system prompt tone. Penalize invented product features.",
    "rubric_type": "adherence",
    "baseline_model": "Qwen/Qwen3-32B",
    "candidate_models": ["Qwen/Qwen3-30B-A3B-Instruct-2507"],
    "sample_count": 30,
    "sample_filters": { "tag": "support-bot" }
  }'
# → { "id": "ev_...", "status": "PENDING", ... }

# Poll until DONE, then read the verdict
curl https://gateway.omnia-voice.com/v1/evals/ev_... \
  -H "Authorization: Bearer $OMNIA_API_KEY"
```

The report gives `winRate`, `savingsPct`, and `avgLatencyMs` per candidate;
`GET /v1/evals/{id}/samples` returns the individual test cases. See the
[Evals API reference](/api-reference/introduction).

<Note>
  Creating a run spends your wallet, so it needs an owner/admin key. Reads
  (list, get, samples) work with any workspace key.
</Note>

## Tool-using traffic

Agent traffic benchmarks first-class, not as an afterthought:

* Logged requests keep their **tool definitions**, and every generated model
  is offered the same tools, so a candidate answering agent traffic can act,
  not just talk.
* A reply that acted renders canonically as `[tool call] name(args)`: the
  same notation in stored references, generated answers, and what the judge
  reads, so an incumbent's tool call and a candidate's tool call compare
  byte-for-byte fairly.
* Two deterministic assertions gate on actions: `tool_called` (value = the
  tool name that must be invoked) and `no_tool_call`.
* The stored-answer baseline includes exchanges where the incumbent's answer
  **was** a tool call, so agent steps are no longer silently excluded from
  bake-offs.

### Whole-run replay

Against the stored-answer baseline, check **Replay whole agent runs** (or pass
`sample_filters.trace_replay: true`): completed runs are sampled whole, and at
every step the candidate sees exactly the history and tools your agent saw;
its chosen action is judged head-to-head against the recorded one. The report
adds a run-level number: **runs the candidate drove without losing a single
step**. A run is only as good as its worst action, so this can't be rescued by
twenty good steps hiding one bad one.

One honest boundary: the candidate's divergent actions are **judged, never
followed**. Replay measures action fidelity against a real trajectory, not
free-running behavior. For live whole-run quality, monitor with
[trace-unit judges](/judges/calibration).

## Limitations

<Warning>
  LLM-as-judge is a strong signal, not ground truth. Small samples (5–10) are
  directional; use 100+ for decisions, and believe the confidence interval, not
  the headline number. A judge from the same model family as a
  contestant can be mildly self-preferring: pick a judge outside both families
  when the comparison is close (the judge is selectable for exactly this
  reason). And a rubric measures what you wrote, not what
  you meant: read a few test cases before trusting a headline number.
</Warning>

Runs are resumable and billing is exactly-once: every generation and judge
call settles under a deterministic request id, so an interrupted run resumes
where it stopped without double-billing.

## Acting on the result

A won eval is only worth something if you can act on it. Point a
[model alias](/reference/model-aliases) at the winner and your **next request**
routes there. No redeploy, no code change:

```bash theme={null}
curl -X PUT https://gateway.omnia-voice.com/v1/aliases \
  -H "Authorization: Bearer $OMNIA_API_KEY" \
  -d '{"name":"support-bot","target_model":"Qwen/Qwen3-30B-A3B-Instruct-2507"}'
```

Not fully convinced? Repoint with a `canary_percent` first and move the traffic
over gradually.