Skip to main content
Evals answer the question every team eventually asks: “could a smaller, cheaper model handle our traffic?”, with evidence from your own prompts instead of generic benchmarks. In the dashboard these are comparisons (eval runs in the API), and they live on the Comparisons tab of the Evals page.
Pairwise comparisons are half the story. For judges calibrated against your own grades, corrected pass rates, CI deploy gates, and the quality-gated canary, see Judges & calibration; the Evals page’s Judges and Live switches tabs.
A comparison samples real requests from your logged traffic, generates fresh answers on your current model (the baseline) and one or more candidate models under identical conditions, and has an LLM judge score each pair against a rubric you write in plain English. The result is a savings report: quality delta, latency delta, and cost delta, side by side.

​
Requirements

  • Request logging enabled, with at least 5 distinct prompts captured (comparisons sample from your logs).
  • Wallet balance: eval generations and judge calls bill as normal usage, disclosed before you start a run.

​
Running a comparison

Dashboard → Evals → Comparisons → Run a comparison (owner/admin). The tab is table-first: every past comparison is a row with its result inline, and the guided flow walks you through:
1

Write a rubric

Describe what “good” means for your traffic, in plain English, e.g. “Answers must be factually correct, concise, and follow the system prompt’s tone. Penalize hallucinated product features.”
2

Pick models

Your baseline (what you run today) and up to 3 candidates. 5–500 samples. Optionally choose the judge (a strong instruct model is pre-selected).
3

Scope the prompts (optional)

Sample only one task’s traffic by tag, by auto-detected segment, by logged model, or from a saved dataset. Unscoped, a comparison mixes every kind of request you serve, which measures an average nobody actually runs.
4

Wait a few minutes

Runs process in the background; progress is visible live. Then open the report.
When a candidate wins with a clear interval, the result row offers Switch safely: point a model alias at the winner, behind a canary split and a quality gate if you want the rollout supervised, without leaving the page.

​
How many samples?

Enough that the answer means something. A win rate is a statistic, and a small one carries a margin of error wide enough to hide the truth: Every report shows the 95% confidence interval next to the win rate, and marks a run inconclusive when that interval straddles 50%, because a run that can’t tell the models apart should say so, not hand you a number that looks like a decision. Use 30+ to point a direction. Use 100–500 when you intend to actually switch models on the result. The dialog shows the estimated cost in dollars before you commit, so you can trade confidence against spend with your eyes open.

​
Judging modes

Direct quality (default): the judge picks whichever answer better satisfies your rubric. Use it to ask “is the cheap model good enough?” Match reference: the judge compares each answer against a stored reference and picks whichever is closer to it. The reference is the answer you already trust: the reply your current model actually gave (when sampling logged traffic), or the assistant turn in a curated dataset. Use it to ask “does this model reproduce what we already ship?”, the natural check for a cheaper model, or for a fine-tune against the model it was distilled from.

​
How scoring works

  • Fresh generations: every model answers the same sampled prompts anew; logged responses are never reused, so the comparison is same-conditions.
  • Position-bias control: the judge sees each pair twice with the answers swapped. A model must win both orderings to score a win; anything inconsistent counts as a tie.
  • Final answers only: reasoning traces (<think> blocks) are stripped before judging, so thinking models are scored on what they actually answer, not their scratchpads.
  • Win rate = wins + half of ties, over all judged samples, reported with a 95% confidence interval, and flagged inconclusive when that interval straddles even odds.
  • Cost & savings: the actual billed cost of running the same prompts on each model; savings is relative to the baseline. Judge cost is shown separately.
Every report includes “View the test cases”: each sampled prompt, both models’ answers, and the per-sample outcome, so you can see exactly what the numbers are made of.

​
From the API

Comparisons run from CI too: queue one after a deploy, poll it, and gate on the result. sample_filters takes tag, segment (an auto-detected traffic segment), model, dataset_id, or trace_replay:
The report gives winRate, savingsPct, and avgLatencyMs per candidate; GET /v1/evals/{id}/samples returns the individual test cases. See the Evals API reference.
Creating a run spends your wallet, so it needs an owner/admin key. Reads (list, get, samples) work with any workspace key.

​
Tool-using traffic

Agent traffic benchmarks first-class, not as an afterthought:
  • Logged requests keep their tool definitions, and every generated model is offered the same tools, so a candidate answering agent traffic can act, not just talk.
  • A reply that acted renders canonically as [tool call] name(args): the same notation in stored references, generated answers, and what the judge reads, so an incumbent’s tool call and a candidate’s tool call compare byte-for-byte fairly.
  • Two deterministic assertions gate on actions: tool_called (value = the tool name that must be invoked) and no_tool_call.
  • The stored-answer baseline includes exchanges where the incumbent’s answer was a tool call, so agent steps are no longer silently excluded from bake-offs.

​
Whole-run replay

Against the stored-answer baseline, check Replay whole agent runs (or pass sample_filters.trace_replay: true): completed runs are sampled whole, and at every step the candidate sees exactly the history and tools your agent saw; its chosen action is judged head-to-head against the recorded one. The report adds a run-level number: runs the candidate drove without losing a single step. A run is only as good as its worst action, so this can’t be rescued by twenty good steps hiding one bad one. One honest boundary: the candidate’s divergent actions are judged, never followed. Replay measures action fidelity against a real trajectory, not free-running behavior. For live whole-run quality, monitor with trace-unit judges.

​
Limitations

LLM-as-judge is a strong signal, not ground truth. Small samples (5–10) are directional; use 100+ for decisions, and believe the confidence interval, not the headline number. A judge from the same model family as a contestant can be mildly self-preferring: pick a judge outside both families when the comparison is close (the judge is selectable for exactly this reason). And a rubric measures what you wrote, not what you meant: read a few test cases before trusting a headline number.
Runs are resumable and billing is exactly-once: every generation and judge call settles under a deterministic request id, so an interrupted run resumes where it stopped without double-billing.

​
Acting on the result

A won eval is only worth something if you can act on it. Point a model alias at the winner and your next request routes there. No redeploy, no code change:
Not fully convinced? Repoint with a canary_percent first and move the traffic over gradually.