Queue an eval run
Compare a baseline model against up to 3 candidates on your own prompts, scored by an LLM judge.
Prompts are sampled from your logged traffic (or a managed dataset). Every model generates a fresh answer to each prompt — the baseline included — so you’re comparing models, not comparing a new answer against a stale logged one. Each pair is judged twice with the answers swapped; a candidate must win both orderings to score a win (position bias is the classic LLM-judge failure).
Requires an owner/admin key: generations and judging bill your wallet as normal usage. The run is queued and processed in the background — poll GET /v1/evals/{id} until status is DONE or ERROR.
Requires request logging to be on, with at least 5 distinct captured prompts matching your filter.
Authorizations
Your workspace API key, e.g. sk_sovereign_..., sent as Authorization: Bearer <key>.
Body
Snake_case body. Comparison runs also require rubric. Criterion runs require criterion_id. With screening: true, no field is required at all.
80"support-bot: 32B vs 235B"
The model you run today — or the literal __stored__ to use your logged answers as the baseline arm (the certified-switch shape): candidates are judged against what you actually shipped, a calibrated judge criterion is required, population matching is strict (a mismatched judge/sample population refuses instead of warning), and the gate applies a noninferiority margin scaled by the judge's measured error. An underpowered run answers 412 rather than concluding.
"Qwen/Qwen3-32B"
Up to 6. Every candidate answers every sample and is judged, so cost is linear in this number. Comparison runs need at least 1; a criterion run may grade the baseline alone.
1 - 6 elements5 <= x <= 50020
Screening mode: the server auto-fills the run like the dashboard's one-click screening (incumbent = your dominant logged model, baseline = its stored answers, candidates = the cheapest model of each distinct family, fixed quality rubric, sample_count = min(40, population)). With screening: true every other field becomes an optional override; name, sample_count, max_output_tokens, candidate_models (1-6, validated against the catalog), and sample_filters.dataset_id are honored.
What "good" means for this task. The judge uses it verbatim. Required for comparison runs; ignored for criterion runs (the criterion's judge prompt is the rubric).
10 - 2000"Answers must be accurate, concise, and follow the system prompt's tone. Penalize hallucinated product features."
direct: which answer better satisfies the rubric. adherence: which answer is closer to the stored reference (the logged reply, or the dataset's assistant turn) — use this to check a cheaper or fine-tuned model against answers you already trust.
direct, adherence Defaults to a strong instruct model. Prefer a judge from a different family than your candidates — LLM judges measurably favour their own family.
Which prompts to run. Omit to sample recent logged traffic. Keys: tag (X-Omnia-Tag), model (sample only that model's traffic), segment (an auto-detected traffic segment, a segment value from GET /v1/logs), dataset_id (sample a managed dataset, e.g. a holdout split, instead of live logs), trace_replay (replay bake-off: sample whole completed agent runs, one teacher-forced sample per step; requires baseline_model "stored").
comparison: pairwise judging of candidates against the baseline. criterion: every model is graded pass/fail against a validated criterion, reported with the Rogan–Gladen correction from your alignment run.
comparison, criterion Required when eval_kind is criterion. The run freezes the criterion's judge prompt, judge model, and alignment metrics at creation — a later re-alignment never rewrites a finished run's corrected rates.
Deterministic checks run against every generated response at finalize (final answer, reasoning stripped). Free, exact, and CI-gateable via GET /v1/evals/{id}/gate. Configs are validated at create time: an uncompilable regex or unparseable schema is a 400, never a queued-broken run.
10Per-answer generation output cap. Sized by default so thinking models can finish reasoning AND answer — a candidate cut mid-reasoning loses comparisons for reasons unrelated to quality. The cost estimate shown before a run uses your traffic's MEASURED median tokens; this cap bounds the worst case.
256 <= x <= 16384Response
The queued run.
What "good" means, in plain English. The judge scores against this.
direct, adherence comparison: pairwise judging against the baseline. criterion: a saved judge grades each model pass/fail.
comparison, criterion Criterion runs only: the judge frozen at creation (name, prompt, and its TPR/TNR/kappa at that moment), so the run's corrected rates stay reproducible after the judge changes.
Per-answer generation cap the run was created with (see max_output_tokens on create).
Where the prompts came from: tag, model, segment, dataset_id, trace_replay, and screening (set on screening runs). Echoes what the run was created with.
PENDING, RUNNING, DONE, ERROR, CANCELLED The deterministic assertion configs the run was created with, when any.
0–1. Completed inference calls over total.
Present when status is DONE. Comparison runs: sample_count, clipped_samples, baseline {model, stored_answers, truncated, avg_latency_ms, eval_cost_micros}, per_candidate [{model, win_rate, wins, ties, losses, ci95, failed, attempted, unreportable, truncated, avg_latency_ms, eval_cost_micros, savings_pct, replay?}], judge_cost_micros, and screening (screening runs only: incumbent, token_shape, per_candidate similarity + estimated cost placement, recommendation). Criterion runs: per_model [{model, observed_pass_rate, observed_ci, corrected_pass_rate, corrected_ci, ...}] using the frozen judge's measured error rates. Assertion runs add assertion_results.