Get an eval run
Status and, once status is DONE, the report: per-candidate win rate, W/T/L, average latency, cost, and savings versus the baseline on the same prompts. Poll this after queueing a run.
Authorizations
Your workspace API key, e.g. sk_sovereign_..., sent as Authorization: Bearer <key>.
Path Parameters
Response
The run.
What "good" means, in plain English. The judge scores against this.
direct, adherence comparison: pairwise judging against the baseline. criterion: a saved judge grades each model pass/fail.
comparison, criterion Criterion runs only: the judge frozen at creation (name, prompt, and its TPR/TNR/kappa at that moment), so the run's corrected rates stay reproducible after the judge changes.
Per-answer generation cap the run was created with (see max_output_tokens on create).
Where the prompts came from: tag, model, segment, dataset_id, trace_replay, and screening (set on screening runs). Echoes what the run was created with.
PENDING, RUNNING, DONE, ERROR, CANCELLED The deterministic assertion configs the run was created with, when any.
0–1. Completed inference calls over total.
Present when status is DONE. Comparison runs: sample_count, clipped_samples, baseline {model, stored_answers, truncated, avg_latency_ms, eval_cost_micros}, per_candidate [{model, win_rate, wins, ties, losses, ci95, failed, attempted, unreportable, truncated, avg_latency_ms, eval_cost_micros, savings_pct, replay?}], judge_cost_micros, and screening (screening runs only: incumbent, token_shape, per_candidate similarity + estimated cost placement, recommendation). Criterion runs: per_model [{model, observed_pass_rate, observed_ci, corrected_pass_rate, corrected_ci, ...}] using the frozen judge's measured error rates. Assertion runs add assertion_results.