> ## Documentation Index > Fetch the complete documentation index at: https://docs.omnia-voice.com/llms.txt > Use this file to discover all available pages before exploring further. # Queue an eval run > Compare a baseline model against up to 3 candidates on **your own prompts**, scored by an LLM judge. Prompts are sampled from your logged traffic (or a managed dataset). Every model generates a **fresh** answer to each prompt — the baseline included — so you're comparing models, not comparing a new answer against a stale logged one. Each pair is judged twice with the answers swapped; a candidate must win both orderings to score a win (position bias is the classic LLM-judge failure). Requires an owner/admin key: generations and judging bill your wallet as normal usage. The run is queued and processed in the background — poll `GET /v1/evals/{id}` until `status` is `DONE` or `ERROR`. Requires request logging to be on, with at least 5 distinct captured prompts matching your filter. ## OpenAPI ````yaml /openapi.json post /v1/evals openapi: 3.1.0 info: title: Omnia Management API description: >- The management API behind the improvement loop: capture and setup, request logs and datasets, grades (labels), judges (criteria), evals and deploy gates, fine-tuning and reinforcement learning, dedicated GPU endpoints, and model aliases and versions. Authenticated with a workspace API key (sk_sovereign_...). The inference API (chat, embeddings, rerank, responses) is OpenAI-compatible and documented separately. Responses are snake_case, list endpoints on the loop products use the {"object": "list", "data": [...]} envelope, and refusals use the same nested error shape the gateway emits: {"error": {"message", "type", "code"}}. Request bodies on the loop products (logs, labels, criteria, evals, datasets, aliases) are snake_case; the training and infrastructure products (fine-tuning, GRPO, environment tools, dedicated, model-version adoption) validate camelCase bodies, and each schema below says which it is. Endpoints that spend money require a key minted by a workspace owner or admin and return 403 otherwise. version: 1.0.0 servers: - url: https://gateway.omnia-voice.com description: Production - url: https://platform.omnia-voice.com/api description: Production (legacy alias — same API, older base URL) security: - bearerAuth: [] paths: /v1/evals: post: tags: - Evals summary: Queue an eval run description: >- Compare a baseline model against up to 3 candidates on **your own prompts**, scored by an LLM judge. Prompts are sampled from your logged traffic (or a managed dataset). Every model generates a **fresh** answer to each prompt — the baseline included — so you're comparing models, not comparing a new answer against a stale logged one. Each pair is judged twice with the answers swapped; a candidate must win both orderings to score a win (position bias is the classic LLM-judge failure). Requires an owner/admin key: generations and judging bill your wallet as normal usage. The run is queued and processed in the background — poll `GET /v1/evals/{id}` until `status` is `DONE` or `ERROR`. Requires request logging to be on, with at least 5 distinct captured prompts matching your filter. operationId: createEvalRun requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/CreateEvalRun' responses: '201': description: The queued run. content: application/json: schema: $ref: '#/components/schemas/EvalRun' '400': $ref: '#/components/responses/BadRequest' '401': $ref: '#/components/responses/Unauthorized' '402': $ref: '#/components/responses/InsufficientBalance' '403': $ref: '#/components/responses/Forbidden' '422': description: >- Screening only: nothing to screen. There is no (or too little) logged traffic; the error names the fix, and screening also works on imported provider logs via sample_filters.dataset_id. content: application/json: schema: $ref: '#/components/schemas/Error' components: schemas: CreateEvalRun: type: object required: - name - baseline_model - candidate_models - sample_count properties: screening: type: boolean description: >- Screening mode: the server auto-fills the run like the dashboard's one-click screening (incumbent = your dominant logged model, baseline = its stored answers, candidates = the cheapest model of each distinct family, fixed quality rubric, sample_count = min(40, population)). With screening: true every other field becomes an optional override; name, sample_count, max_output_tokens, candidate_models (1-6, validated against the catalog), and sample_filters.dataset_id are honored. default: false name: type: string maxLength: 80 example: 'support-bot: 32B vs 235B' rubric: type: string minLength: 10 maxLength: 2000 description: >- What "good" means for this task. The judge uses it verbatim. Required for comparison runs; ignored for criterion runs (the criterion's judge prompt is the rubric). example: >- Answers must be accurate, concise, and follow the system prompt's tone. Penalize hallucinated product features. rubric_type: type: string enum: - direct - adherence default: direct description: >- direct: which answer better satisfies the rubric. adherence: which answer is closer to the stored reference (the logged reply, or the dataset's assistant turn) — use this to check a cheaper or fine-tuned model against answers you already trust. baseline_model: type: string description: >- The model you run today — or the literal `__stored__` to use your logged answers as the baseline arm (the certified-switch shape): candidates are judged against what you actually shipped, a calibrated judge criterion is required, population matching is strict (a mismatched judge/sample population refuses instead of warning), and the gate applies a noninferiority margin scaled by the judge's measured error. An underpowered run answers 412 rather than concluding. example: Qwen/Qwen3-32B candidate_models: type: array minItems: 1 maxItems: 6 items: type: string example: - Qwen/Qwen3-30B-A3B-Instruct-2507 description: >- Up to 6. Every candidate answers every sample and is judged, so cost is linear in this number. Comparison runs need at least 1; a criterion run may grade the baseline alone. sample_count: type: integer minimum: 5 maximum: 500 example: 20 judge_model: type: string description: >- Defaults to a strong instruct model. Prefer a judge from a different family than your candidates — LLM judges measurably favour their own family. sample_filters: type: object description: >- Which prompts to run. Omit to sample recent logged traffic. Keys: tag (X-Omnia-Tag), model (sample only that model's traffic), segment (an auto-detected traffic segment, a `segment` value from GET /v1/logs), dataset_id (sample a managed dataset, e.g. a holdout split, instead of live logs), trace_replay (replay bake-off: sample whole completed agent runs, one teacher-forced sample per step; requires baseline_model "__stored__"). properties: tag: type: string description: The X-Omnia-Tag your requests carried. example: support-bot model: type: string description: Only prompts originally served by this model. segment: type: string description: >- Auto-detected traffic segment (a `segment` value from GET /v1/logs — the family of system prompts sharing one template) — sample only that application surface's traffic. Composes with tag. dataset_id: type: string description: Sample a managed dataset instead of live logs. trace_replay: type: boolean description: >- Sample whole completed agent runs, one teacher-forced sample per step; the recorded actions are the incumbent. Requires baseline_model "__stored__". Results gain per-candidate replay {traces, clean_runs}. eval_kind: type: string enum: - comparison - criterion default: comparison description: >- comparison: pairwise judging of candidates against the baseline. criterion: every model is graded pass/fail against a validated criterion, reported with the Rogan–Gladen correction from your alignment run. criterion_id: type: string description: >- Required when eval_kind is criterion. The run freezes the criterion's judge prompt, judge model, and alignment metrics at creation — a later re-alignment never rewrites a finished run's corrected rates. assertions: type: array maxItems: 10 description: >- Deterministic checks run against every generated response at finalize (final answer, reasoning stripped). Free, exact, and CI-gateable via GET /v1/evals/{id}/gate. Configs are validated at create time: an uncompilable regex or unparseable schema is a 400, never a queued-broken run. items: type: object required: - type properties: type: type: string enum: - json_valid - json_schema - regex_match - contains - not_contains - max_length - min_length - completed - tool_called - no_tool_call value: type: string description: >- Pattern / substring / length bound / stringified JSON Schema (subset: type, required, properties, items, enum). json_valid takes no value. description: >- tool_called (value = tool name) and no_tool_call check the canonical action notation — generations offer the sample's tool definitions and every tool call renders as "[tool call] name(args)". max_output_tokens: type: integer minimum: 256 maximum: 16384 default: 4096 description: >- Per-answer generation output cap. Sized by default so thinking models can finish reasoning AND answer — a candidate cut mid-reasoning loses comparisons for reasons unrelated to quality. The cost estimate shown before a run uses your traffic's MEASURED median tokens; this cap bounds the worst case. description: >- Snake_case body. Comparison runs also require rubric. Criterion runs require criterion_id. With screening: true, no field is required at all. EvalRun: type: object properties: id: type: string name: type: string rubric: type: string description: What "good" means, in plain English. The judge scores against this. rubric_type: type: string enum: - direct - adherence eval_kind: type: string description: >- comparison: pairwise judging against the baseline. criterion: a saved judge grades each model pass/fail. enum: - comparison - criterion criterion_snapshot: type: - object - 'null' description: >- Criterion runs only: the judge frozen at creation (name, prompt, and its TPR/TNR/kappa at that moment), so the run's corrected rates stay reproducible after the judge changes. baseline_model: type: string candidate_models: type: array items: type: string judge_model: type: string sample_count: type: integer gen_max_output_tokens: type: integer description: >- Per-answer generation cap the run was created with (see max_output_tokens on create). sample_filters: type: object description: >- Where the prompts came from: tag, model, segment, dataset_id, trace_replay, and screening (set on screening runs). Echoes what the run was created with. status: type: string enum: - PENDING - RUNNING - DONE - ERROR - CANCELLED error: type: - string - 'null' assertions: type: - array - 'null' description: >- The deterministic assertion configs the run was created with, when any. progress_ratio: type: number description: 0–1. Completed inference calls over total. results: type: - object - 'null' description: >- Present when status is DONE. Comparison runs: sample_count, clipped_samples, baseline {model, stored_answers, truncated, avg_latency_ms, eval_cost_micros}, per_candidate [{model, win_rate, wins, ties, losses, ci95, failed, attempted, unreportable, truncated, avg_latency_ms, eval_cost_micros, savings_pct, replay?}], judge_cost_micros, and screening (screening runs only: incumbent, token_shape, per_candidate similarity + estimated cost placement, recommendation). Criterion runs: per_model [{model, observed_pass_rate, observed_ci, corrected_pass_rate, corrected_ci, ...}] using the frozen judge's measured error rates. Assertion runs add assertion_results. created_at: type: string format: date-time Error: type: object properties: error: type: object properties: message: type: string type: type: string description: >- invalid_request_error, insufficient_quota, rate_limit_error, or api_error. code: type: string description: >- Machine-stable cause, e.g. invalid_api_key, not_found, insufficient_permissions, precondition_failed. required: - message - type - code description: >- Every refusal — gateway and management API alike — uses this one envelope. responses: BadRequest: description: Malformed request or invalid field. content: application/json: schema: $ref: '#/components/schemas/Error' Unauthorized: description: Missing, malformed, or revoked API key. content: application/json: schema: $ref: '#/components/schemas/Error' example: error: message: Invalid API key type: invalid_request_error code: invalid_api_key InsufficientBalance: description: Your wallet can't cover the required prepaid runway. content: application/json: schema: $ref: '#/components/schemas/Error' example: error: message: >- Insufficient balance: deploying this endpoint requires at least 1h of runway. Top up and try again. type: insufficient_quota code: insufficient_balance Forbidden: description: The key lacks the required owner/admin permission. content: application/json: schema: $ref: '#/components/schemas/Error' securitySchemes: bearerAuth: type: http scheme: bearer description: >- Your workspace API key, e.g. `sk_sovereign_...`, sent as `Authorization: Bearer `. ````