Skip to main content
POST
Create a criterion

Authorizations

Authorization
string
header
required

Your workspace API key, e.g. sk_sovereign_..., sent as Authorization: Bearer <key>.

Body

application/json
name
string
required
Maximum string length: 80
judge_prompt
string
required
Required string length: 10 - 4000
Example:

"FAIL any response that never delivers a final answer — e.g. it consists only of reasoning, stops mid-sentence, or is cut off before answering."

judge_model
string
required
Example:

"Qwen/Qwen3-235B-A22B-Instruct-2507"

description
string | null
Maximum string length: 500
unit
enum<string>

Create-only. Trace-unit judges read whole runs and align against whole-run labels.

Available options:
request,
trace
population
string

Tag scoping what the judge scores online and which labels it aligns against.

population_family
string

Auto-detected traffic segment scoping what the judge scores online and which grades it calibrates against — a segment value from GET /v1/logs; '' = no segment scope. Composes with population (tag).

Response

The draft criterion.

id
string
name
string
Example:

"Complete final answer"

description
string | null
judge_prompt
string

The binary instruction the judge applies. State precisely what makes a response FAIL.

judge_model
string
status
enum<string>

Lifecycle only: draft (in use) or retired (excluded from online scoring and pickers; set via PATCH). Alignment QUALITY is never stored here — read tier, tpr, tnr, and aligned_at. A criterion can be fully aligned and still say status=draft; that is by design, not a missing promotion.

Available options:
draft,
retired
source
enum<string>
Available options:
human,
assist_suggested
tier
enum<string>

LEGACY point-estimate badge, kept for continuity. Computed by comparing the raw TPR/TNR to fixed thresholds, so it cannot distinguish 9-of-10 from 90-of-100. Read trust instead — that is what every gate uses.

Available options:
aligned,
weak,
misaligned,
unmeasured
trust
enum<string>

What this judge's evidence PROVES, derived from the 95% Wilson interval rather than the point estimate. trustworthy = both lower bounds clear the 90% bar. under-measured = straddles it, and more grading would settle it (see fail_grades_needed / pass_grades_needed). borderline = straddles it with the rate hugging the bar, so more grading will NOT settle it — narrow what the judge asks instead. misaligned = an upper bound sits below the bar, i.e. proven wrong rather than merely unproven. unmeasured = no interval at all (never calibrated, or one-class grades). Auto-routing and RL rewards require trustworthy; dataset curation also accepts under-measured.

Available options:
trustworthy,
under-measured,
borderline,
misaligned,
unmeasured
tpr_ci
array | null

95% Wilson interval [low, high] for tpr — the evidence trust was derived from. Null when no human-failed grades exist. A tpr of 1.0 with tpr_ci [0.646, 1.0] means the judge caught every failure it was shown, and was shown seven.

tnr_ci
array | null

95% Wilson interval [low, high] for tnr. Null when no human-passed grades exist.

fail_grades_needed
integer | null

Additional grades OF FAILING TRACES needed for TPR's lower bound to clear the bar, assuming the judge keeps its observed rate. TPR is measured only on human-failed rows, so grading more passes does not move it. Null when already proven, already below the bar, or too close to it for grading to help.

pass_grades_needed
integer | null

The same for TNR, measured only on human-passed rows.

tpr
number | null

P(judge says fail | you said fail) — how often it catches real failures.

tnr
number | null

P(judge says pass | you said pass).

kappa
number | null

Cohen's κ — chance-corrected agreement (raw agreement is inflated by class imbalance).

alignment_n
integer | null
aligned_at
string<date-time> | null
created_at
string<date-time>
unit
enum<string>

What this judge grades: one exchange, or a whole agent run / trace-grouped conversation. Fixed at creation.

Available options:
request,
trace
population
string

The tag this criterion judges online AND aligns against ('' = all logged traffic). Changing it voids alignment.

population_family
string

The auto-detected traffic segment this judge is scoped to ('' = none): a segment value from GET /v1/logs. Scopes what it judges online AND which grades it calibrates against, together with population (tag) and unit. Changing it voids calibration.

online_enabled
boolean

Whether the online monitor is scoring a sample of fresh logged traffic with this judge.

online_percent
integer

Percent of eligible fresh traffic the online monitor samples — deterministic, never double-judging a request.

drift_status
enum<string>

flagged = the judge may have drifted on live traffic (stale calibration or an anomalous corrected-quality drop); re-calibrate on fresh grades.

Available options:
ok,
flagged
online_cap_usd
number

Weekly online-judging spend ceiling in USD (0 = uncapped). Reaching it pauses this criterion's online scoring until the week rolls over.

online_spent_usd
number

Online judging spend in the current weekly window.

drift_signal
enum<string> | null

Which signal raised the flag: "stale" (calibration aged out) or "quality_drop" (live corrected rate fell below what the judge validated at). Null when drift_status is ok.

Available options:
stale,
quality_drop,
null
drift_reason
string | null

Human-readable receipt behind a flagged drift_status.

drift_checked_at
string | null

When the drift sweep last examined this judge; null for a judge it has never reached.