Align a criterion against your labels
Judge every labeled trace with this criterion and measure agreement: TPR and TNR with 95% Wilson intervals, plus Cohen’s κ — never raw agreement (class imbalance inflates it). Needs at least 30 labels (aim for 100). Billed as judging; the confusion matrix is stored row by row so every disagreement stays auditable.
This number is what turns “the judge said 80%” into “the judge says 80%, agrees with YOU 91% of the time, so the corrected rate is 85% [79–90%]”.
Authorizations
Your workspace API key, e.g. sk_sovereign_..., sent as Authorization: Bearer <key>.
Path Parameters
Response
Label sets of 50 or fewer are judged synchronously and return the AlignmentReport. Larger sets return AlignmentQueued ({queued: true, total_labels}) and a background worker builds the report over the following minutes; poll GET /v1/criteria/{id} for the metrics and GET /v1/criteria/{id}/alignment for the full report.
- Option 1
- Option 2
Returned by POST /v1/criteria/{id}/align for label sets of 50 or fewer (larger sets return AlignmentQueued and the report is built over the following minutes).
The population measured — the criterion's tag ('' = all labeled traffic). Alignment numbers only describe the population they were measured on.
The auto-detected traffic segment measured ("" = no segment scope). The second population handle, same binding rule as scope_tag.
What the measured labels were made of, by tag ('' = untagged). One entry for a scoped criterion; the pollution report for an unscoped one.
True when an UNSCOPED criterion was measured across more than one tag — the metrics describe a blend of tasks, not any one of them. Scope the criterion and re-align.
Fails attributed to other criteria: in scope but excluded from this criterion's calibration set by cause attribution.
In-population fails with no cause attribution. They still count as fails for every criterion; attributing them tightens narrow judges' numbers.
n, tpr, tnr, kappa and the 95% Wilson intervals (tprCi, tnrCi; camelCase on this payload). tpr/tnr are null when the label set has no human fails/passes to measure against.
aligned, weak, misaligned, unmeasured True under 100 judgeable labels: intervals stay wide; label more.
Labeled traces whose content aged out, or whose judge verdict was unparseable twice.
Present at 80+ judged labels: {tune_n, report_n}. Metrics above come from the report half only; tune-half rows exist for prompt iteration.
Set when null TPR/TNR is a one-class measured set (a labeling gap), naming the missing class, so it is never mistaken for "never aligned" or a broken judge.