Skip to main content
Assists get a judge from draft to calibrated faster, never one-shot automation. Every assist amplifies grades you already made, and you stay the judge: nothing here grades anything on its own authority, and adopting any suggestion stays a human decision.
  • Suggest from critiques (POST /v1/criteria/suggest) clusters your failure critiques (10+ required) into at most 5 draft judges: your failure taxonomy, written for you, reviewed by you. A draft whose name already exists is skipped, never overwritten.
  • Scan traffic (POST /v1/criteria/{id}/scan) runs a judge over recent ungraded traffic and queues suspected failures for one-click confirm/dismiss in the Review queue. Confirming creates a real grade, so the calibration set grows itself; dismissing means it never resurfaces. A segment- or tag-scoped judge scans only its own population.
  • Auto-improve (POST /v1/criteria/{id}/auto_improve) mines the last calibration’s tune-half disagreements, rewrites the judge prompt, and creates a successor draft with its calibration queued; the report half stays untouched, so the next measurement is honest. It requires a holdout-scale calibration (80+ judged grades). In the dashboard, Auto-improve runs a budgeted multi-round loop; over the API each call is exactly one iteration. Deliberate, so every round of spend is a consented act. Adopting a successor stays a human decision.
  • Preview an edit (dashboard: the judge’s prompt lab): before a prompt edit lands, the candidate is judged against your existing grades (tune half only at holdout scale; the measured half stays untouched) and shown next to the live prompt on the same rows, with CIs. Saving still voids the calibration (a projection is a forecast, not a measurement), but the save records which forecast it adopted, and every preview is counted: iterate against the same grades too often and the report says so.
  • Draft a split (dashboard: offered when calibration shows the bundled-criterion signature: agreement that differs sharply across strata): two child judges are drafted from your rulebook and critiques, and projected against your existing compound grades. Only the conjunction (fails if either child fails) is scored; a compound fail can’t say which half failed, so per-child κ is deliberately not invented. Instead you get the fail decomposition (caught by A alone / B alone / both / neither), and creation is blocked while any fail escapes both children.
All of these spend the wallet like any judging, and none of them grades anything on its own authority.