Skip to main content
This is a real investigation run on Omnia’s own support assistant, start to finish, using the tools described in Grading and Judges & calibration. It’s here because the failure it found is the most common one in production LLM apps, and the hardest to see: every request returned HTTP 200.

​
The symptom: nothing looked wrong

Twenty real support questions went through the gateway. Twenty 200s. Latency normal, error rate zero, no alerts. By every operational metric the bot was healthy.

​
What the grades showed

We wrote one judge (a criterion in the API), does the answer invent Omnia facts it cannot know?, and hand-graded 40 real answers (20 questions × 2 models) in the Review queue. 15 of 40 were fabrications. Not truncations, not errors: fluent, confident, plausible lies:
  • Invented four AWS-style regions the platform doesn’t have.
  • Invented a base URL (api.omn.ai) that doesn’t exist.
  • Invented a refund policy.
  • Flatly denied having SOC 2: an answer that would sink a vendor review.
  • Invented HTTP response headers for latency debugging.
Pass rates by human judgment: Llama-3.3-70B 75%, Qwen3-30B 50%. This is what “operationally healthy” looked like.

​
The judge had to be calibrated first

Before trusting any automated grading, we calibrated the judge against those grades. It came back weakly calibrated: TPR 76%, κ 0.59. The judge missed a quarter of the fabrications a human caught. That measurement is what makes the number usable. An uncalibrated judge reporting “80% pass” is a number with no error bar and no accountability. Only after sharpening the judge prompt (specifically teaching it that fluent and plausible is not the same as true, and that honest hedging is a PASS) did it become trustworthy on this traffic.
Calibrate a judge against the traffic it will actually judge. A judge measured against grades from a different task reports a trust score for a population it will never see.

​
The fix was not a better model

The obvious reaction to “the cheap model fabricates twice as often” is to buy the expensive one. That would have been wrong, and expensive. The root cause was the system prompt. It listed facts, but never told the model what to do when a question fell outside them. Faced with a gap, a helpful model fills it. That’s not a bug in the model; it’s an unspecified contract. The fix is a closed-world rule plus a refusal template:
Two details earned their place the hard way:
  1. The capability clause. Without it, the cheap model still asserted “Yes, Omnia supports function calling and streaming together”: a feature question that sounds like standard API behavior. Naming that failure mode explicitly closed the last leak.
  2. “‘I don’t know’ is CORRECT.” Without this, judges and models alike treat hedging as failure, and you train the bot back into guessing.

​
The result

Same models. Same questions. Only the prompt changed: The cheap model went from a liability to production-viable, and became statistically indistinguishable from the model 2× its price. The expensive model was never buying safety. A specified contract was. Crucially, the bot did not become useless: it still answers every question it legitimately knows, and defers only where it genuinely can’t know.

​
The lesson

Three things this investigation depended on, in order:
  1. Grades. No amount of tooling substitutes for a human reading real traffic. Fifteen fabrications were invisible to every dashboard.
  2. A calibrated judge. The first judge missed a quarter of them. Measuring the judge is what separates a number from a fact.
  3. Fixing the cause, not the symptom. The evidence pointed at the prompt, not the model, and the comparison proved the fix instead of assuming it.
Keep the graded traces: grades are permanent, and they are now a regression suite: any future prompt or model change can be graded against them before it reaches a customer.