> ## Documentation Index
> Fetch the complete documentation index at: https://docs.omnia-voice.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Case study: catching a support bot that lies

> A real walkthrough — how validated evals found confident fabrication in a production assistant, and how a prompt change fixed it without changing models.

This is a real investigation run on Omnia's own support assistant, start to
finish, using the tools described in [Grading](/judges/grading) and
[Judges & calibration](/judges/calibration).
It's here because the failure it found is the most common one in production
LLM apps, and the *hardest to see*: **every request returned HTTP 200.**

## The symptom: nothing looked wrong

Twenty real support questions went through the gateway. Twenty 200s. Latency
normal, error rate zero, no alerts. By every operational metric the bot was
healthy.

## What the grades showed

We wrote one judge (a criterion in the API), *does the answer invent Omnia
facts it cannot know?*, and hand-graded 40 real answers (20 questions × 2
models) in the Review queue.

**15 of 40 were fabrications.** Not truncations, not errors: fluent,
confident, plausible lies:

* Invented four AWS-style regions the platform doesn't have.
* Invented a base URL (`api.omn.ai`) that doesn't exist.
* Invented a refund policy.
* **Flatly denied having SOC 2**: an answer that would sink a vendor review.
* Invented HTTP response headers for latency debugging.

Pass rates by human judgment: **Llama-3.3-70B 75%, Qwen3-30B 50%.**

This is what "operationally healthy" looked like.

## The judge had to be calibrated first

Before trusting any automated grading, we calibrated the judge against those
grades. It came back **weakly calibrated**: TPR 76%, κ 0.59. The judge
missed a quarter of the fabrications a human caught.

That measurement is what makes the number usable. An uncalibrated judge reporting "80%
pass" is a number with no error bar and no accountability. Only after
sharpening the judge prompt (specifically teaching it that *fluent and
plausible is not the same as true*, and that honest hedging is a PASS) did
it become trustworthy on this traffic.

<Note>
  Calibrate a judge against the traffic it will actually judge. A judge measured
  against grades from a different task reports a trust score for a population it
  will never see.
</Note>

## The fix was not a better model

The obvious reaction to "the cheap model fabricates twice as often" is to buy
the expensive one. That would have been wrong, and expensive.

The root cause was the system prompt. It listed facts, but never told the
model **what to do when a question fell outside them**. Faced with a gap, a
helpful model fills it. That's not a bug in the model; it's an unspecified
contract.

The fix is a **closed-world rule** plus a refusal template:

```text theme={null}
KNOWN FACTS — the COMPLETE set of Omnia-specific facts you have:
- The API is OpenAI-compatible.
- Billing: prepaid wallet, pay-per-token. No monthly subscription.
- Request logging is OPT-IN and off by default.
- Dedicated GPU endpoints exist, billed per GPU-hour.
- There is NO self-hosted version.

CLOSED-WORLD RULE — your most important instruction:
Any Omnia-specific detail NOT in KNOWN FACTS is UNKNOWN to you. Never invent,
estimate, or infer it from how other platforms work.

This applies especially to CAPABILITY questions ("do you support X?"). Just
because a feature is standard elsewhere does NOT mean you know Omnia has it.
Answering "Yes, we support that" about an unlisted capability is a FABRICATION,
even if the feature is common.

For anything UNKNOWN, respond in this shape:
"I don't have confirmed information about <X>. Please check the docs or contact
support/sales, who can give you an authoritative answer."

A confident wrong answer is far worse than saying you don't know. "I don't
know" is CORRECT and expected.
```

Two details earned their place the hard way:

1. **The capability clause.** Without it, the cheap model still asserted
   "Yes, Omnia supports function calling and streaming together": a feature
   question that *sounds* like standard API behavior. Naming that failure mode
   explicitly closed the last leak.
2. **"'I don't know' is CORRECT."** Without this, judges and models alike
   treat hedging as failure, and you train the bot back into guessing.

## The result

Same models. Same questions. Only the prompt changed:

| Model         | Before | After    |
| ------------- | ------ | -------- |
| Llama-3.3-70B | 75%    | **100%** |
| Qwen3-30B     | 50%    | **95%**  |

The cheap model went from a liability to production-viable, and became
**statistically indistinguishable from the model 2× its price.** The expensive
model was never buying safety. A specified contract was.

Crucially, the bot did not become useless: it still answers every question it
legitimately knows, and defers only where it genuinely can't know.

## The lesson

Three things this investigation depended on, in order:

1. **Grades.** No amount of tooling substitutes for a human reading real
   traffic. Fifteen fabrications were invisible to every dashboard.
2. **A calibrated judge.** The first judge missed a quarter of them. Measuring
   the judge is what separates a number from a fact.
3. **Fixing the cause, not the symptom.** The evidence pointed at the prompt,
   not the model, and the comparison proved the fix instead of assuming it.

Keep the graded traces: grades are permanent, and they are now a regression
suite: any future prompt or model change can be graded against them before it
reaches a customer.