Every accuracy figure QuietNode publishes for QuietLens comes from one place: a golden-question eval that gates CI on every commit — 121 frozen cases, 121/121 passing, zero grounding violations. This page is the methodology behind that number: what the cases pin, how the harness stays deterministic, and the one command that reproduces the score.
A business user forgives “I can't see that yet.” They never forgive an invented revenue figure, because it launders a guess into something that looks probed. So the product's core promise is not “smart answers” — it's that every number, date, and system name traces to evidence the engine actually gathered, or the engine says it can't answer. The eval exists to prove, on every commit, that this holds.
The system prompt asks the model not to invent figures; the grounding layer (lib/llm/grounding.ts) makes it a guarantee — any sentence carrying a numeric or date token that doesn’t trace to gathered evidence is stripped before the answer is ever returned. Prompts drift between model versions; a token-set intersection does not.
After every run, the harness re-scans each substantive answer: it builds the set of numeric tokens allowed by the probed evidence and the user’s question, then flags any token in the narrative outside that set — using the same token pattern and normalization as the production enforcement. The gate tests what ships, not a lookalike.
The moment the violation count is non-zero, the run exits non-zero: “HARD FAIL: grounding violations are never acceptable.” Refusals and clarifications are exempt by construction — they carry no probed claims. The rule is narrow and absolute: a substantive answer may never contain a figure it cannot trace.
The golden set only grows: every user thumbs-down becomes a new frozen case, and the standing rule is fix the engine, never edit the case. The count here matches the machine-readable scorecard attached to the current tagged release (expanded 43 → 121 on 2026-07-05) and is updated with each release.
What one case pins
A golden case is a fixture (a small, readable JSON world of connectors, reports, metrics, and lineage), a natural-language question, and an expected block compared field-by-field against the real investigation result:
The 👎 rule: when a user thumbs-down an answer, the workspace state that produced it becomes a fixture and the correct behavior becomes a frozen case. The engine is fixed until the case passes — the case is never edited to match a wrong answer. Each regression caught this way can never return.
What the set covers
Freshness
“Is it stale, did it load, why is it late” — healthy paths, failed probes, and lineage-walk root cause.
Warehouse probes
Live catalog and freshness probes across the supported warehouses — healthy and deliberately broken.
BI refresh evidence
Dashboard refresh checks — confirmed, failed-and-surfaced, and honestly unavailable.
Orchestration
Pipeline-run evidence feeding root-cause answers, across the orchestration connector surface.
Trust & discrepancy
“Can I trust this number right now” — including no false alarms on a healthy graph — and “the dashboard doesn't match finance” reconciliation.
Ownership & catalog
Who owns a report, who's on the hook, and what's actually in the warehouse.
Ambiguous → must ask
Two plausible reports means the engine asks a clarifying question. Guessing is a failed case.
Out of scope → must refuse
Hiring plans, poems, legal advice, stock tips: the only correct behavior is a refusal.
Honest limits
Systems that aren't connected get an honest “not connected”, and write requests are declined — the product is read-only by design.
The harness, the golden set, and the scorecard are all in the product repository — they ship with the product. A pilot team doesn't take the number on our word; they run it in their own environment.
The harness imports investigate() — the same function the chat API calls in production — and drives it case by case. Nothing is mocked at the answer layer; only the outside world is pinned.
Each case gets a fresh in-memory workspace seeded from its fixture, and warehouse / BI probes are supplied as functions returning the fixture’s rows or throwing its typed errors. No socket is ever opened.
The harness runs with no API key, so the engine takes its deterministic composer path. The same commit scores the same on any machine and in CI — a regression is always a code change, never model noise.
The eval runs on every push and every PR to main. A dropped case — or a single ungrounded token — blocks the merge. The score cannot silently rot, and the standing rule is that it never regresses.
npm run eval# prints a per-category breakdown and writes eval/results/scorecard.json# exits non-zero on ANY grounding violation — before the pass rate is even consideredThe repository is private today, so this command runs inside a pilot engagement — access is part of the same per-pilot grant as the container image. Each tagged release also attaches the machine-readable scorecard, so you can diff the number we claim against the artifact CI produced.
Honest boundaries, stated on purpose: the eval scores the deterministic composer path — that’s what makes the number reproducible and CI-safe. A scheduled eval of the LLM-composed prose is future work; the grounding enforcement it would measure already ships and already runs on that path in production.