The eval said 0.96. The product was dropping every number in the story.

what I found

what gets measured

Coverage — of the facts I hand-labelled as essential to a story, how many did the digest actually convey?

Two design decisions I’d defend:

Coverage is a checklist, not a rating. I don’t ask the judge to score quality 1–5; that number drifts between runs and can’t be checked against a human. It gets 3–6 pre-labelled atomic facts per article and returns present or absent for each, with a quote from the digest as evidence. If it can’t quote it, it’s absent. Binary, auditable, and directly comparable to my own judgement.

Partial counts as absent. If the label reads “~8,000 victims” and the digest says “thousands of victims”, that’s a miss — the number was the news. Without this rule the judge waves through exactly the failure mode I care about.

the golden set

Six articles, 24 atomic must-include facts, labelled by hand. Small — see limitations. Three rules I only arrived at by getting them wrong first:

• One fact per item. My first label read Zyaire Wilkins, 21, Florida student, arrested by FBI — four facts in one checkbox. The judge marked it present on the strength of the name and age; “student” appears nowhere in the digest. A compound item can’t be half-ticked, so the judge defaults to lenient.
• Label from the full source, not the snippet. One label was written against a truncated excerpt, which made a correctly-reported figure look invented. A golden set is only as good as the text it was written from.
• Don’t label what isn’t there. Twice I caught myself writing a plausible fact the source never stated — the same failure I’m asking the judge to catch in the model.

validating the judge

An unvalidated judge is a random number generator with good manners. Before trusting a single number it produced, I graded all 24 items myself — blind, with the judge’s column hidden in the sheet until mine was finished — and compared.

This is weaker than it looks, and I’d rather say so than oversell it. 22 of 24 items are “present”, so a judge that answered “yes” to everything would still score 91.7% agreement. The test barely pressed on it. What it does establish is that we agree on both negatives — the two real misses — which is the direction that matters. To make this result mean something I need a golden set with more failure cases in it.

finding 1 — measuring a system that doesn’t exist

Baseline came back at 0.96. That felt too high, so I checked it against the digest that had actually been emailed that morning, on the same article.

The cause was one line of preprocessing in the production workflow:

.slice(0, 600)

Production truncated each article to 600 characters of RSS snippet. The figures live further down the article — the model never received them. Meanwhile the golden set held the full text I’d pasted in by hand.

So the model wasn’t dropping the numbers. It never had them.

The fix

Production now fetches each article URL, strips the page to text, and passes 4,000 characters instead of 600. The next morning’s digest — a different story, this one on US EV sales — carried “5.8% market share, 247,226 units, −20.5%, $7,500 tax credit”: figures that sit well beyond the 600-character cutoff the old pipeline used.

What I take from it: an offline eval is only as valid as its resemblance to the production path. Mine diverged at one line, and the divergence flattered the score by roughly six times.

finding 2 — the same input doesn’t give the same output

Re-running the eval unchanged, one digest named five games; the next named three. Coverage on that article moved 1.00 → 0.83. Nothing had changed — not the prompt, not the data, not the model.

This is ordinary sampling behaviour, but it has a blunt consequence for anyone reading a single eval score as truth: one run is one sample, not a measurement. Which leads directly to the next thing worth knowing.

finding 3 — the noise floor

Four identical runs. No changes of any kind between them.

This is the cheapest useful number in the whole project — four clicks — and it’s the one that prevents weeks spent tuning prompts against movement that was always going to happen anyway. It’s also the answer to the first question anyone should ask about an A/B result: how do you know that isn’t noise?

how it’s wired

The eval is a second workflow, deliberately separate from production. It reads the golden set row by row, so every run scores the same frozen sample — otherwise a prompt change and a news cycle would move at the same time and neither could be blamed.

When fetching a dataset row   → Google Sheets, one row per run
  → HTTP Request                write the digest   (prompt under test)
  → HTTP Request                judge the digest   (temp 0, JSON out)
  → Code                        parse, score, fail loud on bad JSON
  → Set Outputs                 write results back to the sheet
  → Set Metrics                 track coverage across runs

One detail worth naming: if the judge returns unparseable JSON, the workflow throws rather than recording a zero. A zero looks like a bad digest; it’s actually a broken judge. Silently conflating the two would quietly poison every number downstream.

limitations

Stated plainly, because an eval that hides these isn’t an eval.

• n = 6 articles, 24 items. Enough to build and validate the mechanism; not enough for confident deltas.
• Class imbalance 22:2 makes κ = 1.00 far less impressive than the number suggests.
• Single labeller. No second annotator, so no measure of how much my own labels drift.
• Coverage only. It checks that required facts are present and is structurally blind to invented ones — a digest carrying every labelled fact plus a fabricated sentence would still score 1.00.
• Four runs is a thin basis for a noise estimate. The ±0.04 is indicative, not precise.
• No A/B yet — and at this sample size, against this noise floor, a small prompt delta would be unresolvable anyway.

what six articles bought

The sample is small, and I’d rather be the one to say so. But the three things this surfaced don’t depend on sample size, and that’s the argument for building the smallest version first.

A truncation bug, a non-deterministic model and a noise floor look the same at n=6 as at n=600. The 600-character cutoff was silently costing the product every verifiable number in a story, and it had been doing that in my inbox every weekday morning without me noticing. Six labelled articles found it in an evening. Six hundred would have found the same thing, later and at more cost.

The mechanism now exists and has been checked. Dataset, judge, scoring, write-back, tracked metrics — and, importantly, a judge whose verdicts I’ve compared against my own rather than assumed. Growing this is labelling work now, not engineering work.

The useful output was never 0.935. It was finding out that my first number described a system that doesn’t exist, and that my second one moves by ±0.04 on its own. At this stage an eval’s job isn’t to certify that the product is good — it’s to tell you whether you can trust your own numbers yet. Mine couldn’t, twice. Now it can, within a range I can state.

The alternative was continuing to ship something that looked fine.

what I’d do next

• Faithfulness — check each claim in the digest against its source. It needs no golden labels, because the source article is the reference, which means unlike coverage it can run on live production traffic.
• A guardrail on faithfulness, thresholded off the observed baseline distribution rather than a hand-picked number: block the digest and alert instead of shipping a fabricated claim.
• Grow the golden set toward failure cases — paywalled sources, multi-story roundups, corrections, opinion pieces with no hard facts.
• Cost and latency as first-class metrics. An eval that ignores them will happily recommend a prompt that triples the bill.