The article quality eval-harness we built for ourselves

Last autumn we collected 312 of our own published articles that the newsroom unanimously called craft. Not the most-read, not the most viral — the ones we would put forward as the ceiling of our own standard. That corpus became the foundation of the eval-harness that now gates every model update.

Fourteen dimensions

Industry LLM-as-a-judge defaults — G-Eval, MT-Bench, generic helpfulness rubrics — are a usable starting point but lose teeth when applied to journalism. We defined fourteen dimensions of our own:

1. Lede strength — does the first sentence hold the reader
2. Claim density — information per word
3. Structural coherence — is the argument followable
4. Reference specificity — names, dates, sources
5. Quote quality — functional or decorative
6. Data-narrative balance — neither stat dump nor anecdote drift
7. Subject-matter credibility — does the author understand the field
8. Self-critical distance — is the counter-question asked
9. Title fulfillment — does the piece deliver what the title promises
10. Spatio-temporal accuracy — is the event placed in its context
11. Linguistic economy — can anything be cut
12. Conclusion weight — is the last paragraph worth reaching
13. SEO unobtrusiveness — does keyword optimization show through
14. Reader respect — no talk-down, no assumed stupidity

Each scored 1–7 on a Likert scale with detailed rubrics. Anthropic Claude Opus 4.7 is the judge, run two-shot with calibration examples drawn from our 312-piece corpus.

The regression gate

Every model bump — Claude, OpenAI, or any open-weights option we are evaluating — runs the full 312-article harness. The gate is simple: per-dimension mean must stay within ±2% of the previous run. Slip below or above and the deploy blocks. Why does going above also block? Because a +5% jump in structural coherence usually means the model has started imposing a template on every article, which looks like improvement in the raw score but flattens the corpus.

The gate has stopped three model upgrades so far. In all three cases the apparent quality lift turned out to be stylistic homogenization, exposed by a simultaneous drop on dimension 8 (self-critical distance).

Two dimensions where humans still win

Subject-matter credibility (7) and Reader respect (14) — on these two dimensions our human authors hold a stable 0.4–0.6 point lead over the best model generations. On the first the model invents specifics or over-abstracts; on the second it lands either too explanatory or too clubby. Neither is our house voice.

One dimension where we trust the model more than ourselves

Linguistic economy (11) — humans defer to humans here. We will read each other's 850-word drafts and accept the 850. The LLM-judge is undiplomatically honest: it flags when 720 would carry the same content. Three months ago we promoted this dimension into the CI itself — every article above 1,000 words gets an automatic economy report, and the editor must acknowledge it before publish.

What the harness will not solve

It does not tell us whether an article is important. The 312 reference pieces implicitly encode our definition of important, but we could not operationalize it. We tried citation graphs, social shares, returning-reader cohorts — none correlated convincingly with the editorial judgment of 'this matters'. Importance remains the editor-in-chief's call, and on current evidence it probably should.

The harness is a discipline tool, not a taste oracle. We are clearer about that at month nine than we were at month one.