AI editorial assistant, six months in: what it ships and what it does not

Six months ago we put our in-house AI editorial assistant into production. It is not a chatbot and it is not a comment moderator. It lives as a Tiptap extension on the editor side and as a CI-style background pipeline on the publishing side. Enough usage data has accumulated that we can now be honest about where it earns its keep and where it does not.

What it ships

Three functions hold up under daily use. Title alternatives: five suggestions per draft, each from a different angle — interrogative, declarative, numeric, quote-led, location-led. It does not pick the headline. It dissolves the standing meeting where four people argue about a title for fifteen minutes.

Lede polish: the opening paragraph gets rewritten in three registers — compressed, narrative, question-led. Editors take one of the suggestions 64% of the time, hand-edit using a suggestion as scaffolding 28% of the time, and discard all three 8% of the time. That 8% has been stable for six months. It is the floor where the model simply has nothing to add.

Structural cuts: when a draft exceeds 1,200 words the assistant flags two or three paragraphs whose removal would not damage the argument. It does not delete. It suggests in a Tiptap comment with its reasoning. The editor accepts or rejects.

What it does not ship

It does not arbitrate tone. Our house voice cannot be reproduced by an OpenAI or Anthropic model without sliding into a flatter, more neutral editorial register. We removed the tone-normalization step three months in. Editors told us, in writing, that it irritated more than it helped.

It does not perform fact-checking. We tried twice — once with retrieval augmentation, once with web-search grounding — and stopped. Fact-checking is not document lookup. It is the human act of doubting and picking up a phone. The model either invents confidently or asks about everything, both of which generate noise rather than safety.

It does not make the headline call. The five suggestions are throwaway scaffolding. The final headline is almost always a sixth, written by a human after seeing the five. That is the correct outcome. Headlining is a marketing decision, not a writing one.

The four weekly metrics

Every Monday a small Vercel-hosted dashboard updates four numbers: editor acceptance rate on the lede, BLEU overlap between suggested and published titles, average manuscript edit time pulled from Tiptap timestamps, and average expert-user minutes per draft. We deliberately do not compute a composite score. A composite generates debate; four numbers generate action. Each metric has one owner.

The one feature we pulled

We shipped a 'collaborative quote-tuning' feature that took quotes from interviews and smoothed them grammatically while attempting to preserve the speaker's voice. We pulled it after two months. In journalism a quote is not text. It is a speech act. Even the smallest word substitution is a judgment that only the reporter who conducted the interview can make. Automation here does not save time — it costs credibility.

What six months taught us

The assistant earns its place because it recognizes its own boundaries and does not cross them. We designed it scoped from day one: three jobs with measurable quality lift, nothing else. Every new feature request gets one question first: can we measure it in a weekly cycle, or does it just look good on the roadmap?

The next six months are a test of discipline. The pressure to stuff generative features into every part of the newsroom is constant, and the political cost of saying 'no, we measured it and it did not help' is not zero.