SEO meta tags: LLM-generated vs. rule-based — 8 weeks, 4,800 articles
Published: · seo
Eight weeks, 4,800 articles, three meta fields — the title meaningfully wins with an LLM, the description does not, anchor text neutral.
For two years we have run a rule-based meta generator: title patterns, description patterns, a curated anchor-text library. This spring we asked whether a model-based replacement would meaningfully beat it. Eight weeks, 4,800 articles, three meta fields. Here are the numbers.
The A/B setup
Every new article was randomly assigned to arm A or B. Arm A: the existing rule-based generator — templates, keyword dictionary, category-specific regexes. Arm B: Claude Sonnet 4.6 with the last three paragraphs and primary entities in context, plus a 'tone: factual, max 60 characters for title' instruction. Both arms reported through Google Search Console: impressions, CTR, position.
Eight weeks, 2,400 articles per arm. Every article had at least three weeks of CTR data at analysis time.
Results by field
Title CTR: arm B beat arm A by 6.2% in mean CTR, p < 0.01 by Welch's t-test. The winning pattern: the model consistently wrote titles that did not duplicate the H1. They functioned as a true 'second title' rather than a restatement. This aligns with what the Search Console team has been saying since 2025.
Description CTR: -1.1%, not statistically significant (p = 0.18). The model wrote linguistically better descriptions, but the CTR did not move. Our hypothesis: in 2026 the description is no longer the dominant click signal — 70%+ of users on the SERP are making click decisions from off-snippet cues (favicon, breadcrumb, sitelinks, rating microdata).
Internal-link anchor text: neutral, no significant difference. The model and the rule-based generator produced anchors of equivalent quality. The rule-based version is cheaper, so we kept it.
What we kept rule-based
The description generator stays rule-based. It is simpler, cheaper, and equally effective. The anchor-text library also stays — every in-category link feeding 'related topics' and 'continue reading' modules is rule-generated, because the model produces equivalent output at higher cost.
The hybrid that won
The final architecture is hybrid. Title: model-generated, three suggestions, editor picks or rewrites. Description: rule-based, category templates, with a compression of the article's first sentence. Anchor text: rule-based, two libraries (in-topic, out-of-topic). The per-article model-call cost dropped by 60% compared to a fully model-generated alternative, with no measurable loss of any signal that moved.
What we did not measure, and it bothers us
We did not measure long-horizon title quality — whether a title still cleanly describes the article when re-read six months later. CTR is a short-horizon signal. A model-generated title with a +6.2% same-day click bump might be ambiguous in the archive. A retro-eval would settle this. We have not built one yet.
We also did not measure archive compatibility — whether the model-style new titles sit visually alongside the rule-based 2018 titles on a category index without aesthetic dissonance. There is more to explore here.
The takeaway
Not every meta field is equal. The title meaningfully benefits from a model. The description does not. The anchor text does not. The default 'replace everything with an LLM' impulse would have been expensive enough to be worth resisting, and resisting it was the right call.