AI content moderation: the 87% you can automate, and the 13% that stays a brand decision

On a publisher platform 18,000-22,000 user-generated items move through every day — comments, forum posts, image uploads, live-event chats. Two years ago a 14-person human moderation team ran three shifts. Today it is a multimodal AI layer plus a 3-person audit team — the model handles 87%, humans handle 13%. Here is the logic of the split.

The multimodal model as we use it

Not one model, four orchestrated. Text classifier: 6 categories (hate, harassment, spam, sexual, violence, self-harm) plus tone. Image classifier: NSFW + graphic violence + face detection (only for 'familiar face' signal, not identification) + brand-logo detection. Audio classifier (for live stream chats): keyword spotting plus emotional tone estimation. Cross-modal reconciliation: if the text is harmless but the image is graphic, the combination may still be problematic (memes and context collapse).

The four run in sequence. Full inference for a text comment: 80ms (p99 130ms). For an image comment: 340ms (p99 510ms). This is not real-time UX — the user publishes, the moderation decision lands 0.4-1.2 seconds later as 'publish / quarantine / block'.

The 87% — what the model handles alone

Of the daily 22,000 items, an average 19,140 (87%) get no human audit. The breakdown:

• Hard accept (model confidence > 0.92 on every harm dimension): 76%, publishes immediately
• Soft accept (0.7-0.92 confidence): 9%, publishes but goes on background monitor (post-publish flagging if other users report)
• Hard reject (clear hate, spam, sexual content): 2%, automatic rejection plus user-level strike

The accuracy on the 2% hard reject is the critical number. An 8-week impact study put the hard-reject false-positive rate at 0.4% — i.e. 88 of 22,000 daily items wrongly rejected. That is 8-9 support tickets per day. Acceptable noise compared to the 440 graphic or hate items that would otherwise sit live.

The 13% — what goes to humans

The remaining 13% (average 2,860 items per day) goes to human audit. This does not mean 'the model could not decide' — it often means the decision is a brand call, not a technical one.

Three categories that go to humans:

1. Confidence ambiguity (0.4-0.7 on any dimension): average 1,540 items daily. Classic borderline. Two human auditors label independently; if they agree, decision; if not, supervisor.

2. Policy sensitivity: average 780 items daily. Political posts, religious content, ethnic-tension signals. The model could technically handle this, but the brand impact is large enough that we want a human in the loop. For a publisher's brand position, 'what we let through' matters as much as 'what we block'.

3. New phenomena and training feedback: average 540 items daily. The model actively samples content from areas with thin training data — new slang, emerging political narratives, new spam schemes. Their human labels feed back into the model weekly (continuous fine-tuning).

How the human team is structured

Of the original 14, three remain. A senior policy lead (for brand-sensitive decisions), an ML feedback specialist (for the training loop), a junior auditor (for the daily drift of confidence-ambiguous items). They are not 'doing less of the same job' — their role is different.

The policy lead adjusts thresholds weekly based on model behaviour. The ML feedback specialist reverse-engineers failures — what pattern the model does not recognise, what edge case it fails on. The junior auditor handles daily drift and escalates politically sensitive items to the lead.

The 13% that will not automate

The expectation in 2022 was that within 5 years we would absorb the 13%. After two years the numbers say: no. Within the 13%, the policy sensitivity segment (28%) is not human-handled because the model cannot classify it — it is human-handled because a publisher brand's identity includes who decides what, in which context. The model does not know, and should not. AI does triage; brand policy does the decision.

The lesson: moderation automation is not a 0-to-100% question. The 87% / 13% split looks stable, and the right move is to reassign the people who used to work on the 87% — not lay them off, redirect them to the deeper policy work on the 13%.

Takeaway

AI content moderation does not replace human audit — it rebalances it. The 87% is fast decision, reliable confidence thresholds, automated action. The 13% is slow decision, brand sensitivity, human judgement. Both are necessary. Without the first, user experience suffers (unscalable). Without the second, the brand suffers (homogenised).