Moderation before and after the model

Two directions

Content enters the model from a user and leaves it as an answer. Either can be harmful, and they fail differently. An input can be abusive, an attempt to extract something the product must not produce, or a person in distress. An output can be wrong in a way that hurts: dangerous instructions, defamation, sexual content, a slur the model reproduced from a document. Moderation is a check on both, and it is a separate concern from the security controls of the last three lessons: those protect systems, this protects people.

The free classifiers

You do not need to build one. Llama Guard, released openly by Meta, classifies a message against a set of harm categories and runs on modest hardware; OpenAI’s moderation endpoint is free to call and returns category scores; Detoxify is a small open model for toxicity. Each returns a category and a score in tens of milliseconds. Each has a published set of categories that will not match yours exactly, and each has error rates you must measure rather than assume.

Layering, so the expensive model sees only the grey

The pattern this site uses for its own posts is the general one. Rules first: a small, precise list of patterns for the clear cases: the things that are unambiguous in any context. A cheap classifier second, on everything: fast, tolerable error rate, decides the obvious. A large model third, only for inputs the classifier scored in the uncertain middle, with a prompt that asks for a structured verdict and a reason. A person fourth, for appeals and for the cases the model marked uncertain.

The economics are the reason for the order. Running a frontier model on every message doubles the cost of the feature. Running it on the five per cent the cheap classifier could not decide adds a few per cent. The rules layer costs nothing and, kept small and precise, almost never wrongs anyone.

False positives are the expensive failure

A moderation layer that blocks a legitimate message is not a safe default. It is a broken product for that person, and the people it breaks it for are disproportionately those discussing hard things: a nurse asking about medication doses, a survivor describing what happened, a teenager asking about a body. Measure the false-positive rate on your own traffic (sample a few hundred blocked messages and read them) and set thresholds knowing what each point of recall costs in wrongly refused people. The Scunthorpe problem, a town name blocked by a substring filter, is the canonical example, and every rule you write is a chance to recreate it.

Distress is not a violation

A message that expresses self-harm is a person, not a policy breach. Blocking it, or replying with a refusal, is the worst available response. Route it to support: the message stays, the reply includes helplines for the user’s region, the tone is warm, and nothing is logged as an offence. The only self-harm content that is removed is encouragement or instruction directed at others. This site’s moderation code treats that as an invariant, and it is worth adopting as one, because a classifier that lumps “I want to hurt myself” with “here is how to hurt someone” will make a product that turns away exactly the people who most needed a kind answer.

Output moderation and streaming

Checking the answer before showing it conflicts with streaming, which shows tokens as they arrive. Three approaches, in order of latency cost. Check the input only, and accept that the output is unchecked; adequate for many products where the model is well-behaved and the input check catches the intent. Buffer the first sentence or two, check, then stream; a few hundred milliseconds of delay. Or stream, check the complete answer afterwards, and retract (replace the message with a notice) if it fails; the user may have seen the content for a second. The right choice depends on the audience, and a product for children makes a different one from a developer tool.

Politics, and distribution versus removal

Some categories are not harms but disagreements. A lawful political opinion is not removed by any reasonable system; the choice is where and how widely it is shown. Political abuse (incitement, targeted hate) is a harm and is removed. The distinction is a policy decision your product makes on purpose and writes down, not one a classifier makes by default, and it is a place where different products reasonably differ.

What to log and what to tell the user

Log every moderation decision with the layer that made it, the category, the score and the action, so false positives can be found and appealed. Tell the user when a message was not sent and why, in a sentence, with a way to appeal to a person. A silent drop is the one option that is never right.

Cost and latency

Input classification: ten to fifty milliseconds and a fraction of a cent, or free self-hosted. The large-model grey-zone call: one second and a normal call’s cost, on a few per cent of traffic. Human review: minutes per item, so the layers above must keep the queue small. Write those numbers into the design; a moderation layer that is designed without them ends up bypassed for speed.