Trust, safety and the policy side of AI

The job that decides what a system is allowed to do

Every deployed model is governed by rules: what it must refuse, what gets removed, what gets shown to fewer people, what gets escalated to a human. Somebody writes those rules, somebody turns them into a taxonomy a classifier can be trained on, somebody measures whether enforcement matches the policy, and somebody handles the appeals from people who were wrongly caught.

That cluster of work is trust and safety. It employs far more people than research does, it is growing because regulation is growing, and it hires unusually widely: lawyers, linguists, teachers, journalists, former moderators, and engineers.

The chain, in order

1. Policy. A sentence in ordinary language: “we remove content that harasses a named private individual.”
2. Taxonomy. That sentence broken into categories with definitions and boundary cases, so two people reading it label the same post identically.
3. Detection. Rules, classifiers, models: whatever finds candidates.
4. Enforcement. What happens: removal, throttling, a warning, an account action, a referral.
5. Appeals. The route back for the people you got wrong.
6. Measurement. Precision and recall per category, and how both differ across languages and groups.

Almost every failure in this field is a break between two adjacent links. A good policy with no taxonomy produces inconsistent enforcement. Good detection with no appeals route produces a slow accumulation of injustice that surfaces as a news story.

The trade that never goes away

Every threshold sets a ratio of two harms. Raise recall and you catch more genuine abuse and also remove more innocent posts. Raise precision and you leave more abuse standing.

There is no setting that avoids both, and the mathematics will not choose for you. What makes somebody good at this job is being explicit: which of the two errors is worse for this category, and saying so in writing.

The answer differs sharply by category, and the reasoning is the interesting part. For child sexual abuse material and for direct threats, a false negative is catastrophic and irreversible, so you accept a heavy review load. For political opinion, a false positive silences lawful speech, and the standard response is not removal but reduced distribution: a decision that is contested and that you should be able to argue on both sides. For self-harm, removal is often the wrong action entirely: the post is a person in distress, and the correct response is support resources and keeping them connected, not deletion.

The stack

• Writing. The primary tool. Policies, definitions, incident reports, transparency reports.
• SQL and basic analysis, because every argument here is settled with rates: appeals upheld per thousand actions, prevalence by category, enforcement rate by language.
• The evaluation vocabulary from the evidence module: confusion matrix, precision, recall, inter-annotator agreement, sampling for prevalence.
• Regulatory literacy: the EU’s Digital Services Act, the EU AI Act, India’s IT Rules, the UK Online Safety Act, and the fact that the rules differ per country and are still moving. You do not need to be a lawyer. You need to know which questions belong to one.

Two routes in

From moderation or support. Very common, and the domain knowledge is real. Move from applying policy to writing it by documenting edge cases and enforcement inconsistencies that nobody else has written down.

From data or engineering. Bring measurement. Teams frequently cannot answer “what is our precision on harassment in Hindi”, and the person who can build that answer becomes central quickly.

The portfolio piece: take a public platform’s published policy, build a taxonomy from it, label 200 real public posts against it, report per-category precision and recall for a simple keyword baseline, and write up the boundary cases that broke your definitions. That is the job, done small.

The failure that ends the first year, and the honest limitation

The failure is treating enforcement as a technical problem. A classifier at 0.9 precision applied to twenty million posts a day is wrong about two million of them, and each one is a person. The work is designing the whole system: thresholds, human review, appeals, and what happens at 3am on a public holiday.

The limitation is exposure. Depending on the role you may see material that is distressing. Reputable teams limit exposure time, blur by default, rotate people out and provide real clinical support. Ask about all four in the interview. An employer that treats the question as unusual has answered it.