{"course":{"slug":"ai-careers","title":"Working in AI","url":"https://addaly.com/learn/ai-careers","level":"beginner","track":"work"},"lesson":{"slug":"trust-and-safety-in-depth","title":"Trust, safety and the policy side of AI","url":"https://addaly.com/learn/ai-careers/trust-and-safety-in-depth","position":18,"of":85,"minutes":9,"answer":"Trust and safety is a chain from policy to taxonomy to detection to appeals, and every threshold is an explicit choice about which of two harms you would rather cause.","keyIdea":"Trust and safety is a chain from policy to taxonomy to detection to appeals, and every threshold is an explicit choice about which of two harms you would rather cause."},"body":{"format":"markdown","text":"## The job that decides what a system is allowed to do\n\nEvery deployed model is governed by rules: what it must refuse, what gets removed, what gets shown to fewer people, what gets escalated to a human. Somebody writes those rules, somebody turns them into a taxonomy a classifier can be trained on, somebody measures whether enforcement matches the policy, and somebody handles the appeals from people who were wrongly caught.\n\nThat cluster of work is trust and safety. It employs far more people than research does, it is growing because regulation is growing, and it hires unusually widely: lawyers, linguists, teachers, journalists, former moderators, and engineers.\n\n## The chain, in order\n\n1. **Policy.** A sentence in ordinary language: \"we remove content that harasses a named private individual.\"\n2. **Taxonomy.** That sentence broken into categories with definitions and boundary cases, so two people reading it label the same post identically.\n3. **Detection.** Rules, classifiers, models: whatever finds candidates.\n4. **Enforcement.** What happens: removal, throttling, a warning, an account action, a referral.\n5. **Appeals.** The route back for the people you got wrong.\n6. **Measurement.** Precision and recall per category, and how both differ across languages and groups.\n\nAlmost every failure in this field is a break between two adjacent links. A good policy with no taxonomy produces inconsistent enforcement. Good detection with no appeals route produces a slow accumulation of injustice that surfaces as a news story.\n\n> Figure: The chain, in order. Steps: 1. Policy (one plain sentence); 2. Taxonomy (categories two people label alike); 3. Detection (rules, classifiers, models); 4. Enforcement (remove, throttle, warn, refer); 5. Appeals (the route back); 6. Measurement (precision and recall per category). Almost every failure in this field is a break between two adjacent links: a policy with no taxonomy enforces inconsistently, and detection with no appeals route accumulates injustice until it becomes a news story.\n\n## The trade that never goes away\n\nEvery threshold sets a ratio of two harms. Raise recall and you catch more genuine abuse and also remove more innocent posts. Raise precision and you leave more abuse standing.\n\nThere is no setting that avoids both, and the mathematics will not choose for you. What makes somebody good at this job is being explicit: which of the two errors is worse *for this category*, and saying so in writing.\n\nThe answer differs sharply by category, and the reasoning is the interesting part. For child sexual abuse material and for direct threats, a false negative is catastrophic and irreversible, so you accept a heavy review load. For political opinion, a false positive silences lawful speech, and the standard response is not removal but reduced distribution: a decision that is contested and that you should be able to argue on both sides. For self-harm, removal is often the wrong action entirely: the post is a person in distress, and the correct response is support resources and keeping them connected, not deletion.\n\n## The stack\n\n- **Writing.** The primary tool. Policies, definitions, incident reports, transparency reports.\n- **SQL and basic analysis**, because every argument here is settled with rates: appeals upheld per thousand actions, prevalence by category, enforcement rate by language.\n- **The evaluation vocabulary** from the evidence module: confusion matrix, precision, recall, inter-annotator agreement, sampling for prevalence.\n- **Regulatory literacy**: the EU's Digital Services Act, the EU AI Act, India's IT Rules, the UK Online Safety Act, and the fact that the rules differ per country and are still moving. You do not need to be a lawyer. You need to know which questions belong to one.\n\n## Two routes in\n\n**From moderation or support.** Very common, and the domain knowledge is real. Move from applying policy to writing it by documenting edge cases and enforcement inconsistencies that nobody else has written down.\n\n**From data or engineering.** Bring measurement. Teams frequently cannot answer \"what is our precision on harassment in Hindi\", and the person who can build that answer becomes central quickly.\n\nThe portfolio piece: take a public platform's published policy, build a taxonomy from it, label 200 real public posts against it, report per-category precision and recall for a simple keyword baseline, and write up the boundary cases that broke your definitions. That is the job, done small.\n\n## The failure that ends the first year, and the honest limitation\n\nThe failure is treating enforcement as a technical problem. A classifier at 0.9 precision applied to twenty million posts a day is wrong about two million of them, and each one is a person. The work is designing the whole system: thresholds, human review, appeals, and what happens at 3am on a public holiday.\n\nThe limitation is exposure. Depending on the role you may see material that is distressing. Reputable teams limit exposure time, blur by default, rotate people out and provide real clinical support. Ask about all four in the interview. An employer that treats the question as unusual has answered it."},"check":{"question":"A team raises the confidence threshold on its harassment classifier so that fewer innocent posts are removed. What has necessarily happened, and what should be reported alongside the change?","options":[{"text":"Overall accuracy has improved, and reporting the new accuracy figure is sufficient.","correct":false,"tempting":"Uses a single aggregate on a rare-event problem, where accuracy is dominated by the vast majority of posts that were never going to be flagged."},{"text":"More genuine harassment now stays up, so the change should be reported as a pair (the fall in wrongful removals and the rise in missed abuse) with a stated view on which error is worse for this category.","correct":true},{"text":"The classifier has become more precise, so it should be retrained less often.","correct":false,"tempting":"Confuses a threshold change with a model improvement; the model is identical and its decay rate is unchanged."},{"text":"Appeals will rise, because more users will contest the remaining removals.","correct":false,"tempting":"Predicts the opposite of the likely effect, since fewer removals generally means fewer appeals."}]},"prev":{"slug":"applied-scientist-in-depth","title":"The applied scientist and research engineer","url":"https://addaly.com/learn/ai-careers/applied-scientist-in-depth"},"next":{"slug":"ai-product-manager-in-depth","title":"The AI product manager","url":"https://addaly.com/learn/ai-careers/ai-product-manager-in-depth"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Trust, safety and the policy side of AI', in Working in AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-careers/trust-and-safety-in-depth (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Trust, safety and the policy side of AI","courseTitle":"Working in AI","module":{"index":2,"slug":"the-roles-in-depth","title":"The roles, one at a time"},"answer":null,"keyIdea":"Trust and safety is a chain from policy to taxonomy to detection to appeals, and every threshold is an explicit choice about which of two harms you would rather cause.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Policy","detail":"one plain sentence"},{"label":"Taxonomy","detail":"categories two people label alike"},{"label":"Detection","detail":"rules, classifiers, models"},{"label":"Enforcement","detail":"remove, throttle, warn, refer"},{"label":"Appeals","detail":"the route back"},{"label":"Measurement","detail":"precision and recall per category"}],"title":"The chain, in order","caption":"Almost every failure in this field is a break between two adjacent links: a policy with no taxonomy enforces inconsistently, and detection with no appeals route accumulates injustice until it becomes a news story."},"alt":"Figure: The chain, in order. Steps: 1. Policy (one plain sentence); 2. Taxonomy (categories two people label alike); 3. Detection (rules, classifiers, models); 4. Enforcement (remove, throttle, warn, refer); 5. Appeals (the route back); 6. Measurement (precision and recall per category). Almost every failure in this field is a break between two adjacent links: a policy with no taxonomy enforces inconsistently, and detection with no appeals route accumulates injustice until it becomes a news story."}},"blocks":[{"t":"h2","id":"the-job-that-decides-what-a-system-is-allowed-to-do","runs":[{"text":"The job that decides what a system is allowed to do"}]},{"t":"p","runs":[{"text":"Every deployed model is governed by rules: what it must refuse, what gets removed, what gets shown to fewer people, what gets escalated to a human. Somebody writes those rules, somebody turns them into a taxonomy a classifier can be trained on, somebody measures whether enforcement matches the policy, and somebody handles the appeals from people who were wrongly caught."}]},{"t":"p","runs":[{"text":"That cluster of work is trust and safety. It employs far more people than research does, it is growing because regulation is growing, and it hires unusually widely: lawyers, linguists, teachers, journalists, former moderators, and engineers."}]},{"t":"h2","id":"the-chain-in-order","runs":[{"text":"The chain, in order"}]},{"t":"ol","start":1,"items":[[{"text":"Policy.","b":true},{"text":" A sentence in ordinary language: “we remove content that harasses a named private individual.”"}],[{"text":"Taxonomy.","b":true},{"text":" That sentence broken into categories with definitions and boundary cases, so two people reading it label the same post identically."}],[{"text":"Detection.","b":true},{"text":" Rules, classifiers, models: whatever finds candidates."}],[{"text":"Enforcement.","b":true},{"text":" What happens: removal, throttling, a warning, an account action, a referral."}],[{"text":"Appeals.","b":true},{"text":" The route back for the people you got wrong."}],[{"text":"Measurement.","b":true},{"text":" Precision and recall per category, and how both differ across languages and groups."}]]},{"t":"p","runs":[{"text":"Almost every failure in this field is a break between two adjacent links. A good policy with no taxonomy produces inconsistent enforcement. Good detection with no appeals route produces a slow accumulation of injustice that surfaces as a news story."}]},{"t":"h2","id":"the-trade-that-never-goes-away","runs":[{"text":"The trade that never goes away"}]},{"t":"p","runs":[{"text":"Every threshold sets a ratio of two harms. Raise recall and you catch more genuine abuse and also remove more innocent posts. Raise precision and you leave more abuse standing."}]},{"t":"p","runs":[{"text":"There is no setting that avoids both, and the mathematics will not choose for you. What makes somebody good at this job is being explicit: which of the two errors is worse "},{"text":"for this category","i":true},{"text":", and saying so in writing."}]},{"t":"p","runs":[{"text":"The answer differs sharply by category, and the reasoning is the interesting part. For child sexual abuse material and for direct threats, a false negative is catastrophic and irreversible, so you accept a heavy review load. For political opinion, a false positive silences lawful speech, and the standard response is not removal but reduced distribution: a decision that is contested and that you should be able to argue on both sides. For self-harm, removal is often the wrong action entirely: the post is a person in distress, and the correct response is support resources and keeping them connected, not deletion."}]},{"t":"h2","id":"the-stack","runs":[{"text":"The stack"}]},{"t":"ul","items":[[{"text":"Writing.","b":true},{"text":" The primary tool. Policies, definitions, incident reports, transparency reports."}],[{"text":"SQL and basic analysis","b":true},{"text":", because every argument here is settled with rates: appeals upheld per thousand actions, prevalence by category, enforcement rate by language."}],[{"text":"The evaluation vocabulary","b":true},{"text":" from the evidence module: confusion matrix, precision, recall, inter-annotator agreement, sampling for prevalence."}],[{"text":"Regulatory literacy","b":true},{"text":": the EU’s Digital Services Act, the EU AI Act, India’s IT Rules, the UK Online Safety Act, and the fact that the rules differ per country and are still moving. You do not need to be a lawyer. You need to know which questions belong to one."}]]},{"t":"h2","id":"two-routes-in","runs":[{"text":"Two routes in"}]},{"t":"p","runs":[{"text":"From moderation or support.","b":true},{"text":" Very common, and the domain knowledge is real. Move from applying policy to writing it by documenting edge cases and enforcement inconsistencies that nobody else has written down."}]},{"t":"p","runs":[{"text":"From data or engineering.","b":true},{"text":" Bring measurement. Teams frequently cannot answer “what is our precision on harassment in Hindi”, and the person who can build that answer becomes central quickly."}]},{"t":"p","runs":[{"text":"The portfolio piece: take a public platform’s published policy, build a taxonomy from it, label 200 real public posts against it, report per-category precision and recall for a simple keyword baseline, and write up the boundary cases that broke your definitions. That is the job, done small."}]},{"t":"h2","id":"the-failure-that-ends-the-first-year-and-the-honest-limitati","runs":[{"text":"The failure that ends the first year, and the honest limitation"}]},{"t":"p","runs":[{"text":"The failure is treating enforcement as a technical problem. A classifier at 0.9 precision applied to twenty million posts a day is wrong about two million of them, and each one is a person. The work is designing the whole system: thresholds, human review, appeals, and what happens at 3am on a public holiday."}]},{"t":"p","runs":[{"text":"The limitation is exposure. Depending on the role you may see material that is distressing. Reputable teams limit exposure time, blur by default, rotate people out and provide real clinical support. Ask about all four in the interview. An employer that treats the question as unusual has answered it."}]}],"check":{"question":[{"text":"A team raises the confidence threshold on its harassment classifier so that fewer innocent posts are removed. What has necessarily happened, and what should be reported alongside the change?"}],"options":[{"text":[{"text":"Overall accuracy has improved, and reporting the new accuracy figure is sufficient."}],"correct":false,"tempting":[{"text":"Uses a single aggregate on a rare-event problem, where accuracy is dominated by the vast majority of posts that were never going to be flagged."}]},{"text":[{"text":"More genuine harassment now stays up, so the change should be reported as a pair (the fall in wrongful removals and the rise in missed abuse) with a stated view on which error is worse for this category."}],"correct":true,"tempting":null},{"text":[{"text":"The classifier has become more precise, so it should be retrained less often."}],"correct":false,"tempting":[{"text":"Confuses a threshold change with a model improvement; the model is identical and its decay rate is unchanged."}]},{"text":[{"text":"Appeals will rise, because more users will contest the remaining removals."}],"correct":false,"tempting":[{"text":"Predicts the opposite of the likely effect, since fewer removals generally means fewer appeals."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"applied-scientist-in-depth","title":"The applied scientist and research engineer","minutes":9},"next":{"slug":"ai-product-manager-in-depth","title":"The AI product manager","minutes":9}}}