{"course":{"slug":"ai-and-you","title":"AI, Safety and What Goes Wrong","url":"https://addaly.com/learn/ai-and-you","level":"beginner","track":"start"},"lesson":{"slug":"incident-reporting","title":"Learning from what went wrong","url":"https://addaly.com/learn/ai-and-you/incident-reporting","position":70,"of":73,"minutes":8,"answer":"Aviation's safety record rests on independent investigation and confidential non-punitive reporting of near-misses, and the equivalent for AI barely exists, so define incidents to include near-misses, make reporting safe.","keyIdea":"Aviation's safety record rests on independent investigation and confidential non-punitive reporting of near-misses, and the equivalent for AI barely exists, so define incidents to include near-misses, make reporting safe, and turn every one into a permanent test case."},"body":{"format":"markdown","text":"## The comparison that keeps being made\n\nCommercial aviation is extraordinarily safe, and the usual explanation is technology. The better explanation is reporting. Aviation built, over decades, a system in which failures are reported, investigated by an independent body, published in full, and turned into changes that everyone must adopt.\n\nTwo features do the work.\n\n**Mandatory investigation with independence.** Accident investigators are separate from the regulator and from the operators, publish findings including uncomfortable ones, and their remit is causes rather than blame.\n\n**Confidential, non-punitive voluntary reporting.** The US Aviation Safety Reporting System, run by NASA rather than the aviation regulator specifically so that reporting does not go to the enforcer, receives many tens of thousands of reports a year from pilots and controllers describing their own mistakes. Reporting confers limited protection from enforcement action. That protection is why the reports exist, and the near-misses are where most of the learning is.\n\nAI has neither at present. What it has is journalism, litigation and researchers, which is a poor substitute because all three are adversarial, and adversarial channels select for the cases somebody can be blamed for rather than the cases everybody could learn from.\n\n## What does exist\n\n**The AI Incident Database**, run by the Responsible AI Collaborative, catalogues publicly reported incidents (over a thousand and growing) with structured fields, and is free to search. It is compiled from public sources, so it captures what was reported and not what happened.\n\n**The OECD's AI Incidents Monitor** tracks incidents from news sources internationally, and the OECD has been working on a common incident reporting framework so that different countries' regimes can share definitions.\n\n**The EU AI Act creates a duty.** Providers of high-risk systems must report serious incidents to authorities. This is the first significant mandatory regime and it applies to a defined subset, not to everything.\n\n**Sectoral regimes already bite.** Medical device regulators have adverse-event reporting that covers AI-based devices. Financial regulators require operational incident reporting. Data protection authorities require breach notification, often within 72 hours.\n\n## What counts as an incident\n\nBroader than most people assume, and defining it narrowly is how organisations end up reporting nothing.\n\nA harm that occurred. A near-miss caught before it reached anyone. A system behaving outside its stated scope. A material accuracy degradation. A discovered disparity between groups. An unauthorised use. A confidentiality breach through an AI tool. A supplier's model changing under you.\n\nThe near-misses are the valuable ones, because they are frequent and cheap. Any reporting culture that only captures realised harm is capturing the small tail of its available information.\n\n## Building this where you are\n\nYou do not need a national framework to do this internally.\n\n**Define an incident in writing**, including near-misses, and give examples so people recognise one.\n\n**Make reporting non-punitive and mean it.** If the first report leads to a disciplinary process, it will be the last report, and thereafter you will have a clean record and no information.\n\n**Record a fixed set of fields**: what happened, which system and version, who was affected, how it was detected, what was done, what changed as a result. The last field is the one that makes people keep filing.\n\n**Convert incidents into tests.** Every incident becomes a permanent case in your evaluation set. This is the single practice that stops the same failure recurring, and it is standard in software engineering and rare in AI deployment.\n\n**Review them together, on a schedule.** Individually they are anecdotes; twenty of them are a pattern, and the pattern is usually not what anyone predicted.\n\n**Watch the count.** Zero reported incidents means the reporting is broken, not that nothing happened.\n\n> Figure: A reporting loop you can build without a regulator. Steps: 1. Define an incident, including near-misses (With examples people recognise); 2. Make reporting non-punitive, and mean it (The first disciplinary process ends the reports); 3. Record fixed fields (What, which system and version, who was affected, how detected, what changed); 4. Convert each incident into a test case (A permanent item in the evaluation set); 5. Review them together, on a schedule (Twenty anecdotes are a pattern); 6. Watch the count (Zero is a broken channel). Aviation's record rests on reporting that does not go to the enforcer. The near-misses are where the learning is, and a count of zero means the reporting is broken, not that nothing happened.\n\n## Reporting outward\n\nWhen an incident affects people outside your organisation, the duty may be legal: data protection authorities, sectoral regulators, and under the EU regime, market surveillance authorities. Beyond duty, submitting to a public database is a small act with disproportionate value, because the field's collective knowledge of how these systems fail is currently assembled from whatever happened to become news."},"check":{"question":"Why is the US aviation voluntary reporting system deliberately run by NASA rather than by the aviation regulator?","options":[{"text":"Because NASA can analyse the data better than the aviation regulator could do","correct":false,"tempting":"Capacity is not the reason. The choice is about who receives a report, not who can process it."},{"text":"Because many incidents involve research aircraft that fall under NASA's remit","correct":false,"tempting":"Invents a jurisdictional reason. The system covers ordinary commercial flying, not research flights."},{"text":"To keep reports secret from the public, since publishing them would damage trust","correct":false,"tempting":"Inverts the model. Aggregated findings are published so the industry can learn from them."},{"text":"So reports go to a body that cannot punish, and people will admit their own mistakes","correct":true}],"why":"People report their own errors only to someone who cannot penalise them for it."},"prev":{"slug":"standards-and-audits","title":"What an audit can and cannot see","url":"https://addaly.com/learn/ai-and-you/standards-and-audits"},"next":{"slug":"the-two-debates","title":"The two arguments, and how to read them","url":"https://addaly.com/learn/ai-and-you/the-two-debates"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Learning from what went wrong', in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/incident-reporting (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Learning from what went wrong","courseTitle":"AI, Safety and What Goes Wrong","module":{"index":9,"slug":"living-with-it","title":"Living with it: cost, governance and your own practice"},"answer":null,"keyIdea":"Aviation’s safety record rests on independent investigation and confidential non-punitive reporting of near-misses, and the equivalent for AI barely exists, so define incidents to include near-misses, make reporting safe, and turn every one into a permanent test case.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"caption":"Aviation’s record rests on reporting that does not go to the enforcer. The near-misses are where the learning is, and a count of zero means the reporting is broken, not that nothing happened.","steps":[{"label":"Define an incident, including near-misses","detail":"With examples people recognise"},{"label":"Make reporting non-punitive, and mean it","detail":"The first disciplinary process ends the reports"},{"label":"Record fixed fields","detail":"What, which system and version, who was affected, how detected, what changed"},{"label":"Convert each incident into a test case","detail":"A permanent item in the evaluation set"},{"label":"Review them together, on a schedule","detail":"Twenty anecdotes are a pattern"},{"label":"Watch the count","detail":"Zero is a broken channel"}],"title":"A reporting loop you can build without a regulator","type":"flow"},"alt":"Figure: A reporting loop you can build without a regulator. Steps: 1. Define an incident, including near-misses (With examples people recognise); 2. Make reporting non-punitive, and mean it (The first disciplinary process ends the reports); 3. Record fixed fields (What, which system and version, who was affected, how detected, what changed); 4. Convert each incident into a test case (A permanent item in the evaluation set); 5. Review them together, on a schedule (Twenty anecdotes are a pattern); 6. Watch the count (Zero is a broken channel). Aviation’s record rests on reporting that does not go to the enforcer. The near-misses are where the learning is, and a count of zero means the reporting is broken, not that nothing happened."}},"blocks":[{"t":"h2","id":"the-comparison-that-keeps-being-made","runs":[{"text":"The comparison that keeps being made"}]},{"t":"p","runs":[{"text":"Commercial aviation is extraordinarily safe, and the usual explanation is technology. The better explanation is reporting. Aviation built, over decades, a system in which failures are reported, investigated by an independent body, published in full, and turned into changes that everyone must adopt."}]},{"t":"p","runs":[{"text":"Two features do the work."}]},{"t":"p","runs":[{"text":"Mandatory investigation with independence.","b":true},{"text":" Accident investigators are separate from the regulator and from the operators, publish findings including uncomfortable ones, and their remit is causes rather than blame."}]},{"t":"p","runs":[{"text":"Confidential, non-punitive voluntary reporting.","b":true},{"text":" The US Aviation Safety Reporting System, run by NASA rather than the aviation regulator specifically so that reporting does not go to the enforcer, receives many tens of thousands of reports a year from pilots and controllers describing their own mistakes. Reporting confers limited protection from enforcement action. That protection is why the reports exist, and the near-misses are where most of the learning is."}]},{"t":"p","runs":[{"text":"AI has neither at present. What it has is journalism, litigation and researchers, which is a poor substitute because all three are adversarial, and adversarial channels select for the cases somebody can be blamed for rather than the cases everybody could learn from."}]},{"t":"h2","id":"what-does-exist","runs":[{"text":"What does exist"}]},{"t":"p","runs":[{"text":"The AI Incident Database","b":true},{"text":", run by the Responsible AI Collaborative, catalogues publicly reported incidents (over a thousand and growing) with structured fields, and is free to search. It is compiled from public sources, so it captures what was reported and not what happened."}]},{"t":"p","runs":[{"text":"The OECD’s AI Incidents Monitor","b":true},{"text":" tracks incidents from news sources internationally, and the OECD has been working on a common incident reporting framework so that different countries’ regimes can share definitions."}]},{"t":"p","runs":[{"text":"The EU AI Act creates a duty.","b":true},{"text":" Providers of high-risk systems must report serious incidents to authorities. This is the first significant mandatory regime and it applies to a defined subset, not to everything."}]},{"t":"p","runs":[{"text":"Sectoral regimes already bite.","b":true},{"text":" Medical device regulators have adverse-event reporting that covers AI-based devices. Financial regulators require operational incident reporting. Data protection authorities require breach notification, often within 72 hours."}]},{"t":"h2","id":"what-counts-as-an-incident","runs":[{"text":"What counts as an incident"}]},{"t":"p","runs":[{"text":"Broader than most people assume, and defining it narrowly is how organisations end up reporting nothing."}]},{"t":"p","runs":[{"text":"A harm that occurred. A near-miss caught before it reached anyone. A system behaving outside its stated scope. A material accuracy degradation. A discovered disparity between groups. An unauthorised use. A confidentiality breach through an AI tool. A supplier’s model changing under you."}]},{"t":"p","runs":[{"text":"The near-misses are the valuable ones, because they are frequent and cheap. Any reporting culture that only captures realised harm is capturing the small tail of its available information."}]},{"t":"h2","id":"building-this-where-you-are","runs":[{"text":"Building this where you are"}]},{"t":"p","runs":[{"text":"You do not need a national framework to do this internally."}]},{"t":"p","runs":[{"text":"Define an incident in writing","b":true},{"text":", including near-misses, and give examples so people recognise one."}]},{"t":"p","runs":[{"text":"Make reporting non-punitive and mean it.","b":true},{"text":" If the first report leads to a disciplinary process, it will be the last report, and thereafter you will have a clean record and no information."}]},{"t":"p","runs":[{"text":"Record a fixed set of fields","b":true},{"text":": what happened, which system and version, who was affected, how it was detected, what was done, what changed as a result. The last field is the one that makes people keep filing."}]},{"t":"p","runs":[{"text":"Convert incidents into tests.","b":true},{"text":" Every incident becomes a permanent case in your evaluation set. This is the single practice that stops the same failure recurring, and it is standard in software engineering and rare in AI deployment."}]},{"t":"p","runs":[{"text":"Review them together, on a schedule.","b":true},{"text":" Individually they are anecdotes; twenty of them are a pattern, and the pattern is usually not what anyone predicted."}]},{"t":"p","runs":[{"text":"Watch the count.","b":true},{"text":" Zero reported incidents means the reporting is broken, not that nothing happened."}]},{"t":"h2","id":"reporting-outward","runs":[{"text":"Reporting outward"}]},{"t":"p","runs":[{"text":"When an incident affects people outside your organisation, the duty may be legal: data protection authorities, sectoral regulators, and under the EU regime, market surveillance authorities. Beyond duty, submitting to a public database is a small act with disproportionate value, because the field’s collective knowledge of how these systems fail is currently assembled from whatever happened to become news."}]}],"check":{"question":[{"text":"Why is the US aviation voluntary reporting system deliberately run by NASA rather than by the aviation regulator?"}],"options":[{"text":[{"text":"Because NASA can analyse the data better than the aviation regulator could do"}],"correct":false,"tempting":[{"text":"Capacity is not the reason. The choice is about who receives a report, not who can process it."}]},{"text":[{"text":"Because many incidents involve research aircraft that fall under NASA’s remit"}],"correct":false,"tempting":[{"text":"Invents a jurisdictional reason. The system covers ordinary commercial flying, not research flights."}]},{"text":[{"text":"To keep reports secret from the public, since publishing them would damage trust"}],"correct":false,"tempting":[{"text":"Inverts the model. Aggregated findings are published so the industry can learn from them."}]},{"text":[{"text":"So reports go to a body that cannot punish, and people will admit their own mistakes"}],"correct":true,"tempting":null}],"order":[2,1,0,3],"why":[{"text":"People report their own errors only to someone who cannot penalise them for it."}]},"prev":{"slug":"standards-and-audits","title":"What an audit can and cannot see","minutes":7},"next":{"slug":"the-two-debates","title":"The two arguments, and how to read them","minutes":9}}}