AI, Safety and What Goes WrongLesson 70 of 73

Module 9 · Living with it: cost, governance and your own practice

Learning from what went wrong

In this lesson

  • 8 min read
  • 5 reading sections
  • Knowledge check
  • Lesson 5 of 8 in this chapter

This chapter’s goalSet a written policy naming which tasks AI may touch, the verification each requires and the conditions under which you would stop, and locate the regulator, standard or reporting route that applies to a given system.

This chapter’s media

The film explains chapter ideas. The audiobook narrates the complete teaching text.

FormatEnglishहिंदी
Chapter filmNot availableWatch3:00
Full audiobookNot availableNot available
A reporting loop you can build without a regulatorDefine anincident,includingnear-missesWith examplespeoplerecogniseMakereportingnon-punitive,and mean itThe firstdisciplinaryprocess endsthe reportsRecord fixedfieldsWhat, whichsystem andversion, whowas affected,how detected,what changedConvert eachincidentinto a testcaseA permanentitem in theevaluationsetReview themtogether, ona scheduleTwentyanecdotes area patternWatch thecountZero is abrokenchannelAviation’s record rests on reporting that does not go to the enforcer. The near-misses are where thelearning is, and a count of zero means the reporting is broken, not that nothing happened.
Aviation's record rests on reporting that does not go to the enforcer. The near-misses are where the learning is, and a count of zero means the reporting is broken, not that nothing happened.

Figure

A little at a time

Explore an idea, try it, then check what stays.

Step 1 of 7Free · no account needed
Lesson steps

The comparison that keeps being made

Commercial aviation is extraordinarily safe, and the usual explanation is technology. The better explanation is reporting. Aviation built, over decades, a system in which failures are reported, investigated by an independent body, published in full, and turned into changes that everyone must adopt.

Two features do the work.

Mandatory investigation with independence. Accident investigators are separate from the regulator and from the operators, publish findings including uncomfortable ones, and their remit is causes rather than blame.

Confidential, non-punitive voluntary reporting. The US Aviation Safety Reporting System, run by NASA rather than the aviation regulator specifically so that reporting does not go to the enforcer, receives many tens of thousands of reports a year from pilots and controllers describing their own mistakes. Reporting confers limited protection from enforcement action. That protection is why the reports exist, and the near-misses are where most of the learning is.

Your place and first choices stay on this browser. Classroom credit is recorded separately.

Stuck on something?Ask Rukmini, your AI tutor

How to cite: Addaly (2026) ‘Learning from what went wrong’, in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/incident-reporting (Accessed: 21 November 2026).