Addaly is in open beta. Things will change, and AI answers can be wrong — check anything that matters.

AI, Safety and What Goes Wrong

The failure modes of AI, stated plainly, with the numbers.

Lesson 37 of 739 min

When the gate does not recognise you

A different kind of error

The failures in the last lesson were wrongful accusations. This one is about wrongful absence: a system that simply does not recognise you, and therefore cannot give you what you are entitled to.

Biometric authentication is now the gate to entitlements for hundreds of millions of people. India's Aadhaar programme has enrolled over 1.3 billion people, and fingerprint or iris authentication is used at ration shops, for pension disbursement, for employment guarantee wages and for many other services. Similar systems operate in Nigeria, Kenya, Pakistan, Indonesia and across Latin America.

The technology works most of the time. That sentence contains the entire problem, and it is worth working out why.

The arithmetic of a small failure rate

Suppose fingerprint authentication succeeds 98% of the time. That sounds excellent, and by the standards of most systems it is.

Apply it to 800 million ration beneficiaries drawing food monthly. Two per cent is 16 million failures a month. Now add the decisive detail: the failures are not randomly distributed. The same fingerprint fails repeatedly. Manual labourers wear their fingerprints down. Elderly people's ridges flatten. People with leprosy, certain skin conditions or amputations may have no usable print at all. Poor network connectivity in remote areas produces failures that look identical to biometric ones from the beneficiary's side.

So it is not 16 million different people having one bad month. It is a much smaller number of people being turned away every single month, forever — and they are disproportionately the oldest, the poorest and the most physically worn, which is to say the people the entitlement exists for.

This is the general lesson and it applies far beyond biometrics: a system with a small error rate that is deterministic in the same individuals is not a system with a small error rate. It is a system with a permanently excluded minority.

Two per cent, counted two waysMeasured per transaction2% of attempts fail16 million failures a month, sounds spreadthinReported as a 98% success rateLooks like a small, even problemMeasured per personThe same worn fingerprints fail every monthManual labourers, the elderly, people withskin conditions, remote areas with poor signalA permanently excluded minorityThe figure no system publishes by default800 million beneficiaries at a 98% success rate is 16 million failed authentications a month. Countedper person, it is a much smaller group turned away every month, and they are the people theentitlement exists for.
Two per cent, counted two waysMeasured per transaction2% of attempts fail16 million failures a month, sounds spreadthinReported as a 98% success rateLooks like a small, even problemMeasured per personThe same worn fingerprints fail every monthManual labourers, the elderly, people withskin conditions, remote areas with poorsignalA permanently excluded minorityThe figure no system publishes by default800 million beneficiaries at a 98% success rate is16 million failed authentications a month. Countedper person, it is a much smaller group turned awayevery month, and they are the people the entitlementexists for.

The evidence, and the disputes

Researchers including Jean Drèze and Reetika Khera have documented authentication failure rates in the public distribution system in Jharkhand and elsewhere, with reported figures in some districts running around a tenth of transactions. Government figures have generally been lower, and methodology is contested — measured at the point of the device, at the shop, or across a month, the numbers differ substantially.

Several reported starvation deaths in Jharkhand from 2017 onwards were linked in media reporting and civil society investigations to ration denial following authentication or linkage failures. State authorities disputed the causal attribution in several cases. Two things can be said honestly: the causal chain in any single death is genuinely contested, and the existence of systematic exclusion at the point of authentication is not.

India's Supreme Court upheld Aadhaar for subsidies and benefits in its 2018 judgment while striking down mandatory linkage for private services, and it directed that no one be denied benefits for want of authentication — a direction whose implementation on the ground has been uneven.

What a well-designed gate looks like

The fixes are known, unglamorous and mostly about what happens when the machine says no.

A guaranteed manual fallback, available immediately, without discretion. Not "come back tomorrow", not "the operator may permit an exception". If the fallback depends on the goodwill of the person operating the device, it will be used least where it is needed most.

Multiple modalities. Iris where fingerprints fail; one-time passcodes where both fail; a face check where the phone is shared.

Offline operation. A gate that requires connectivity has made the network a condition of eating.

Measure exclusion, publish it, and measure it per person rather than per transaction. The number that matters is not the failure rate; it is how many individuals fail repeatedly. No system publishes this by default and it is the only figure that reveals the harm.

Separate identification from entitlement. Deduplication of a beneficiary list is a different job from authenticating a person at the counter, and using one mechanism for both means a technical failure at the counter reads as a fraudulent claim.

The reframe worth carrying

When anyone reports an accuracy figure for a system that gates access to something people need, ask two questions. Who are the failures, and are they the same people every time? And what happens to a person at the moment it fails?

The first question turns a percentage into a population. The second is where the humanity of the design either exists or does not.

The one thing to keep

A 98% success rate on an authentication gate is not a small problem when the 2% is the same worn-fingerprint, elderly, remote population every month — measure failures per person, not per transaction, and guarantee a fallback that needs nobody's permission.

Before you move on

An authentication system reports a 97% success rate across 500 million monthly transactions, and officials describe the 3% as a minor residual. What is the strongest objection?

Pick the one you would defend. Nobody sees your answer.

No ads. No data sale. No public scores on people. Ever.

© 2026 Addaly