Machine Learning, Foundations

Vidya Path enrols 2,300 students across one-year and two-year programmes for the engineering entrance exam. Fees arrive in three instalments. A student who stops coming after the second instalment costs the centre the third, and costs the family a great deal more than that.

The management asked for a model that would “predict dropouts early enough to do something about it”. They had five years of attendance registers, weekly test scores, fee receipts and hostel allocations, and they had Reshma, moved across from accounts, who had worked through a free machine learning course on her phone over four months.

Her first week produced no code. It produced one sentence, and an argument about it.

The row was easy. One student-month. A student appears once for every month they were enrolled.

The target was not. Two definitions were available, and they were not variations on a theme.

Definition A: a student is a dropout if they do not re-enrol for the following term. Unambiguous, already recorded in the fee system, and exactly what the director meant by the word. It also produces one label per student per year: about 2,300 rows a year and roughly 180 positives.

Definition B: a student is a dropout if they attend no class for twenty-one consecutive days. One label per student-month, which over five years is about 27,000 rows and roughly 1,900 positives. It is also wrong sometimes. A student with dengue, or one who went home for a cousin’s wedding and came back, gets the same label as one who has left for good.

The director wanted A. “That is what a dropout is.” Reshma wanted B, for a reason that had nothing to do with elegance. With 180 positives and forty candidate features, the events-per-feature arithmetic said nothing she fitted would be stable, and a validation set holding thirty-six positives could not tell a good model from a lucky one.

The cost fell on both sides and neither was small. Definition A would produce a model whose per-student predictions could not be trusted, wearing a validation score with an interval wide enough to cover both success and failure. Definition B would produce a model that fired on students who were about to walk back through the gate, and every false alarm was a counsellor telephoning a family that had done nothing wrong. In Kota, where the pressure on these students is a matter of public concern, an unnecessary call from the institute is not a neutral event.

The moment of prediction settled less than either of them expected, and it settled it the same way for both definitions. Features had to be everything known as of the last day of the month and nothing else. That excluded refund_processed, hostel_vacated_date and final_instalment_status, all three of which were in the export the office had handed over, and all three of which are written after the outcome they would have been used to predict. Reshma put the exclusion list at the top of the notebook before she opened a single CSV.

The baseline took twenty minutes and changed the conversation. Predicting “nobody drops out” scored ninety-three per cent under definition A. More usefully, the rule the counselling team already used (flag any student whose attendance over the last fortnight fell below sixty per cent) caught forty-one per cent of eventual dropouts and produced about nine flags a day, which two counsellors could just about work through.

Nine flags a day was the real constraint, and nobody had written it down before. The question stopped being “can we predict dropouts” and became “can we beat forty-one per cent recall at nine flags a day, without asking the counsellors to make calls they do not have time to make”.

Saathi Sahyog lends between twelve and sixty thousand rupees to women’s self-help groups across four districts of north Bihar. Repayment is weekly and collected in person. The decision they wanted to support is the one a field officer makes on a doorstep: recommend this group’s loan for approval, or refer it for a second visit.

The existing instrument is a paper scorecard with eleven questions and a total out of one hundred. It was built in 2017 by a consultant nobody can now reach, and everyone in the field trusts it: partly because they can see it work and partly because a number they can compute themselves is a number they can argue with.

The analytics team, Farhan and Kavya, fitted a logistic regression on 38,000 completed loans with fourteen features. The target was ninety days past due within the loan term. They fitted it in an afternoon and spent three weeks on what it said.

Two features were the problem: monthly_household_income, collected by asking, and weekly_instalment_as_share_of_income, computed from that same declared income and the loan amount. They are close to mechanically related. In the fitted model, income carried a coefficient of +1.9 on the log-odds of default and the instalment share carried −2.4. Read literally, that says richer households default more and larger instalments relative to income make default less likely. Both readings are absurd, and both are what the numbers said.

Kavya ran the diagnostic from the coefficients lesson: refit on five bootstrap samples. Income’s coefficient changed sign in two of them. The predictions barely moved across the five refits; the coefficients moved a great deal. That is multicollinearity behaving exactly as documented, and it is invisible in any accuracy score.

Now the decision, and it had a genuine cost on each side.

Option one: drop declared income. The instalment share carries most of the information, the remaining coefficients stabilise, and the model becomes explainable. The cost is that income is question three on the paper scorecard. Removing it from the model means the field officers’ instrument and the office’s instrument now disagree about what matters, and forty people who have collected that number every week for nine years are told it is no longer used. Farhan’s estimate, from two conversations in the field, was that a proportion of officers would conclude the office did not understand their job and would stop trusting the output entirely.

Option two: keep both columns and add a ridge penalty. Ridge shares the coefficient across correlated features rather than picking one arbitrarily, and the signs settle. The cost is that the individual coefficients no longer support the sentence the lender needs. A rejection notice under Reserve Bank of India expectations has to carry a reason, and “your score is low because of the combined weighting of two related fields” is not a reason a borrower can act on.

Option three: ship the boosted model instead. They had one. It scored 0.79 AUC against the linear model’s 0.74 and the paper scorecard’s 0.68. It also has no coefficient to read, and the field officer cannot compute it on a doorstep with no signal, which happens on roughly a fifth of visits.

Five points of AUC against forty people’s willingness to use the thing. Nobody in the room thought that was an obvious trade.

Aruna Knits ships about 900 export consignments a year. A consignment that misses its committed date triggers a penalty clause with two of the three buyers and, more expensively, moves the exporter down that buyer’s allocation list for the next season. The managing director wanted to know, at the moment a purchase order is confirmed, which consignments were at risk, so that the production plan could be arranged around them.

The work went to a two-person consultancy in Coimbatore. They were given a warehouse extract of 4,100 historical consignments with sixty-one columns, and four weeks.

Week two produced a gradient boosting model at 0.94 AUC on a held-out sample. The consultants sent a slide. The managing director sent it to his buyers’ merchandising contacts. The number was, by that point, a commitment.

Week three, one of the consultants ran the check from the leakage lesson: sort the features by importance and interrogate anything that dominates. One column did. courier_partner_assigned was carrying more than a third of the model’s gain.

The reconstruction took an afternoon. Consignments that were running on time were handed to the regular sea-freight forwarder and had that value from day one. Consignments that had slipped were reassigned to an air-freight partner, some days before the committed date, precisely because they were late. The column existed at prediction time (it was populated), but not with the value it holds in the training table. That is the second half of the leakage test, and it is the half people skip.

Two more columns were in the same family, found by the same route: overtime_hours_sanctioned, approved when a line is behind, and fabric_reissue_count, which is incremented when a cut is rejected and the cutting has to be redone, an event that happens after the consignment is already in trouble.

Removing all three took the model from 0.94 to 0.71.

The decision was now uncomfortable in a specific way. 0.71 is not a bad model. Against the exporter’s existing practice, the production manager’s judgement, which had been scored on the same 4,100 consignments at roughly 0.63, it is a real improvement worth having. But 0.94 had been on a slide, and the slide had left the building.

Report the 0.71 and explain. The cost is a conversation the consultancy did not want to have, with an exporter who had already told his buyers something, and a real chance of losing the remaining fortnight of the contract and the referral behind it.

Keep the three columns and add a caveat. They could have argued, and it is the sort of argument that wins: the columns are in the operational system, a prediction can be produced from them, the model will run. It would run and it would be useless, because at the moment the managing director needs the answer (purchase order confirmed, twelve weeks out) every consignment has the regular forwarder, no overtime and no reissues, and the model would return the same low risk for all of them. It would fail quietly and it would fail in a way that looks like the world being unpredictable rather than the model being wrong.

Split the difference: build two models. One at order confirmation with the honest features, and one at week eight with everything then known. This is more work than the contract covers, and it is the only version where the reassignment column is a legitimate input, because by week eight it means what it appears to mean.

Kisan Drishti sends a weekly SMS to about 14,000 cotton farmers. The product they were building would upgrade that to a warning: pink bollworm pressure is rising on your plot, scout it this week. The data was three seasons of scouting records from 1,900 plots, each visited fortnightly by a field agronomist who counted larvae in ten randomly chosen bolls, plus weather, sowing date, variety and previous-season history.

The row was one plot-fortnight. The target was whether the next visit’s count crossed the economic threshold. About 78,000 rows and 6,300 positives.

The first model, a gradient boosting classifier with a random eighty-twenty split, scored 0.88 AUC. The deck was written. The number went on slide nine.

Their advisor, an agricultural statistician at a state university, asked one question: how many rows do you have per plot?

Forty-one, on average. Three seasons of fortnightly visits.

A random row split scatters a plot’s forty-one visits across training and validation. The model does not need to learn anything about pest pressure to score well; it needs to learn that plot 1,183 in Yavatmal tends to cross the threshold, which it can do from the other forty visits sitting in the training set. And every plot in the validation set is a plot the model has seen thirty-odd times.

There was a second problem underneath the first. Pest pressure moves regionally within a season. A random split lets the model train on the third week of August while predicting the second, in the same district, in the same year. Neither of those situations occurs when the product runs, because the product runs forward in time on plots whose future is unknown.

They rebuilt the evaluation properly: group by plot, and split by season, training on seasons one and two and validating on season three.

0.62.

Before accepting it they ran one more check, and it is the cheap one. They permuted the target column, retrained the whole pipeline, and looked at the score. Under the grouped, season-ordered split it collapsed to 0.50, which is what a correct pipeline does. Under the original random split it came back at 0.57: well above chance on shuffled labels, which is only possible if something in the pipeline is reaching the answer by a route nobody intended. The route was a per-plot aggregate feature, computed across the whole dataset before splitting, which carried a fingerprint of each plot’s own outcomes into every one of its rows.

0.62 was not a disaster. The baseline (warn everybody every fortnight during the known risk window) is what the extension service effectively does, and against a like-for-like measurement the model was still adding something. But 0.88 was on slide nine, four weeks out.

The decision the founders faced had two costs and no clean side.

Present 0.62 with the explanation. It is defensible, it is the number the product will deliver, and it invites a conversation about validation design with investors who may or may not want to have it. It also invites a comparison against competitors quoting 0.9-something on random splits, which several of them were.

Present 0.88 with a footnote. Nobody would have caught it in the room. The cost arrives later, when the pilot in a fourth district produces field results nowhere near the slide, at which point the gap has to be explained to people who have already given money.

Delay the round by a season and collect a fourth year on new plots. Cleanest evidence, and the runway did not obviously cover it.

The fraud desk at Janseva Sahakari Bank reviews flagged card transactions and telephones the cardholder. Six analysts, each managing about forty reviews in a shift, which is a hard ceiling of roughly 240 reviews a day including the ones that arrive from other channels.

Their model, built the previous year, scored transactions and flagged anything above 0.5. That produced about 90 flags a day at 31 per cent precision, and caught an estimated 54 per cent of fraud by value.

Three things happened in one quarter.

First, the risk committee asked why the threshold was 0.5. Nobody could answer. It was the library default, applied by a contractor who had left. The analyst who inherited the model, Meghana, worked the arithmetic from the cost-sensitive lesson. A missed fraud on this portfolio costs the bank ₹14,200 on average once the chargeback and the investigation are counted. A review costs about ₹110 of analyst time. The break-even threshold is 110 divided by 14,310, which is 0.0077.

At 0.0077 the model flags roughly 2,400 transactions a day. The desk can review 240.

That is not a modelling failure and it is not a mistake in the arithmetic. It is two constraints that do not both fit, and Meghana wrote both numbers on the whiteboard rather than choosing between them quietly. The cost-optimal threshold says the bank is under-investing in the fraud desk by a factor of ten. The capacity threshold says today’s flag list has to be 240 long.

Second, she computed the per-row threshold instead. Review cost divided by review cost plus the transaction amount. A ₹110 review against a ₹4,80,000 transaction gives a threshold near 0.0002: flag on almost any suspicion. Against a ₹900 transaction it gives 0.109. Applied at a fixed capacity of 240 flags a day, ranked by expected loss rather than by probability, the recovered value rose by 38 per cent on a back-test with no change to the model and no extra analyst.

Third, and this is the part that nearly went wrong, the bank absorbed the card portfolio of a smaller co-operative bank in a neighbouring district. Forty thousand new cards, a different customer profile, and a fraud base rate of 0.09 per cent against Janseva’s own 0.31 per cent.

The model was scored on the new portfolio and its AUC held: 0.91 against 0.92. The committee read that as the model transferring cleanly and asked for it to go live on the new cards at the same operating point.

Meghana’s objection was the base rate. Precision is not a property of a model. On the acquired portfolio, at a threshold producing the same flag rate, precision would fall from 31 per cent to roughly 9 per cent: three times as many wrongly telephoned customers per genuine case, on a customer base that had just been told its bank had been taken over.

The decision had a cost either way. Running it as-is meant a queue that was 91 per cent wrong, on new customers, with analysts who would learn within a fortnight to distrust the flags. Not running it meant the acquired portfolio was covered by nothing but the old bank’s rules for at least a quarter.

Annadhan Warehousing stores wheat, gram and soya for traders and for the state procurement agency. Spoilage (grain downgraded on outward inspection) costs the company between two and four crore rupees a year and costs it customers, which is worse.

They had four years of records: 310,000 stack-months, one row per stack per month, with moisture at intake, variety, stack height, days in store, fumigation history, ambient humidity from the nearest weather station, warehouse identifier, and whether the stack was downgraded at outward inspection.

A gradient boosting model reached 0.83 AUC against a baseline of 0.61. That part was uncontroversial and useful: the model was going into the fortnightly inspection schedule, deciding which stacks got opened first.

The controversy was the second slide. Somebody had printed model.feature_importances_ and the top feature, well clear of everything else, was warehouse_id. The operations director read that as the model saying the warehouses themselves were the problem, and the refurbishment paper was drafted around the three warehouses whose identifiers the model split on most often. Nine crore rupees.

The analyst, Devika, had been in the job four months and asked for a week.

Her first test took an hour and is the one from the importance lesson. She added a column of random integers between one and ten thousand, with no relationship to anything, and refitted. The random column came fourth. Nothing was broken. Impurity importance rewards a feature for offering many candidate thresholds, and warehouse_id has forty-six levels while variety has four and fumigated_last_30d has two. A high-cardinality column accumulates gain by having more chances to reduce impurity, whether or not it carries signal.

Permutation importance on held-out data told a different story: moisture at intake first, days in store second, ambient humidity third, warehouse identifier seventh and with an interval that crossed zero.

Then the correlated-feature problem arrived, because these two are not independent. Warehouses in the Hoshangabad belt receive grain from a catchment with systematically higher moisture at intake. Permuting warehouse_id alone barely moved the score, because the model simply read moisture instead. Permuting moisture_at_intake alone barely moved it either, because the model read the warehouse. Both looked unimportant, which is the failure the lesson warns about, and taken literally it would have said nothing mattered.

Devika clustered the correlated columns and permuted the group. Moisture-and-location together accounted for most of the model’s dependence. Which of the two is doing the work is not a question the model can answer, because in this data they never move independently.

The decision, with eleven days left, was genuinely hard.

Recommend the refurbishment as drafted. The three warehouses are in the high-moisture belt and they do have the worst spoilage. Something is wrong there. Spending nine crore on ventilation and flooring is not obviously wasted, and the paper is written.

Recommend intake moisture control instead: testing at the gate, rejection thresholds, a drying arrangement with the procurement agency. Cheaper, and it is what the permutation grouping points at. It is also an intervention, and the model is an association. Nothing in the analysis establishes that lowering intake moisture at those sites lowers spoilage, because intake moisture has never varied at those sites independently of everything else about them.

Ask for the board paper to be delayed by a season and run the only thing that would settle it.

The helpline receives calls about domestic violence, workplace harassment, legal questions, mental distress and a long tail of everything else. Counsellors record a structured summary at the end of each call: category, district, caller age band, whether a referral was made, duration, time of day, whether the caller had called before, and a free-text note.

A proposal had been funded to redesign counsellor training around “caller segments”: the idea being that a small number of recognisable caller types exist, that each needs a different opening approach, and that training modules could be built for each. The proposal named six segments as a target, because six is the number of training modules the calendar could absorb.

Nandana, the research officer, was asked to find them. She had 62,000 records, no labels, and a genuine question with money behind it.

She did the work properly. Categorical fields one-hot encoded, duration log-transformed, everything standardised, k-means for k from two to ten, with n_init left at ten because lowering it to save time was not something she had measured.

The elbow plot was a smooth decay with no corner. Three colleagues looking at it picked three different values.

Silhouette scores ran between 0.11 and 0.17 across every k she tried. The rough guide in her notes said above 0.5 is solid structure, 0.25 to 0.5 is weak, and below 0.25 means there is probably nothing worth naming.

The gap statistic, which is the only one of these that is allowed to answer “one”, preferred k equal to one. She checked it the cheap way as well, clustering a shuffled copy of the data and comparing, and the shuffled version looked much like the real one.

The stability test was the clearest. Two random eighty per cent subsets, clustered separately, compared on the rows they shared with the adjusted Rand index. At k equal to six the agreement was 0.31. Rerunning with a different seed produced six different groups containing largely different callers, all of which could be described and named, none of which reproduced.

She also ran HDBSCAN, which is allowed to decide the count itself and to label points as noise. It found two small dense groups (repeat callers from two districts with an unusual referral pattern, about 900 records between them) and called everything else noise.

Now the decision.

Deliver six segments. She could. k-means returns six clusters from anything, the centres are describable, and a page of prose about “the hesitant first-time caller” and “the informed repeat caller” would read well and would survive a steering committee. The training modules would be built and would probably do some good, because good counsellor training does some good regardless of whether its organising scheme is real.

Report that there are no segments. True, defensible, and it terminates a funded project. It also tells a room of people who have committed to a plan that the plan’s premise does not hold, which is a specific kind of unpopular.

Deliver the two small dense groups and nothing else. Honest, and 900 records out of 62,000 is not a training programme.

Farmers in the district send photographs to a WhatsApp number when a crop looks wrong. An officer looks at each one and replies. The volume had grown to about ninety photographs a day in the season, against one officer, and the reply time had stretched to two days, by which point spraying advice is worth much less.

The proposal was a classifier that would sort incoming photographs into five classes (leaf curl virus, anthracnose, thrips damage, nutrient deficiency, and healthy or unclear) so that the officer worked the queue in a sensible order rather than in arrival order.

The labelled set was 340 photographs, labelled by the officer himself over one season. Not thousands. Three hundred and forty, unevenly split, with the smallest class holding 31.

Prakash had worked through the transfer learning lesson and knew the recipe: feature extraction first, then last-block fine-tuning, then full fine-tuning, stopping at whichever step stopped improving. He also had a free GPU quota and an understandable wish to use it.

Step one took forty minutes. A pretrained ResNet-50 with the classifier head removed, run in evaluation mode over the 340 images to produce 2,048-dimensional feature vectors, then a logistic regression on top. Grouped five-fold cross-validation, grouped by the farmer who sent the photograph, because eleven farmers had sent more than five images each and several had photographed the same plant repeatedly. Macro F1 of 0.71.

Step two, fine-tuning the last block, took two hours including the mistakes. Macro F1 of 0.74, with a spread across folds wide enough that he could not be confident the improvement was real.

Step three, full fine-tuning, produced 0.62 and a training curve that fell off a cliff in the first epoch. That is catastrophic forgetting with the learning rate set too high, and dropping it by a factor of thirty recovered to 0.73: no better than step two, at eight times the training time and with a model he could no longer explain the behaviour of.

So the measurement said step one or step two, and the difference between them was inside the noise. That was not the hard decision.

The hard decision was what the grouped cross-validation had exposed on the way. Every photograph in the training set came from a smartphone camera, in daylight, held about thirty centimetres from the leaf, by somebody who had been told by the officer how to take the photograph. The officer had, in effect, curated his own training distribution over a season of replying to people.

The incoming queue is not that. It contains photographs taken at dusk, at two metres, of a whole plant, through a polythene tunnel, and by farmers who have never been told anything about how to hold a phone.

Deploy the 0.74 model on the whole queue. It is a triage aid, not a diagnosis, and a wrong ordering is not a wrong prescription. The cost is that its confidence on out-of-distribution photographs is not lower than on good ones (a network is typically over-confident, and increasingly so as it gets larger) so the queue would be ordered wrongly with no signal that anything was amiss.

Deploy it only on photographs that resemble the training set, using reconstruction error or a simple image-quality check as a gate, and route everything else to the officer in arrival order. Less coverage, and it requires building the gate.

Spend the season collecting a representative labelled set first. Correct, and it means the reply time stays at two days for another year.

Sanjeevini Pharmacy forecasts daily demand per stock-keeping unit per branch, about 1.4 million forecasts a night, feeding an automatic replenishment order. When the forecast is low, a branch runs out and a customer walks to the shop next door. When it is high, slow stock ages towards expiry.

The model had been stable for fourteen months. Weighted mean absolute percentage error sat between 18 and 21 per cent. In the second week of March it went to 34 and stayed there.

The commercial director asked for a retrain by Friday. The reasoning was not unreasonable: the model is stale, the world has moved, retrain it on recent data. The data team’s lead, Anwar, asked for two days to diagnose first, and had to argue for them, because two days is two days of stockouts.

The monitoring told him something before he opened a notebook. System health was clean. Prediction distributions had shifted downward: the model was forecasting less than it had. Input distributions had a specific and loud alarm: for one feature, pack_size_units, the missing rate had gone from under one per cent to 23 per cent on the eighth of March and stayed there.

That is a covariate shift in the monitoring’s language and a data fault in reality. Twenty-three per cent is not a population moving. It is something upstream breaking on a particular day.

The trace took four hours. The procurement team had migrated their supplier master to a new system on the seventh of March. In the new system pack size for one category of products is recorded in a separate strength field, and the old column is null for anything created after the migration. Nobody told anyone, because from procurement’s point of view nothing had broken.

There was a second effect stacked on the first, and it is the one that would have made a retrain actively harmful. The imputer in the pipeline filled the missing pack size with the training median. For the affected category (largely paediatric syrups) the median across the whole catalogue is meaningfully smaller than the true value, so the forecast for those lines was systematically low, which is exactly the error pattern the branches were reporting.

Now the decision, and both options had a cost that landed on somebody.

Retrain on recent data, as asked. It would have produced a better-scoring model, because a model trained on the last twelve weeks learns that pack size is missing for a quarter of rows and adapts to predict without it. The score recovers. The forecasts stay wrong in a new way, the fault stays in the upstream system, and the next person to look at it inherits a model that has quietly encoded a broken join. Retraining teaches the model the broken data.

Fix the upstream mapping. Correct, and it required a change from a procurement team in the middle of a migration, with no incentive and no spare week, and it would take longer than Friday.

Patch the pipeline to read the new strength field when the old column is null, and leave the model alone.