What a row is, and why that decision is the whole project

The two objects

Almost all of classical machine learning operates on two objects. A matrix X with one row per thing and one column per measurement, and a vector y with one entry per row: the answer you want the model to produce.

import numpy as np
X = np.array([[35, 42000, 3],
              [51, 88000, 0],
              [24, 19000, 7]])   # age, annual income, late payments
y = np.array([0, 0, 1])          # defaulted within 12 months?
X.shape, y.shape   # ((3, 3), (3,))

That is it. Three rows, three columns, three labels. Everything later in this course (trees, boosting, neural networks) is a different way of turning the left object into a guess at the right one.

The interesting part is not the code. It is the sentence you had to commit to before you could type it.

“One row is one ____”

Finish that sentence out loud before you write anything. It is called the unit of analysis, or the grain, and getting it wrong is the most expensive mistake in this course because nothing downstream complains.

A bank asks: can we predict default? There are at least four honest datasets hiding in that question.

• One row is one customer. Then a customer with eleven loans appears once, and you must decide what “defaulted” means across eleven loans.
• One row is one loan. Then the same customer appears eleven times, and eleven rows share almost every column. Your model will learn that customer, and when you split the data randomly, that customer appears on both sides of the split. The score comes out beautifully and means nothing.
• One row is one customer-month. Now you can predict default in the next 30 days, which is a different and often more useful product.
• One row is one application, including the ones you rejected. This is the only grain that can answer should we have approved it, and it is the one you almost never have labels for, because rejected applicants never got the chance to repay.

Those four datasets have different sizes, different targets, different splits and different deployments. Choosing between them is a product decision wearing a data costume.

The target needs a clock

y is not a column you find. It is a definition you write, and it has to contain a moment.

“Churned” is not a label. “Had no transaction in the 60 days following the observation date” is a label. “Fraud” is not a label; “was charged back within 90 days” is. The moment matters because the label has to be knowable later than the features and not before, and because a shorter window gives you more labelled rows but a noisier target, while a longer one gives cleaner labels and less recent data. There is no right answer, only a stated one.

Write it as one line at the top of the notebook:

Row: one loan application. Features: everything known at the moment of submission. Target: 1 if 90 days past due within 12 months of disbursal, else 0. Excluded: anything recorded after disbursal.

That last clause is the one that saves you. We will return to it in the leakage lesson, but the guard is written here, at the point where the dataset is defined.

Columns have types, and the type changes the maths

Three kinds of column behave differently and need different treatment later:

• Numerical: age, income, distance. Order and distance both mean something. income - 1000 is meaningful.
• Categorical: city, product code, device. Values are names. Mumbai + 1 is nonsense, and the moment you store cities as 0, 1, 2 you have told a linear model that Chennai sits between Mumbai and Delhi.
• Ordinal: education level, star rating, T-shirt size. Order means something, distance does not. The gap between 1 star and 2 is not the gap between 4 and 5.

An ID column belongs to none of these and should almost never be a feature. A model that gets customer_id will happily memorise it.

Shape errors are the beginner’s rite of passage

Most first-week errors are shape errors, and the message is usually accurate but blunt. Two habits fix nearly all of them: print X.shape and y.shape after every transformation, and keep the convention that rows are samples and columns are features: scikit-learn assumes it everywhere, and a transposed matrix trains without complaint and predicts nonsense.

A single sample has to be reshaped to a matrix of one row before you can predict on it:

model.predict(np.array([[35, 42000, 3]]))   # note the double brackets

The limitation

This rectangle is a strong assumption, and it is where a lot of real data does not fit. Text, images, audio, graphs and event streams are not naturally rectangular. What we do (and this is worth seeing clearly rather than absorbing by osmosis) is force them into the rectangle: counts of words, pixels flattened into columns, an event log aggregated into “number of logins in the last 7 days”. Every one of those is a lossy choice made by a person, and most of the accuracy in a classical project comes from making those choices well rather than from the algorithm that eats the result.

Deep learning’s claim, which we will reach in module 8, is that it can learn the flattening itself. It is a real claim. It is also why it needs so much more data.