Python, from Zero, for AILesson 76 of 89

Module 8 · Numbers in bulk: NumPy, pandas and the vector under every model

Cleaning a table: missing values, the string that is not missing, and duplicates that are not identical

In this lesson

  • 9 min read
  • 8 reading sections
  • Knowledge check
  • Lesson 7 of 10 in this chapter

This chapter’s goalLoad a messy CSV into pandas, clean its missing values and types, group and join it into the table a question needs, convert it to a NumPy array of the right shape and dtype, compute cosine similarity between one vector and a hundred thousand in a single vectorised expression, plot the result to a file, and explain from memory layout why the array version runs a hundred times faster than the loop it replaced.

This chapter’s media

The film explains chapter ideas. The audiobook narrates the complete teaching text.

FormatEnglishहिंदी
Chapter filmWatch1:43Not available
Full audiobookNot availableNot available
One column of 1,000 rows: what counts as missingReal NaN, which isna()sees42 rowsThe string N/A61 rowsA single hyphen28 rowsEmpty once stripped ofspaces17 rowsThe string Unknown9 rowsdf.isna().sum() reports 42. The column is actually missing 157 values, sixteen per cent rather thanfour, and every mean, count and groupby computed before the sentinels were converted is wrong by thatmuch, with no error anywhere. Pass the na_values list at read_csv, or replace afterwards, then checkvalue_counts on every string column.
df.isna().sum() reports 42. The column is actually missing 157 values, sixteen per cent rather than four, and every mean, count and groupby computed before the sentinels were converted is wrong by that much, with no error anywhere. Pass the na_values list at read_csv, or replace afterwards, then check value_counts on every string column.

Figure

A little at a time

Explore an idea, try it, then check what stays.

Step 1 of 10Free · no account needed
Lesson steps

Missing is a value, until it is not

pandas marks a missing value as NaN (the floating-point not-a-number) or None in object columns, and a family of functions understands it: isna, notna, fillna, dropna, and every aggregation, which skips it by default. A column with three missing prices has a mean over the other rows, not NaN.

None of that machinery sees a missing value that arrived as a string. Real files mark absence with N/A, NA, -, ?, Unknown, none, null, an empty string, a single space, or 0 where zero is impossible. To pandas those are all present values. The first job of cleaning is to make missingness real:

Python
SENTINELS = ["N/A", "NA", "n/a", "-", "?", "", " ", "Unknown", "unknown", "null", "None"]
df = pd.read_csv(path, na_values=SENTINELS, keep_default_na=True)

or after the fact:

Python
df["city"] = df["city"].replace(SENTINELS, pd.NA)

Your place and first choices stay on this browser. Classroom credit is recorded separately.

Stuck on something?Ask Divya, your AI tutor

How to cite: Addaly (2026) ‘Cleaning a table: missing values, the string that is not missing, and duplicates that are not identical’, in Python, from Zero, for AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/python-for-ai/cleaning-data-in-pandas (Accessed: 21 November 2026).