{"course":{"slug":"python-for-ai","title":"Python, from Zero, for AI","url":"https://addaly.com/learn/python-for-ai","level":"beginner","track":"build"},"lesson":{"slug":"pandas-dataframes","title":"pandas: a table with named columns, and the four things to look at first","url":"https://addaly.com/learn/python-for-ai/pandas-dataframes","position":75,"of":89,"minutes":9,"answer":"A DataFrame is a dict of typed columns sharing an index; read_csv guesses the types and gets dates and IDs wrong, loc selects by label and iloc by position.","keyIdea":"A DataFrame is a dict of typed columns sharing an index; read_csv guesses the types and gets dates and IDs wrong, loc selects by label and iloc by position, and a column of object dtype is a column pandas could not understand."},"body":{"format":"markdown","text":"## What a DataFrame is\n\nModule 5 read CSV files with the `csv` module, one row at a time as dicts. That is right for streaming and for files whose rows you process independently. When you want to ask questions *across* rows (averages by group, joins, missing-value counts) you want the whole table in memory as columns, and that is pandas.\n\n```python\nimport pandas as pd\n\ndf = pd.read_csv(\"orders.csv\")\n```\n\nA `DataFrame` is best understood as a **dict of columns**, where each column is a NumPy-backed `Series` with one dtype, and all of them share an **index**: the row labels, which default to 0, 1, 2. Everything fast in pandas is fast because a column is an array; everything slow is slow because you made it loop over rows.\n\n## The four things to look at first\n\nBefore any analysis, four commands, every time:\n\n```python\ndf.shape  # (rows, columns)\n# first five rows — does it look like what you expected?\ndf.head()\n# one line per column: int64, float64, object, bool,\n# datetime64\ndf.dtypes\ndf.isna().sum()  # missing values per column\n```\n\n`dtypes` is where the surprises are. `read_csv` guesses each column's type from its contents, and the guesses fail in predictable ways:\n\n- **`object`** means \"strings, or a mix\". A numeric column with one stray value (`N/A`, `7002A`, a trailing space) becomes `object` for every row. Arithmetic on it fails; a merge against an `int64` column of the same IDs matches nothing, because `7002 != \"7002\"`.\n- **IDs read as numbers.** `007002` becomes `7002`, losing the leading zeros that a phone number, a postcode or an account ID needs. Pass `dtype={\"account\": str}` to keep them.\n- **Dates read as strings.** `2026-09-04` is `object` until you say `parse_dates=[\"created_at\"]` or convert afterwards with `pd.to_datetime`.\n\nFix these at read time, not afterwards:\n\n```python\ndf = pd.read_csv(\"orders.csv\", dtype={\"order_id\": str, \"postcode\": str},\n                 parse_dates=[\"created_at\"], na_values=[\"N/A\", \"-\", \"\"])\n```\n\n`df.info()` prints dtypes, non-null counts and memory in one report; `df.describe()` gives min, max, mean and quartiles for numeric columns, which catches a price column with a maximum of 999999.\n\n## Selecting\n\n```python\ndf[\"price\"]                     # one column: a Series\ndf[[\"order_id\", \"price\"]]       # several columns: a DataFrame\ndf[df[\"price\"] > 100]           # rows by condition — a boolean mask, as in NumPy\ndf.loc[df[\"price\"] > 100, \"order_id\"]      # rows by condition, one column\ndf.iloc[0:5, 0:3]               # rows and columns by position\ndf.loc[42]                      # the row whose index label is 42\n```\n\n`loc` selects by **label**: index values and column names. `iloc` selects by **integer position**. They differ the moment the index is not 0..n−1, which happens after filtering or sorting: `df.iloc[0]` is the first row now; `df.loc[0]` is the row that was labelled 0 before, if it survived. Combining conditions uses `&`, `|`, `~` with brackets, exactly as NumPy did.\n\n`df.query(\"price > 100 and status == 'paid'\")` is the same filter as a string, easier to read for long conditions.\n\n## Adding and changing columns\n\n```python\ndf[\"total\"] = df[\"price\"] * df[\"quantity\"]              # vectorised\ndf[\"month\"] = df[\"created_at\"].dt.month                 # datetime accessor\ndf[\"email_domain\"] = df[\"email\"].str.split(\"@\").str[1]  # string accessor\n```\n\nWhole-column operations. The `.str` and `.dt` accessors apply string and date methods across a column without a loop. If you find yourself writing `for i, row in df.iterrows()`, stop: `iterrows` is the slowest way to touch a table, hundreds of times slower than a column expression, and the column expression almost always exists. `df.apply(func, axis=1)` is a little better and still a loop; use it only for logic that cannot be written by column.\n\n## The copy warning\n\n```python\npaid = df[df[\"status\"] == \"paid\"]\npaid[\"flag\"] = True\n# SettingWithCopyWarning: A value is trying to be set on a copy of a slice...\n```\n\n`paid` may be a view of `df` or a copy, and pandas is telling you it cannot promise which, so your assignment may or may not reach `df`. The fix is to say what you mean: `paid = df[df[\"status\"] == \"paid\"].copy()` if you want an independent table, or `df.loc[df[\"status\"] == \"paid\", \"flag\"] = True` if you want to change `df`. Recent pandas versions with copy-on-write enabled make every such selection a copy and the warning disappears; the habit of `.copy()` or `.loc` assignment is right either way.\n\n## Getting back out\n\n```python\ndf.to_csv(\"clean.csv\", index=False)\ndf.to_parquet(\"clean.parquet\")            # needs pyarrow; smaller, typed, fast\ndf[\"price\"].to_numpy()                    # a NumPy array\ndf[[\"x\", \"y\"]].to_numpy(dtype=np.float32) # a (n, 2) matrix for a model\n```\n\n`index=False` on `to_csv` stops pandas writing the row numbers as a column, which otherwise comes back as `Unnamed: 0` on the next read. Parquet preserves dtypes, so the ID-as-string and the parsed dates survive a round trip; CSV forgets them and you re-fix them on every load.\n\n## Try this now\n\nLoad any CSV with at least one ID column and one date. Run the four commands. Find every `object` column and decide whether it should be. Re-read with `dtype=` and `parse_dates=` until `dtypes` says what you mean, then write it to Parquet and read it back."},"check":{"question":"After `df = pd.read_csv(\"orders.csv\")`, `df[\"order_id\"].head()` shows values like `7001`, `7002`, and `df.dtypes` reports `order_id int64`. Sorting by it later puts `10023` after `7002`... correctly, but a merge with another file where the same column reads as `object` matches nothing. What most likely happened in the second file?","options":[{"text":"The second file has an ID like `7002A` or `07002`, so pandas kept the column as strings, and `7002` never equals `\"7002\"`.","correct":true},{"text":"`read_csv` reads the second file with a different encoding, so the digits are different Unicode characters that look identical on screen but hash and compare differently.","correct":false,"tempting":"Digit lookalikes are possible in principle but rare in practice; the ordinary cause is a single non-numeric value forcing the column to strings."},{"text":"Merging requires both columns to be named the same and indexed, and the second file's column is not the index.","correct":false,"tempting":"Merge works on columns regardless of index; a type mismatch between int and str is what stops rows matching."},{"text":"pandas reads `int64` columns as 64-bit but `object` columns as 32-bit, so the values overflow before comparison.","correct":false,"tempting":"`object` columns hold Python objects of any size; there is no 32-bit truncation involved."}]},"prev":{"slug":"vectors-and-cosine-similarity","title":"Dot products and cosine similarity: one vector against a hundred thousand in a single line","url":"https://addaly.com/learn/python-for-ai/vectors-and-cosine-similarity"},"next":{"slug":"cleaning-data-in-pandas","title":"Cleaning a table: missing values, the string that is not missing, and duplicates that are not identical","url":"https://addaly.com/learn/python-for-ai/cleaning-data-in-pandas"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'pandas: a table with named columns, and the four things to look at first', in Python, from Zero, for AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/python-for-ai/pandas-dataframes (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"pandas: a table with named columns, and the four things to look at first","courseTitle":"Python, from Zero, for AI","module":{"index":8,"slug":"numbers-in-bulk","title":"Numbers in bulk: NumPy, pandas and the vector under every model"},"answer":null,"keyIdea":"A DataFrame is a dict of typed columns sharing an index; read_csv guesses the types and gets dates and IDs wrong, loc selects by label and iloc by position, and a column of object dtype is a column pandas could not understand.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-a-dataframe-is","runs":[{"text":"What a DataFrame is"}]},{"t":"p","runs":[{"text":"Module 5 read CSV files with the "},{"text":"csv","code":true},{"text":" module, one row at a time as dicts. That is right for streaming and for files whose rows you process independently. When you want to ask questions "},{"text":"across","i":true},{"text":" rows (averages by group, joins, missing-value counts) you want the whole table in memory as columns, and that is pandas."}]},{"t":"code","lang":"python","label":"Python","text":"import pandas as pd\n\ndf = pd.read_csv(\"orders.csv\")"},{"t":"p","runs":[{"text":"A "},{"text":"DataFrame","code":true},{"text":" is best understood as a "},{"text":"dict of columns","b":true},{"text":", where each column is a NumPy-backed "},{"text":"Series","code":true},{"text":" with one dtype, and all of them share an "},{"text":"index","b":true},{"text":": the row labels, which default to 0, 1, 2. Everything fast in pandas is fast because a column is an array; everything slow is slow because you made it loop over rows."}]},{"t":"h2","id":"the-four-things-to-look-at-first","runs":[{"text":"The four things to look at first"}]},{"t":"p","runs":[{"text":"Before any analysis, four commands, every time:"}]},{"t":"code","lang":"python","label":"Python","text":"df.shape  # (rows, columns)\n# first five rows — does it look like what you expected?\ndf.head()\n# one line per column: int64, float64, object, bool,\n# datetime64\ndf.dtypes\ndf.isna().sum()  # missing values per column"},{"t":"p","runs":[{"text":"dtypes","code":true},{"text":" is where the surprises are. "},{"text":"read_csv","code":true},{"text":" guesses each column’s type from its contents, and the guesses fail in predictable ways:"}]},{"t":"ul","items":[[{"text":"object","b":true,"code":true},{"text":" means “strings, or a mix”. A numeric column with one stray value ("},{"text":"N/A","code":true},{"text":", "},{"text":"7002A","code":true},{"text":", a trailing space) becomes "},{"text":"object","code":true},{"text":" for every row. Arithmetic on it fails; a merge against an "},{"text":"int64","code":true},{"text":" column of the same IDs matches nothing, because "},{"text":"7002 != \"7002\"","code":true},{"text":"."}],[{"text":"IDs read as numbers.","b":true},{"text":" "},{"text":"007002","code":true},{"text":" becomes "},{"text":"7002","code":true},{"text":", losing the leading zeros that a phone number, a postcode or an account ID needs. Pass "},{"text":"dtype={\"account\": str}","code":true},{"text":" to keep them."}],[{"text":"Dates read as strings.","b":true},{"text":" "},{"text":"2026-09-04","code":true},{"text":" is "},{"text":"object","code":true},{"text":" until you say "},{"text":"parse_dates=[\"created_at\"]","code":true},{"text":" or convert afterwards with "},{"text":"pd.to_datetime","code":true},{"text":"."}]]},{"t":"p","runs":[{"text":"Fix these at read time, not afterwards:"}]},{"t":"code","lang":"python","label":"Python","text":"df = pd.read_csv(\"orders.csv\", dtype={\"order_id\": str, \"postcode\": str},\n                 parse_dates=[\"created_at\"], na_values=[\"N/A\", \"-\", \"\"])"},{"t":"p","runs":[{"text":"df.info()","code":true},{"text":" prints dtypes, non-null counts and memory in one report; "},{"text":"df.describe()","code":true},{"text":" gives min, max, mean and quartiles for numeric columns, which catches a price column with a maximum of 999999."}]},{"t":"h2","id":"selecting","runs":[{"text":"Selecting"}]},{"t":"code","lang":"python","label":"Python","text":"df[\"price\"]                     # one column: a Series\ndf[[\"order_id\", \"price\"]]       # several columns: a DataFrame\ndf[df[\"price\"] > 100]           # rows by condition — a boolean mask, as in NumPy\ndf.loc[df[\"price\"] > 100, \"order_id\"]      # rows by condition, one column\ndf.iloc[0:5, 0:3]               # rows and columns by position\ndf.loc[42]                      # the row whose index label is 42"},{"t":"p","runs":[{"text":"loc","code":true},{"text":" selects by "},{"text":"label","b":true},{"text":": index values and column names. "},{"text":"iloc","code":true},{"text":" selects by "},{"text":"integer position","b":true},{"text":". They differ the moment the index is not 0..n−1, which happens after filtering or sorting: "},{"text":"df.iloc[0]","code":true},{"text":" is the first row now; "},{"text":"df.loc[0]","code":true},{"text":" is the row that was labelled 0 before, if it survived. Combining conditions uses "},{"text":"&","code":true},{"text":", "},{"text":"|","code":true},{"text":", "},{"text":"~","code":true},{"text":" with brackets, exactly as NumPy did."}]},{"t":"p","runs":[{"text":"df.query(\"price > 100 and status == 'paid'\")","code":true},{"text":" is the same filter as a string, easier to read for long conditions."}]},{"t":"h2","id":"adding-and-changing-columns","runs":[{"text":"Adding and changing columns"}]},{"t":"code","lang":"python","label":"Python","text":"df[\"total\"] = df[\"price\"] * df[\"quantity\"]              # vectorised\ndf[\"month\"] = df[\"created_at\"].dt.month                 # datetime accessor\ndf[\"email_domain\"] = df[\"email\"].str.split(\"@\").str[1]  # string accessor"},{"t":"p","runs":[{"text":"Whole-column operations. The "},{"text":".str","code":true},{"text":" and "},{"text":".dt","code":true},{"text":" accessors apply string and date methods across a column without a loop. If you find yourself writing "},{"text":"for i, row in df.iterrows()","code":true},{"text":", stop: "},{"text":"iterrows","code":true},{"text":" is the slowest way to touch a table, hundreds of times slower than a column expression, and the column expression almost always exists. "},{"text":"df.apply(func, axis=1)","code":true},{"text":" is a little better and still a loop; use it only for logic that cannot be written by column."}]},{"t":"h2","id":"the-copy-warning","runs":[{"text":"The copy warning"}]},{"t":"code","lang":"python","label":"Python","text":"paid = df[df[\"status\"] == \"paid\"]\npaid[\"flag\"] = True\n# SettingWithCopyWarning: A value is trying to be set on a copy of a slice..."},{"t":"p","runs":[{"text":"paid","code":true},{"text":" may be a view of "},{"text":"df","code":true},{"text":" or a copy, and pandas is telling you it cannot promise which, so your assignment may or may not reach "},{"text":"df","code":true},{"text":". The fix is to say what you mean: "},{"text":"paid = df[df[\"status\"] == \"paid\"].copy()","code":true},{"text":" if you want an independent table, or "},{"text":"df.loc[df[\"status\"] == \"paid\", \"flag\"] = True","code":true},{"text":" if you want to change "},{"text":"df","code":true},{"text":". Recent pandas versions with copy-on-write enabled make every such selection a copy and the warning disappears; the habit of "},{"text":".copy()","code":true},{"text":" or "},{"text":".loc","code":true},{"text":" assignment is right either way."}]},{"t":"h2","id":"getting-back-out","runs":[{"text":"Getting back out"}]},{"t":"code","lang":"python","label":"Python","text":"df.to_csv(\"clean.csv\", index=False)\ndf.to_parquet(\"clean.parquet\")            # needs pyarrow; smaller, typed, fast\ndf[\"price\"].to_numpy()                    # a NumPy array\ndf[[\"x\", \"y\"]].to_numpy(dtype=np.float32) # a (n, 2) matrix for a model"},{"t":"p","runs":[{"text":"index=False","code":true},{"text":" on "},{"text":"to_csv","code":true},{"text":" stops pandas writing the row numbers as a column, which otherwise comes back as "},{"text":"Unnamed: 0","code":true},{"text":" on the next read. Parquet preserves dtypes, so the ID-as-string and the parsed dates survive a round trip; CSV forgets them and you re-fix them on every load."}]},{"t":"h2","id":"try-this-now","runs":[{"text":"Try this now"}],"role":"try"},{"t":"p","runs":[{"text":"Load any CSV with at least one ID column and one date. Run the four commands. Find every "},{"text":"object","code":true},{"text":" column and decide whether it should be. Re-read with "},{"text":"dtype=","code":true},{"text":" and "},{"text":"parse_dates=","code":true},{"text":" until "},{"text":"dtypes","code":true},{"text":" says what you mean, then write it to Parquet and read it back."}]}],"check":{"question":[{"text":"After "},{"text":"df = pd.read_csv(\"orders.csv\")","code":true},{"text":", "},{"text":"df[\"order_id\"].head()","code":true},{"text":" shows values like "},{"text":"7001","code":true},{"text":", "},{"text":"7002","code":true},{"text":", and "},{"text":"df.dtypes","code":true},{"text":" reports "},{"text":"order_id int64","code":true},{"text":". Sorting by it later puts "},{"text":"10023","code":true},{"text":" after "},{"text":"7002","code":true},{"text":"... correctly, but a merge with another file where the same column reads as "},{"text":"object","code":true},{"text":" matches nothing. What most likely happened in the second file?"}],"options":[{"text":[{"text":"The second file has an ID like "},{"text":"7002A","code":true},{"text":" or "},{"text":"07002","code":true},{"text":", so pandas kept the column as strings, and "},{"text":"7002","code":true},{"text":" never equals "},{"text":"\"7002\"","code":true},{"text":"."}],"correct":true,"tempting":null},{"text":[{"text":"read_csv","code":true},{"text":" reads the second file with a different encoding, so the digits are different Unicode characters that look identical on screen but hash and compare differently."}],"correct":false,"tempting":[{"text":"Digit lookalikes are possible in principle but rare in practice; the ordinary cause is a single non-numeric value forcing the column to strings."}]},{"text":[{"text":"Merging requires both columns to be named the same and indexed, and the second file’s column is not the index."}],"correct":false,"tempting":[{"text":"Merge works on columns regardless of index; a type mismatch between int and str is what stops rows matching."}]},{"text":[{"text":"pandas reads "},{"text":"int64","code":true},{"text":" columns as 64-bit but "},{"text":"object","code":true},{"text":" columns as 32-bit, so the values overflow before comparison."}],"correct":false,"tempting":[{"text":"object","code":true},{"text":" columns hold Python objects of any size; there is no 32-bit truncation involved."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"vectors-and-cosine-similarity","title":"Dot products and cosine similarity: one vector against a hundred thousand in a single line","minutes":9},"next":{"slug":"cleaning-data-in-pandas","title":"Cleaning a table: missing values, the string that is not missing, and duplicates that are not identical","minutes":9}}}