{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"map-filter-and-batching","title":"Transforming a million rows without waiting an hour","url":"https://addaly.com/learn/hugging-face/map-filter-and-batching","position":41,"of":75,"minutes":9,"answer":"map(batched=True) hands your function a dict of lists and can return a different number of rows than it received, which makes it both the fast path and the tool for chunking.","keyIdea":"`map(batched=True)` hands your function a dict of lists and can return a different number of rows than it received, which makes it both the fast path and the tool for chunking, while `set_transform` applies on access for anything random that should not be cached."},"body":{"format":"markdown","text":"## The naive version, and the fast one\n\n```python\n# slow: one Python call per row\nds = ds.map(lambda r: {\"n_chars\": len(r[\"text\"])})\n\n# fast: one Python call per thousand rows\nds = ds.map(\n    lambda b: {\"n_chars\": [len(t) for t in b[\"text\"]]},\n    batched=True, batch_size=1000,\n)\n```\n\nIn batched mode the function receives a dict of **lists** and must return a dict of lists of the same length. Ten to fifty times faster is typical, and the reason is not clever: the per-call overhead of crossing between Arrow and Python dominates, and batching pays it a thousand times less often.\n\nFor tokenization the gain is larger still, because the Rust tokenizer parallelises across a batch internally:\n\n```python\ndef tok_fn(batch):\n    return tok(batch[\"text\"], truncation=True, max_length=512)\n\nds = ds.map(tok_fn, batched=True, remove_columns=[\"text\"])\n```\n\n`remove_columns` drops the original text once you have the ids. On a large corpus this is the difference between a 30 GB cached file and a 6 GB one.\n\n## Parallelism across cores\n\n```python\nds = ds.map(fn, batched=True, num_proc=8)\n```\n\nThis forks processes, each mapping a shard, and then concatenates. Two cautions that cost people real time:\n\n- **Fast tokenizers already use threads.** Combining `num_proc` with them can oversubscribe the CPU and run slower. The library will warn about `TOKENIZERS_PARALLELISM`; set it to `false` when using `num_proc`, or use one or the other.\n- **Anything holding a GPU does not fork well.** For GPU work, use one process and a large batch, not eight processes fighting over one card.\n\n## A batch does not have to stay the same size\n\nThis is the feature people miss. A batched map may return more or fewer rows than it received, which makes it the tool for splitting and dropping:\n\n```python\ndef chunk(batch, size=800, overlap=100):\n    out = {\"chunk\": [], \"doc_id\": []}\n    for doc_id, text in zip(batch[\"id\"], batch[\"text\"]):\n        for i in range(0, len(text), size - overlap):\n            out[\"chunk\"].append(text[i:i + size])\n            out[\"doc_id\"].append(doc_id)\n    return out\n\nchunks = docs.map(chunk, batched=True, remove_columns=docs.column_names)\n```\n\nOne thousand documents in, forty thousand chunks out, with the source id carried along. `remove_columns=docs.column_names` is required here, because the old columns no longer have the right length and the library will refuse the mismatch.\n\n## `filter`, and doing it in one pass\n\n```python\nds = ds.filter(lambda b: [len(t) > 50 for t in b[\"text\"]], batched=True, num_proc=4)\n```\n\nFiltering reads everything, so chain your conditions into one predicate rather than calling `filter` four times. Four filters are four full passes.\n\n## `with_transform`: when you do not want a new file\n\n`map` materialises a new Arrow file. When the transformation is cheap and varies per epoch (image augmentation, random cropping, dynamic masking) that is wasteful and wrong, because you want different randomness each time.\n\n```python\ndef augment(batch):\n    batch[\"pixel_values\"] = [aug(img) for img in batch[\"image\"]]\n    return batch\n\nds.set_transform(augment)      # applied on access, nothing written\n```\n\nRule: `map` for deterministic preprocessing you want cached, `set_transform` for anything random or cheap.\n\n## Making it reproducible\n\nEvery map that involves randomness needs a seed passed in, not read from global state, or your cache and your results diverge from each other. And keep the transformation code in a module rather than a notebook cell, because a notebook cell that was edited and re-run is not a record of what produced your data.\n\n## Sorting, grouping and joining\n\n`sort`, `class_encode_column` and `flatten_indices` cover most reshaping. There is no join; for that, convert to pandas or Polars, join, and convert back:\n\n```python\nimport pandas as pd\nfrom datasets import Dataset\nmerged = Dataset.from_pandas(ds.to_pandas().merge(other_df, on=\"id\"))\n```\n\nThat materialises everything in RAM, so it is fine for a few million rows and not for a corpus. For large joins, DuckDB over the parquet files is the next lesson and the right tool.\n\n## Do this now\n\nTake one `map` you have written per row and rewrite it batched. Time both on 50,000 rows. The number you get is the tax you have been paying on every dataset you have ever processed one row at a time."},"check":{"question":"A chunking function returns 40,000 chunks from 1,000 documents inside `map(batched=True)` and raises an error about mismatched lengths. What is missing?","options":[{"text":"`num_proc` must be set so the output shards are concatenated rather than aligned against the input.","correct":false,"tempting":"Parallelism is unrelated to column length alignment."},{"text":"The original columns must be removed; they still have the input's length.","correct":true},{"text":"The batch size must divide evenly into the output count, so that each batch produces a whole number of chunks.","correct":false,"tempting":"Invents a divisibility rule; batches may produce any number of rows."},{"text":"The function must return a list of dicts rather than a dict of lists when the row count changes.","correct":false,"tempting":"Inverts the batched contract, which is always a dict of lists regardless of length changes."}]},"prev":{"slug":"arrow-and-memory-mapping","title":"Why a 300 GB dataset opens instantly","url":"https://addaly.com/learn/hugging-face/arrow-and-memory-mapping"},"next":{"slug":"the-viewer-and-sql","title":"Query a dataset with SQL before downloading it","url":"https://addaly.com/learn/hugging-face/the-viewer-and-sql"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Transforming a million rows without waiting an hour', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/map-filter-and-batching (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Transforming a million rows without waiting an hour","courseTitle":"Hugging Face, End to End","module":{"index":5,"slug":"datasets-and-the-work-before-the-model","title":"Datasets, and the work that happens before any model"},"answer":null,"keyIdea":"`map(batched=True)` hands your function a dict of lists and can return a different number of rows than it received, which makes it both the fast path and the tool for chunking, while `set_transform` applies on access for anything random that should not be cached.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-naive-version-and-the-fast-one","runs":[{"text":"The naive version, and the fast one"}]},{"t":"code","lang":"python","label":"Python","text":"# slow: one Python call per row\nds = ds.map(lambda r: {\"n_chars\": len(r[\"text\"])})\n\n# fast: one Python call per thousand rows\nds = ds.map(\n    lambda b: {\"n_chars\": [len(t) for t in b[\"text\"]]},\n    batched=True, batch_size=1000,\n)"},{"t":"p","runs":[{"text":"In batched mode the function receives a dict of "},{"text":"lists","b":true},{"text":" and must return a dict of lists of the same length. Ten to fifty times faster is typical, and the reason is not clever: the per-call overhead of crossing between Arrow and Python dominates, and batching pays it a thousand times less often."}]},{"t":"p","runs":[{"text":"For tokenization the gain is larger still, because the Rust tokenizer parallelises across a batch internally:"}]},{"t":"code","lang":"python","label":"Python","text":"def tok_fn(batch):\n    return tok(batch[\"text\"], truncation=True, max_length=512)\n\nds = ds.map(tok_fn, batched=True, remove_columns=[\"text\"])"},{"t":"p","runs":[{"text":"remove_columns","code":true},{"text":" drops the original text once you have the ids. On a large corpus this is the difference between a 30 GB cached file and a 6 GB one."}]},{"t":"h2","id":"parallelism-across-cores","runs":[{"text":"Parallelism across cores"}]},{"t":"code","lang":"python","label":"Python","text":"ds = ds.map(fn, batched=True, num_proc=8)"},{"t":"p","runs":[{"text":"This forks processes, each mapping a shard, and then concatenates. Two cautions that cost people real time:"}]},{"t":"ul","items":[[{"text":"Fast tokenizers already use threads.","b":true},{"text":" Combining "},{"text":"num_proc","code":true},{"text":" with them can oversubscribe the CPU and run slower. The library will warn about "},{"text":"TOKENIZERS_PARALLELISM","code":true},{"text":"; set it to "},{"text":"false","code":true},{"text":" when using "},{"text":"num_proc","code":true},{"text":", or use one or the other."}],[{"text":"Anything holding a GPU does not fork well.","b":true},{"text":" For GPU work, use one process and a large batch, not eight processes fighting over one card."}]]},{"t":"h2","id":"a-batch-does-not-have-to-stay-the-same-size","runs":[{"text":"A batch does not have to stay the same size"}]},{"t":"p","runs":[{"text":"This is the feature people miss. A batched map may return more or fewer rows than it received, which makes it the tool for splitting and dropping:"}]},{"t":"code","lang":"python","label":"Python","text":"def chunk(batch, size=800, overlap=100):\n    out = {\"chunk\": [], \"doc_id\": []}\n    for doc_id, text in zip(batch[\"id\"], batch[\"text\"]):\n        for i in range(0, len(text), size - overlap):\n            out[\"chunk\"].append(text[i:i + size])\n            out[\"doc_id\"].append(doc_id)\n    return out\n\nchunks = docs.map(chunk, batched=True, remove_columns=docs.column_names)"},{"t":"p","runs":[{"text":"One thousand documents in, forty thousand chunks out, with the source id carried along. "},{"text":"remove_columns=docs.column_names","code":true},{"text":" is required here, because the old columns no longer have the right length and the library will refuse the mismatch."}]},{"t":"h2","id":"filter-and-doing-it-in-one-pass","runs":[{"text":"filter","code":true},{"text":", and doing it in one pass"}]},{"t":"code","lang":"python","label":"Python","text":"ds = ds.filter(lambda b: [len(t) > 50 for t in b[\"text\"]], batched=True, num_proc=4)"},{"t":"p","runs":[{"text":"Filtering reads everything, so chain your conditions into one predicate rather than calling "},{"text":"filter","code":true},{"text":" four times. Four filters are four full passes."}]},{"t":"h2","id":"withtransform-when-you-do-not-want-a-new-file","runs":[{"text":"with_transform","code":true},{"text":": when you do not want a new file"}]},{"t":"p","runs":[{"text":"map","code":true},{"text":" materialises a new Arrow file. When the transformation is cheap and varies per epoch (image augmentation, random cropping, dynamic masking) that is wasteful and wrong, because you want different randomness each time."}]},{"t":"code","lang":"python","label":"Python","text":"def augment(batch):\n    batch[\"pixel_values\"] = [aug(img) for img in batch[\"image\"]]\n    return batch\n\nds.set_transform(augment)      # applied on access, nothing written"},{"t":"p","runs":[{"text":"Rule: "},{"text":"map","code":true},{"text":" for deterministic preprocessing you want cached, "},{"text":"set_transform","code":true},{"text":" for anything random or cheap."}]},{"t":"h2","id":"making-it-reproducible","runs":[{"text":"Making it reproducible"}]},{"t":"p","runs":[{"text":"Every map that involves randomness needs a seed passed in, not read from global state, or your cache and your results diverge from each other. And keep the transformation code in a module rather than a notebook cell, because a notebook cell that was edited and re-run is not a record of what produced your data."}]},{"t":"h2","id":"sorting-grouping-and-joining","runs":[{"text":"Sorting, grouping and joining"}]},{"t":"p","runs":[{"text":"sort","code":true},{"text":", "},{"text":"class_encode_column","code":true},{"text":" and "},{"text":"flatten_indices","code":true},{"text":" cover most reshaping. There is no join; for that, convert to pandas or Polars, join, and convert back:"}]},{"t":"code","lang":"python","label":"Python","text":"import pandas as pd\nfrom datasets import Dataset\nmerged = Dataset.from_pandas(ds.to_pandas().merge(other_df, on=\"id\"))"},{"t":"p","runs":[{"text":"That materialises everything in RAM, so it is fine for a few million rows and not for a corpus. For large joins, DuckDB over the parquet files is the next lesson and the right tool."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Take one "},{"text":"map","code":true},{"text":" you have written per row and rewrite it batched. Time both on 50,000 rows. The number you get is the tax you have been paying on every dataset you have ever processed one row at a time."}]}],"check":{"question":[{"text":"A chunking function returns 40,000 chunks from 1,000 documents inside "},{"text":"map(batched=True)","code":true},{"text":" and raises an error about mismatched lengths. What is missing?"}],"options":[{"text":[{"text":"num_proc","code":true},{"text":" must be set so the output shards are concatenated rather than aligned against the input."}],"correct":false,"tempting":[{"text":"Parallelism is unrelated to column length alignment."}]},{"text":[{"text":"The original columns must be removed; they still have the input’s length."}],"correct":true,"tempting":null},{"text":[{"text":"The batch size must divide evenly into the output count, so that each batch produces a whole number of chunks."}],"correct":false,"tempting":[{"text":"Invents a divisibility rule; batches may produce any number of rows."}]},{"text":[{"text":"The function must return a list of dicts rather than a dict of lists when the row count changes."}],"correct":false,"tempting":[{"text":"Inverts the batched contract, which is always a dict of lists regardless of length changes."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"arrow-and-memory-mapping","title":"Why a 300 GB dataset opens instantly","minutes":8},"next":{"slug":"the-viewer-and-sql","title":"Query a dataset with SQL before downloading it","minutes":8}}}