{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"arrow-and-memory-mapping","title":"Why a 300 GB dataset opens instantly","url":"https://addaly.com/learn/hugging-face/arrow-and-memory-mapping","position":40,"of":75,"minutes":8,"answer":"Datasets are memory-mapped columnar Arrow files, so size is limited by disk rather than RAM and row access faults in only the pages it needs.","keyIdea":"Datasets are memory-mapped columnar Arrow files, so size is limited by disk rather than RAM and row access faults in only the pages it needs, while `streaming=True` drops random access and true shuffling in exchange for needing no disk at all."},"body":{"format":"markdown","text":"## The thing that should not work\n\n```python\nds = load_dataset(\"wikipedia\", \"20220301.en\", split=\"train\")\nprint(len(ds))      # 6,458,670\nprint(ds[3_000_000][\"title\"])\n```\n\nThat is roughly 20 GB of text, opened on a laptop with 8 GB of RAM, and the row access returns immediately. Nothing was loaded into memory. Understanding why changes how you work with data for good.\n\n## Arrow, and memory mapping\n\nThe library stores datasets as **Apache Arrow** files on disk: a columnar binary layout where every value's position is computable rather than searched for. It then **memory-maps** the file: asks the operating system to make the file's bytes appear at an address range, without reading them.\n\nWhen you touch row three million, the OS faults in the few pages that hold it. Pages you never touch are never read. Pages you stop using are evicted by the OS under memory pressure, with no work from Python.\n\nThree consequences, all practical:\n\n1. **RAM is not the limit.** Disk is. A dataset larger than memory works normally.\n2. **Several processes share it.** Four dataloader workers on one machine map the same file and the pages are shared, rather than each holding a copy.\n3. **Startup is instant** regardless of size, because nothing is parsed at load time.\n\nThis is also why `Dataset` objects cannot simply be pickled and sent across a network: they are a handle on a file, not the data.\n\n> Figure: Memory-mapped or streamed: what you trade. A two by two grid, rows Downloaded and streaming=True, columns Random access and Disk it needs: Downloaded and Random access: Instant; Downloaded and Disk it needs: The whole set; streaming=True and Random access: None; streaming=True and Disk it needs: Nothing. Neither is limited by RAM. The rule for choosing is how many passes you will make: stream when you will read the data roughly once, download when you will read it several times, because re-streaming every epoch means re-downloading every epoch.\n\n## Columnar, which is why one column is cheap\n\nArrow stores all of `text` together and all of `label` together, rather than interleaving them row by row. Reading one column of a twelve-column dataset reads about a twelfth of the bytes. `ds.remove_columns([...])` before a heavy operation is therefore not tidiness, it is a real speed-up, and on wide datasets it is a large one.\n\nParquet, which the Hub converts every dataset into, is the same idea with compression added, and it is the format the SQL lesson queries directly.\n\n## The cache, and the file that reappears\n\nEvery transformation writes a new Arrow file to `~/.cache/huggingface/datasets`, keyed by a hash of the operation. This is why `map` is slow the first time and instant the second, and why your disk fills quietly.\n\n```python\nprint(ds.cache_files)\nprint(ds.dataset_size / 1e9, \"GB\")\n```\n\nManage it deliberately:\n\n```python\nds = ds.map(fn, load_from_cache_file=False)     # force recompute\nds.cleanup_cache_files()                        # delete this dataset's caches\n```\n\nThe hash covers the function's bytecode, so editing your function invalidates the cache correctly. It does **not** cover a global variable your function reads. Change a threshold that lives outside the function and you will get stale cached results with no warning: a nasty bug, and the reason to pass parameters as arguments rather than closing over globals.\n\n## Streaming: when even disk is too much\n\n```python\nds = load_dataset(\"HuggingFaceFW/fineweb\", split=\"train\", streaming=True)\nfor row in ds.take(5):\n    print(row[\"text\"][:100])\n```\n\nWith `streaming=True` you get an `IterableDataset`: rows are fetched over HTTP as you consume them and nothing lands on disk. This is how you work with a multi-terabyte corpus from a laptop or a free Colab session.\n\nWhat you give up is random access. `ds[42]` does not exist, `len(ds)` usually does not, and shuffling becomes a buffer rather than a true permutation:\n\n```python\nds = ds.shuffle(seed=42, buffer_size=10_000)\n```\n\nThat fills a 10,000-row buffer and samples from it. Good enough for training; not a real shuffle, which matters if the file order correlates with anything.\n\n`map` and `filter` work on streams and are applied lazily as rows pass. The rule for choosing: **stream when you will read the data roughly once**, download when you will make several passes, because re-streaming for every epoch means re-downloading.\n\n## Interleaving sources\n\n```python\nfrom datasets import interleave_datasets\nmixed = interleave_datasets([hindi, english], probabilities=[0.3, 0.7], seed=42)\n```\n\nThis samples from several streams at stated proportions: the standard way to mix languages or domains for pretraining without materialising a combined file.\n\n## Do this now\n\nLoad a dataset of several gigabytes without streaming and watch your memory usage while you index random rows. It will not move. Then check `ds.cache_files` and note where those bytes actually live, because that directory is the one that fills your disk."},"check":{"question":"A `map` function reads a threshold from a module-level variable. The threshold is changed and the pipeline re-run, but the output is unchanged. What happened?","options":[{"text":"Memory mapping served the previous transformation's pages from the OS cache rather than recomputing them.","correct":false,"tempting":"Confuses page caching, which serves bytes from an existing file, with the fingerprint that decides whether to recompute."},{"text":"Arrow files are immutable, so a second `map` over the same dataset is silently ignored until the cache is cleared.","correct":false,"tempting":"Immutability is real but each map writes a new file rather than being ignored."},{"text":"The cache fingerprint hashes the function itself, which did not change, so the stale result was reused.","correct":true},{"text":"Streaming datasets apply transformations lazily, so the change takes effect only once the iterator is exhausted and restarted.","correct":false,"tempting":"Describes streaming behaviour in a situation where the caching path implies a downloaded dataset."}]},"prev":{"slug":"the-datasets-library","title":"A dataset that behaves like a list and is not one","url":"https://addaly.com/learn/hugging-face/the-datasets-library"},"next":{"slug":"map-filter-and-batching","title":"Transforming a million rows without waiting an hour","url":"https://addaly.com/learn/hugging-face/map-filter-and-batching"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why a 300 GB dataset opens instantly', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/arrow-and-memory-mapping (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why a 300 GB dataset opens instantly","courseTitle":"Hugging Face, End to End","module":{"index":5,"slug":"datasets-and-the-work-before-the-model","title":"Datasets, and the work that happens before any model"},"answer":null,"keyIdea":"Datasets are memory-mapped columnar Arrow files, so size is limited by disk rather than RAM and row access faults in only the pages it needs, while `streaming=True` drops random access and true shuffling in exchange for needing no disk at all.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Random access","Disk it needs"],"rows":["Downloaded","streaming=True"],"type":"matrix","cells":[["Instant","The whole set"],["None","Nothing"]],"title":"Memory-mapped or streamed: what you trade","caption":"Neither is limited by RAM. The rule for choosing is how many passes you will make: stream when you will read the data roughly once, download when you will read it several times, because re-streaming every epoch means re-downloading every epoch.","cellLabels":[["Row three million works","Limited by disk, not RAM"],["Iterate, and no len()","Rows arrive over HTTP"]]},"alt":"Figure: Memory-mapped or streamed: what you trade. A two by two grid, rows Downloaded and streaming=True, columns Random access and Disk it needs: Downloaded and Random access: Instant; Downloaded and Disk it needs: The whole set; streaming=True and Random access: None; streaming=True and Disk it needs: Nothing. Neither is limited by RAM. The rule for choosing is how many passes you will make: stream when you will read the data roughly once, download when you will read it several times, because re-streaming every epoch means re-downloading every epoch."}},"blocks":[{"t":"h2","id":"the-thing-that-should-not-work","runs":[{"text":"The thing that should not work"}]},{"t":"code","lang":"python","label":"Python","text":"ds = load_dataset(\"wikipedia\", \"20220301.en\", split=\"train\")\nprint(len(ds))      # 6,458,670\nprint(ds[3_000_000][\"title\"])"},{"t":"p","runs":[{"text":"That is roughly 20 GB of text, opened on a laptop with 8 GB of RAM, and the row access returns immediately. Nothing was loaded into memory. Understanding why changes how you work with data for good."}]},{"t":"h2","id":"arrow-and-memory-mapping","runs":[{"text":"Arrow, and memory mapping"}]},{"t":"p","runs":[{"text":"The library stores datasets as "},{"text":"Apache Arrow","b":true},{"text":" files on disk: a columnar binary layout where every value’s position is computable rather than searched for. It then "},{"text":"memory-maps","b":true},{"text":" the file: asks the operating system to make the file’s bytes appear at an address range, without reading them."}]},{"t":"p","runs":[{"text":"When you touch row three million, the OS faults in the few pages that hold it. Pages you never touch are never read. Pages you stop using are evicted by the OS under memory pressure, with no work from Python."}]},{"t":"p","runs":[{"text":"Three consequences, all practical:"}]},{"t":"ol","start":1,"items":[[{"text":"RAM is not the limit.","b":true},{"text":" Disk is. A dataset larger than memory works normally."}],[{"text":"Several processes share it.","b":true},{"text":" Four dataloader workers on one machine map the same file and the pages are shared, rather than each holding a copy."}],[{"text":"Startup is instant","b":true},{"text":" regardless of size, because nothing is parsed at load time."}]]},{"t":"p","runs":[{"text":"This is also why "},{"text":"Dataset","code":true},{"text":" objects cannot simply be pickled and sent across a network: they are a handle on a file, not the data."}]},{"t":"h2","id":"columnar-which-is-why-one-column-is-cheap","runs":[{"text":"Columnar, which is why one column is cheap"}]},{"t":"p","runs":[{"text":"Arrow stores all of "},{"text":"text","code":true},{"text":" together and all of "},{"text":"label","code":true},{"text":" together, rather than interleaving them row by row. Reading one column of a twelve-column dataset reads about a twelfth of the bytes. "},{"text":"ds.remove_columns([...])","code":true},{"text":" before a heavy operation is therefore not tidiness, it is a real speed-up, and on wide datasets it is a large one."}]},{"t":"p","runs":[{"text":"Parquet, which the Hub converts every dataset into, is the same idea with compression added, and it is the format the SQL lesson queries directly."}]},{"t":"h2","id":"the-cache-and-the-file-that-reappears","runs":[{"text":"The cache, and the file that reappears"}]},{"t":"p","runs":[{"text":"Every transformation writes a new Arrow file to "},{"text":"~/.cache/huggingface/datasets","code":true},{"text":", keyed by a hash of the operation. This is why "},{"text":"map","code":true},{"text":" is slow the first time and instant the second, and why your disk fills quietly."}]},{"t":"code","lang":"python","label":"Python","text":"print(ds.cache_files)\nprint(ds.dataset_size / 1e9, \"GB\")"},{"t":"p","runs":[{"text":"Manage it deliberately:"}]},{"t":"code","lang":"python","label":"Python","text":"ds = ds.map(fn, load_from_cache_file=False)     # force recompute\nds.cleanup_cache_files()                        # delete this dataset's caches"},{"t":"p","runs":[{"text":"The hash covers the function’s bytecode, so editing your function invalidates the cache correctly. It does "},{"text":"not","b":true},{"text":" cover a global variable your function reads. Change a threshold that lives outside the function and you will get stale cached results with no warning: a nasty bug, and the reason to pass parameters as arguments rather than closing over globals."}]},{"t":"h2","id":"streaming-when-even-disk-is-too-much","runs":[{"text":"Streaming: when even disk is too much"}]},{"t":"code","lang":"python","label":"Python","text":"ds = load_dataset(\"HuggingFaceFW/fineweb\", split=\"train\", streaming=True)\nfor row in ds.take(5):\n    print(row[\"text\"][:100])"},{"t":"p","runs":[{"text":"With "},{"text":"streaming=True","code":true},{"text":" you get an "},{"text":"IterableDataset","code":true},{"text":": rows are fetched over HTTP as you consume them and nothing lands on disk. This is how you work with a multi-terabyte corpus from a laptop or a free Colab session."}]},{"t":"p","runs":[{"text":"What you give up is random access. "},{"text":"ds[42]","code":true},{"text":" does not exist, "},{"text":"len(ds)","code":true},{"text":" usually does not, and shuffling becomes a buffer rather than a true permutation:"}]},{"t":"code","lang":"python","label":"Python","text":"ds = ds.shuffle(seed=42, buffer_size=10_000)"},{"t":"p","runs":[{"text":"That fills a 10,000-row buffer and samples from it. Good enough for training; not a real shuffle, which matters if the file order correlates with anything."}]},{"t":"p","runs":[{"text":"map","code":true},{"text":" and "},{"text":"filter","code":true},{"text":" work on streams and are applied lazily as rows pass. The rule for choosing: "},{"text":"stream when you will read the data roughly once","b":true},{"text":", download when you will make several passes, because re-streaming for every epoch means re-downloading."}]},{"t":"h2","id":"interleaving-sources","runs":[{"text":"Interleaving sources"}]},{"t":"code","lang":"python","label":"Python","text":"from datasets import interleave_datasets\nmixed = interleave_datasets([hindi, english], probabilities=[0.3, 0.7], seed=42)"},{"t":"p","runs":[{"text":"This samples from several streams at stated proportions: the standard way to mix languages or domains for pretraining without materialising a combined file."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Load a dataset of several gigabytes without streaming and watch your memory usage while you index random rows. It will not move. Then check "},{"text":"ds.cache_files","code":true},{"text":" and note where those bytes actually live, because that directory is the one that fills your disk."}]}],"check":{"question":[{"text":"A "},{"text":"map","code":true},{"text":" function reads a threshold from a module-level variable. The threshold is changed and the pipeline re-run, but the output is unchanged. What happened?"}],"options":[{"text":[{"text":"Memory mapping served the previous transformation’s pages from the OS cache rather than recomputing them."}],"correct":false,"tempting":[{"text":"Confuses page caching, which serves bytes from an existing file, with the fingerprint that decides whether to recompute."}]},{"text":[{"text":"Arrow files are immutable, so a second "},{"text":"map","code":true},{"text":" over the same dataset is silently ignored until the cache is cleared."}],"correct":false,"tempting":[{"text":"Immutability is real but each map writes a new file rather than being ignored."}]},{"text":[{"text":"The cache fingerprint hashes the function itself, which did not change, so the stale result was reused."}],"correct":true,"tempting":null},{"text":[{"text":"Streaming datasets apply transformations lazily, so the change takes effect only once the iterator is exhausted and restarted."}],"correct":false,"tempting":[{"text":"Describes streaming behaviour in a situation where the caching path implies a downloaded dataset."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"the-datasets-library","title":"A dataset that behaves like a list and is not one","minutes":9},"next":{"slug":"map-filter-and-batching","title":"Transforming a million rows without waiting an hour","minutes":9}}}