Why a 300 GB dataset opens instantly

The thing that should not work

ds = load_dataset("wikipedia", "20220301.en", split="train")
print(len(ds))      # 6,458,670
print(ds[3_000_000]["title"])

That is roughly 20 GB of text, opened on a laptop with 8 GB of RAM, and the row access returns immediately. Nothing was loaded into memory. Understanding why changes how you work with data for good.

Arrow, and memory mapping

The library stores datasets as Apache Arrow files on disk: a columnar binary layout where every value’s position is computable rather than searched for. It then memory-maps the file: asks the operating system to make the file’s bytes appear at an address range, without reading them.

When you touch row three million, the OS faults in the few pages that hold it. Pages you never touch are never read. Pages you stop using are evicted by the OS under memory pressure, with no work from Python.

Three consequences, all practical:

1. RAM is not the limit. Disk is. A dataset larger than memory works normally.
2. Several processes share it. Four dataloader workers on one machine map the same file and the pages are shared, rather than each holding a copy.
3. Startup is instant regardless of size, because nothing is parsed at load time.

This is also why Dataset objects cannot simply be pickled and sent across a network: they are a handle on a file, not the data.

Columnar, which is why one column is cheap

Arrow stores all of text together and all of label together, rather than interleaving them row by row. Reading one column of a twelve-column dataset reads about a twelfth of the bytes. ds.remove_columns([...]) before a heavy operation is therefore not tidiness, it is a real speed-up, and on wide datasets it is a large one.

Parquet, which the Hub converts every dataset into, is the same idea with compression added, and it is the format the SQL lesson queries directly.

The cache, and the file that reappears

Every transformation writes a new Arrow file to ~/.cache/huggingface/datasets, keyed by a hash of the operation. This is why map is slow the first time and instant the second, and why your disk fills quietly.

print(ds.cache_files)
print(ds.dataset_size / 1e9, "GB")

Manage it deliberately:

ds = ds.map(fn, load_from_cache_file=False)     # force recompute
ds.cleanup_cache_files()                        # delete this dataset's caches

The hash covers the function’s bytecode, so editing your function invalidates the cache correctly. It does not cover a global variable your function reads. Change a threshold that lives outside the function and you will get stale cached results with no warning: a nasty bug, and the reason to pass parameters as arguments rather than closing over globals.

Streaming: when even disk is too much

ds = load_dataset("HuggingFaceFW/fineweb", split="train", streaming=True)
for row in ds.take(5):
    print(row["text"][:100])

With streaming=True you get an IterableDataset: rows are fetched over HTTP as you consume them and nothing lands on disk. This is how you work with a multi-terabyte corpus from a laptop or a free Colab session.

What you give up is random access. ds[42] does not exist, len(ds) usually does not, and shuffling becomes a buffer rather than a true permutation:

ds = ds.shuffle(seed=42, buffer_size=10_000)

That fills a 10,000-row buffer and samples from it. Good enough for training; not a real shuffle, which matters if the file order correlates with anything.

map and filter work on streams and are applied lazily as rows pass. The rule for choosing: stream when you will read the data roughly once, download when you will make several passes, because re-streaming for every epoch means re-downloading.

Interleaving sources

from datasets import interleave_datasets
mixed = interleave_datasets([hindi, english], probabilities=[0.3, 0.7], seed=42)

This samples from several streams at stated proportions: the standard way to mix languages or domains for pretraining without materialising a combined file.

Do this now

Load a dataset of several gigabytes without streaming and watch your memory usage while you index random rows. It will not move. Then check ds.cache_files and note where those bytes actually live, because that directory is the one that fills your disk.