{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"the-datasets-library","title":"A dataset that behaves like a list and is not one","url":"https://addaly.com/learn/hugging-face/the-datasets-library","position":39,"of":75,"minutes":9,"answer":"A Dataset is a typed table where indexing a row returns a dict and slicing returns a dict of lists, and operations like shuffle and select build an indices map rather than copying data.","keyIdea":"A `Dataset` is a typed table where indexing a row returns a dict and slicing returns a dict of lists, and operations like shuffle and select build an indices map rather than copying data, which is why reordering a dataset far larger than RAM is instantaneous."},"body":{"format":"markdown","text":"## Loading, and what comes back\n\n```python\nfrom datasets import load_dataset\n\nds = load_dataset(\"imdb\")\nprint(ds)\n```\n\n```\nDatasetDict({\n    train: Dataset({features: ['text', 'label'], num_rows: 25000}),\n    test:  Dataset({features: ['text', 'label'], num_rows: 25000}),\n    unsupervised: Dataset({features: ['text', 'label'], num_rows: 50000}),\n})\n```\n\nA `DatasetDict` is a dictionary of splits. Each `Dataset` has features (named, typed columns) and a row count. Ask for one split directly and you get the `Dataset` alone:\n\n```python\ntrain = load_dataset(\"imdb\", split=\"train\")\nprint(train[0])                    # one row, as a dict\nprint(train[:3][\"text\"][0][:80])   # a slice returns a dict of lists\nprint(train.features)\n```\n\nThat last line is worth internalising. **Indexing a row gives a dict; slicing gives a dict of lists.** `train[0]` is `{'text': ..., 'label': 1}`; `train[:3]` is `{'text': [...], 'label': [...]}`. Half the confusing errors in a first hour with this library come from expecting a list of dicts.\n\n## Features are typed, and the types do work\n\n```python\nprint(train.features[\"label\"])\n# ClassLabel(names=['neg', 'pos'], id=None)\n\nprint(train.features[\"label\"].int2str(1))   # 'pos'\n```\n\n`ClassLabel` carries the names, so you never have to guess whether 1 means positive: the same problem `id2label` solves on the model side. Other types you will meet are `Value(\"string\")`, `Value(\"int32\")`, `Sequence`, `Image`, and `Audio`, and the last two decode lazily, which the media lesson returns to.\n\n## Split slicing without writing Python\n\n```python\nsmall = load_dataset(\"imdb\", split=\"train[:1%]\")\nmiddle = load_dataset(\"imdb\", split=\"train[100:200]\")\nboth  = load_dataset(\"imdb\", split=\"train+test\")\n```\n\n`train[:1%]` is the habit worth forming. Develop on one per cent, then remove the suffix. A pipeline debugged on 250 rows runs unchanged on 25,000, and you will have saved yourself ten minutes per mistake.\n\n## Splitting when the dataset has no split\n\n```python\nparts = ds[\"train\"].train_test_split(test_size=0.1, seed=42, stratify_by_column=\"label\")\n```\n\nThree things here are deliberate. **`seed`** makes it reproducible, so the numbers you quote next week come from the same split. **`stratify_by_column`** keeps class proportions equal across the two halves, which matters enormously when a class is rare: an unstratified 10% test split of a dataset with 2% positives can easily land with almost none, and your recall figure then measures nothing.\n\nAnd a validation set is a third split, not a second use of the test set:\n\n```python\ntmp = ds[\"train\"].train_test_split(test_size=0.2, seed=42)\nval_test = tmp[\"test\"].train_test_split(test_size=0.5, seed=42)\n# tmp['train'] = 80% train, val_test['train'] = 10% val, val_test['test'] = 10% test\n```\n\nYou tune on validation. You touch test once, at the end. Every time you look at the test score and change something, it becomes a little more like a training set, and the number it reports becomes a little more optimistic than reality.\n\n## Loading your own files\n\nNo upload required: `load_dataset` takes formats directly:\n\n```python\nds = load_dataset(\"csv\", data_files=\"reviews.csv\")\nds = load_dataset(\"json\", data_files={\"train\": \"tr.jsonl\", \"test\": \"te.jsonl\"})\nds = load_dataset(\"parquet\", data_files=\"data/*.parquet\")\nds = load_dataset(\"imagefolder\", data_dir=\"photos/\")   # labels from folder names\n```\n\nSo the whole library is available for local work. You do not need to publish anything to get streaming, mapping, caching and typed columns.\n\n## Configurations, when one name holds many datasets\n\nSome repositories hold several variants, and the second positional argument chooses between them:\n\n```python\nfrom datasets import get_dataset_config_names\nprint(get_dataset_config_names(\"mozilla-foundation/common_voice_17_0\")[:5])\n\nhi = load_dataset(\"mozilla-foundation/common_voice_17_0\", \"hi\", split=\"train\")\n```\n\nFor multilingual corpora the configuration is usually the language, and loading without it either fails with a list of options or silently gives you the default, which, more often than is comfortable, is English. Print the configuration names before you assume.\n\n## Sorting, shuffling, selecting\n\n```python\nds = ds.shuffle(seed=42)\nsmall = ds.select(range(1000))\npos = ds.filter(lambda r: r[\"label\"] == 1)\nds = ds.remove_columns([\"unused\"]).rename_column(\"text\", \"sentence\")\n```\n\nNone of these copy the data. `select` and `shuffle` build an **indices map** (a list of row positions) and the underlying table stays where it is. This is why shuffling a 40 GB dataset is instant and takes no memory, and it is a genuine surprise the first time.\n\nThe exception is `filter`, which has to read every row to decide. On a large dataset, pass `num_proc` to parallelise it.\n\n## Do this now\n\nLoad any dataset with `split=\"train[:1%]\"`, print `features`, print row zero, and print a slice of three. Then run `train_test_split` with a seed and check that running it twice gives the same first row. That reproducibility is what makes every later comparison meaningful."},"check":{"question":"A team splits a dataset with 2% positive examples into 90/10 train and test without stratifying, and their reported recall swings wildly between runs with different seeds. Why?","options":[{"text":"The test split holds very few positives, so recall is computed over a handful of rows.","correct":true},{"text":"Unstratified splitting leaves some positives in both splits, so the model has memorised part of the test set.","correct":false,"tempting":"Describes leakage, which random splitting does not cause; each row lands in exactly one split."},{"text":"Recall is undefined for imbalanced data and should be replaced with accuracy, which is stable across seeds.","correct":false,"tempting":"Accuracy is stable here precisely because it is uninformative at 2% prevalence."},{"text":"The shuffle indices map is rebuilt per seed, so row order changes the order of training updates and therefore the model.","correct":false,"tempting":"Training order does introduce variance, but it would not produce swings of the size an almost-empty positive class does."}]},"prev":{"slug":"datasets-without-downloading","title":"Do not download the dataset","url":"https://addaly.com/learn/hugging-face/datasets-without-downloading"},"next":{"slug":"arrow-and-memory-mapping","title":"Why a 300 GB dataset opens instantly","url":"https://addaly.com/learn/hugging-face/arrow-and-memory-mapping"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A dataset that behaves like a list and is not one', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/the-datasets-library (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A dataset that behaves like a list and is not one","courseTitle":"Hugging Face, End to End","module":{"index":5,"slug":"datasets-and-the-work-before-the-model","title":"Datasets, and the work that happens before any model"},"answer":null,"keyIdea":"A `Dataset` is a typed table where indexing a row returns a dict and slicing returns a dict of lists, and operations like shuffle and select build an indices map rather than copying data, which is why reordering a dataset far larger than RAM is instantaneous.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"loading-and-what-comes-back","runs":[{"text":"Loading, and what comes back"}]},{"t":"code","lang":"python","label":"Python","text":"from datasets import load_dataset\n\nds = load_dataset(\"imdb\")\nprint(ds)"},{"t":"code","lang":"","label":"Text","text":"DatasetDict({\n    train: Dataset({features: ['text', 'label'], num_rows: 25000}),\n    test:  Dataset({features: ['text', 'label'], num_rows: 25000}),\n    unsupervised: Dataset({features: ['text', 'label'], num_rows: 50000}),\n})"},{"t":"p","runs":[{"text":"A "},{"text":"DatasetDict","code":true},{"text":" is a dictionary of splits. Each "},{"text":"Dataset","code":true},{"text":" has features (named, typed columns) and a row count. Ask for one split directly and you get the "},{"text":"Dataset","code":true},{"text":" alone:"}]},{"t":"code","lang":"python","label":"Python","text":"train = load_dataset(\"imdb\", split=\"train\")\nprint(train[0])                    # one row, as a dict\nprint(train[:3][\"text\"][0][:80])   # a slice returns a dict of lists\nprint(train.features)"},{"t":"p","runs":[{"text":"That last line is worth internalising. "},{"text":"Indexing a row gives a dict; slicing gives a dict of lists.","b":true},{"text":" "},{"text":"train[0]","code":true},{"text":" is "},{"text":"{'text': ..., 'label': 1}","code":true},{"text":"; "},{"text":"train[:3]","code":true},{"text":" is "},{"text":"{'text': [...], 'label': [...]}","code":true},{"text":". Half the confusing errors in a first hour with this library come from expecting a list of dicts."}]},{"t":"h2","id":"features-are-typed-and-the-types-do-work","runs":[{"text":"Features are typed, and the types do work"}]},{"t":"code","lang":"python","label":"Python","text":"print(train.features[\"label\"])\n# ClassLabel(names=['neg', 'pos'], id=None)\n\nprint(train.features[\"label\"].int2str(1))   # 'pos'"},{"t":"p","runs":[{"text":"ClassLabel","code":true},{"text":" carries the names, so you never have to guess whether 1 means positive: the same problem "},{"text":"id2label","code":true},{"text":" solves on the model side. Other types you will meet are "},{"text":"Value(\"string\")","code":true},{"text":", "},{"text":"Value(\"int32\")","code":true},{"text":", "},{"text":"Sequence","code":true},{"text":", "},{"text":"Image","code":true},{"text":", and "},{"text":"Audio","code":true},{"text":", and the last two decode lazily, which the media lesson returns to."}]},{"t":"h2","id":"split-slicing-without-writing-python","runs":[{"text":"Split slicing without writing Python"}]},{"t":"code","lang":"python","label":"Python","text":"small = load_dataset(\"imdb\", split=\"train[:1%]\")\nmiddle = load_dataset(\"imdb\", split=\"train[100:200]\")\nboth  = load_dataset(\"imdb\", split=\"train+test\")"},{"t":"p","runs":[{"text":"train[:1%]","code":true},{"text":" is the habit worth forming. Develop on one per cent, then remove the suffix. A pipeline debugged on 250 rows runs unchanged on 25,000, and you will have saved yourself ten minutes per mistake."}]},{"t":"h2","id":"splitting-when-the-dataset-has-no-split","runs":[{"text":"Splitting when the dataset has no split"}]},{"t":"code","lang":"python","label":"Python","text":"parts = ds[\"train\"].train_test_split(test_size=0.1, seed=42, stratify_by_column=\"label\")"},{"t":"p","runs":[{"text":"Three things here are deliberate. "},{"text":"seed","b":true,"code":true},{"text":" makes it reproducible, so the numbers you quote next week come from the same split. "},{"text":"stratify_by_column","b":true,"code":true},{"text":" keeps class proportions equal across the two halves, which matters enormously when a class is rare: an unstratified 10% test split of a dataset with 2% positives can easily land with almost none, and your recall figure then measures nothing."}]},{"t":"p","runs":[{"text":"And a validation set is a third split, not a second use of the test set:"}]},{"t":"code","lang":"python","label":"Python","text":"tmp = ds[\"train\"].train_test_split(test_size=0.2, seed=42)\nval_test = tmp[\"test\"].train_test_split(test_size=0.5, seed=42)\n# tmp['train'] = 80% train, val_test['train'] = 10% val, val_test['test'] = 10% test"},{"t":"p","runs":[{"text":"You tune on validation. You touch test once, at the end. Every time you look at the test score and change something, it becomes a little more like a training set, and the number it reports becomes a little more optimistic than reality."}]},{"t":"h2","id":"loading-your-own-files","runs":[{"text":"Loading your own files"}]},{"t":"p","runs":[{"text":"No upload required: "},{"text":"load_dataset","code":true},{"text":" takes formats directly:"}]},{"t":"code","lang":"python","label":"Python","text":"ds = load_dataset(\"csv\", data_files=\"reviews.csv\")\nds = load_dataset(\"json\", data_files={\"train\": \"tr.jsonl\", \"test\": \"te.jsonl\"})\nds = load_dataset(\"parquet\", data_files=\"data/*.parquet\")\nds = load_dataset(\"imagefolder\", data_dir=\"photos/\")   # labels from folder names"},{"t":"p","runs":[{"text":"So the whole library is available for local work. You do not need to publish anything to get streaming, mapping, caching and typed columns."}]},{"t":"h2","id":"configurations-when-one-name-holds-many-datasets","runs":[{"text":"Configurations, when one name holds many datasets"}]},{"t":"p","runs":[{"text":"Some repositories hold several variants, and the second positional argument chooses between them:"}]},{"t":"code","lang":"python","label":"Python","text":"from datasets import get_dataset_config_names\nprint(get_dataset_config_names(\"mozilla-foundation/common_voice_17_0\")[:5])\n\nhi = load_dataset(\"mozilla-foundation/common_voice_17_0\", \"hi\", split=\"train\")"},{"t":"p","runs":[{"text":"For multilingual corpora the configuration is usually the language, and loading without it either fails with a list of options or silently gives you the default, which, more often than is comfortable, is English. Print the configuration names before you assume."}]},{"t":"h2","id":"sorting-shuffling-selecting","runs":[{"text":"Sorting, shuffling, selecting"}]},{"t":"code","lang":"python","label":"Python","text":"ds = ds.shuffle(seed=42)\nsmall = ds.select(range(1000))\npos = ds.filter(lambda r: r[\"label\"] == 1)\nds = ds.remove_columns([\"unused\"]).rename_column(\"text\", \"sentence\")"},{"t":"p","runs":[{"text":"None of these copy the data. "},{"text":"select","code":true},{"text":" and "},{"text":"shuffle","code":true},{"text":" build an "},{"text":"indices map","b":true},{"text":" (a list of row positions) and the underlying table stays where it is. This is why shuffling a 40 GB dataset is instant and takes no memory, and it is a genuine surprise the first time."}]},{"t":"p","runs":[{"text":"The exception is "},{"text":"filter","code":true},{"text":", which has to read every row to decide. On a large dataset, pass "},{"text":"num_proc","code":true},{"text":" to parallelise it."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Load any dataset with "},{"text":"split=\"train[:1%]\"","code":true},{"text":", print "},{"text":"features","code":true},{"text":", print row zero, and print a slice of three. Then run "},{"text":"train_test_split","code":true},{"text":" with a seed and check that running it twice gives the same first row. That reproducibility is what makes every later comparison meaningful."}]}],"check":{"question":[{"text":"A team splits a dataset with 2% positive examples into 90/10 train and test without stratifying, and their reported recall swings wildly between runs with different seeds. Why?"}],"options":[{"text":[{"text":"The test split holds very few positives, so recall is computed over a handful of rows."}],"correct":true,"tempting":null},{"text":[{"text":"Unstratified splitting leaves some positives in both splits, so the model has memorised part of the test set."}],"correct":false,"tempting":[{"text":"Describes leakage, which random splitting does not cause; each row lands in exactly one split."}]},{"text":[{"text":"Recall is undefined for imbalanced data and should be replaced with accuracy, which is stable across seeds."}],"correct":false,"tempting":[{"text":"Accuracy is stable here precisely because it is uninformative at 2% prevalence."}]},{"text":[{"text":"The shuffle indices map is rebuilt per seed, so row order changes the order of training updates and therefore the model."}],"correct":false,"tempting":[{"text":"Training order does introduce variance, but it would not produce swings of the size an almost-empty positive class does."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"datasets-without-downloading","title":"Do not download the dataset","minutes":9},"next":{"slug":"arrow-and-memory-mapping","title":"Why a 300 GB dataset opens instantly","minutes":8}}}