{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"building-your-own-dataset","title":"Turning your files into a dataset","url":"https://addaly.com/learn/hugging-face/building-your-own-dataset","position":43,"of":75,"minutes":9,"answer":"Building a dataset is from_dict, from_generator or a folder convention plus explicitly typed Features, and publishing is one call; so the decisions that matter are the ones before the push: personal data.","keyIdea":"Building a dataset is `from_dict`, `from_generator` or a folder convention plus explicitly typed `Features`, and publishing is one call; so the decisions that matter are the ones before the push: personal data, the right to redistribute, a licence, and starting private because public cannot be undone."},"body":{"format":"markdown","text":"## From whatever you have\n\n```python\nfrom datasets import Dataset, DatasetDict\n\nds = Dataset.from_dict({\"text\": texts, \"label\": labels})\nds = Dataset.from_pandas(df)\nds = Dataset.from_list([{\"text\": \"...\", \"label\": 1}, ...])\n```\n\nFor anything too large to hold in memory, use a generator so rows stream in:\n\n```python\ndef gen():\n    for path in pathlib.Path(\"notes\").glob(\"*.txt\"):\n        yield {\"id\": path.stem, \"text\": path.read_text(encoding=\"utf-8\")}\n\nds = Dataset.from_generator(gen)\n```\n\nFor media, the folder conventions do the labelling:\n\n```python\nds = load_dataset(\"imagefolder\", data_dir=\"photos\")   # photos/cat/*.jpg -> label 'cat'\nds = load_dataset(\"audiofolder\", data_dir=\"clips\")\n```\n\nA `metadata.csv` in the folder, with a `file_name` column, attaches captions, transcripts or extra fields to each file.\n\n## Set the types deliberately\n\n```python\nfrom datasets import Features, Value, ClassLabel\n\nfeatures = Features({\n    \"text\": Value(\"string\"),\n    \"label\": ClassLabel(names=[\"neg\", \"neu\", \"pos\"]),\n    \"source\": Value(\"string\"),\n})\nds = Dataset.from_dict(data, features=features)\n```\n\n`ClassLabel` stores the names with the data, so nobody downstream has to guess what 2 means. Without it your labels are bare integers and the meaning lives in a message somebody sent you in March. This is the same class of mistake as a model with `LABEL_0` outputs, and it is equally cheap to prevent.\n\nInclude a `source` or `provenance` column from the start. Six months later, every argument about a surprising result begins with where a row came from.\n\n## Publishing\n\n```python\nds.push_to_hub(\"your-name/support-tickets\", private=True)\n```\n\nThat creates the repository if needed, writes parquet shards, and commits. Loading it back is `load_dataset(\"your-name/support-tickets\")`.\n\n**Start private.** Making a private dataset public later is one click; un-publishing something that has been indexed, cloned and mirrored is not possible. The asymmetry should decide the default.\n\nPush a `DatasetDict` and the splits are preserved:\n\n```python\nDatasetDict({\"train\": tr, \"validation\": va, \"test\": te}).push_to_hub(repo)\n```\n\nYou can also push a configuration alongside others, which is how one repository holds several variants:\n\n```python\nds_hi.push_to_hub(repo, config_name=\"hindi\")\nds_en.push_to_hub(repo, config_name=\"english\")\n```\n\n## Before you push anything\n\nThis list is short and every item on it has ruined somebody's week:\n\n1. **Personal data.** Names, emails, phone numbers, addresses, account numbers, and the sentence where somebody names their employer. Run an automatic redaction pass, then read a sample by hand, because automatic redaction misses context. If the data is about people and you cannot redact it, the dataset stays private.\n2. **Do you have the right to publish it?** Scraped content, customer data, anything under a contract, anything with a licence that forbids redistribution. Being able to download something is not permission to republish it.\n3. **Credentials in the rows.** API keys pasted into support tickets are extremely common. Grep for them.\n4. **A licence field.** Decide before publishing, not after. No licence is not the same as permissive.\n5. **The intended use and the known gaps**, in the card.\n\n## The upload that silently changes your data\n\nOne behaviour surprises people. `push_to_hub` writes the dataset as it currently exists, including every transformation you applied in the session. If you tokenised and then pushed, you have published token ids rather than text, and nobody else can use it with a different model.\n\nPublish the raw form. Keep preprocessing in a script in a repository beside it, so somebody with a different tokenizer can run your pipeline rather than inherit your choices. The rule generalises: **publish the thing, not your view of the thing.**\n\n## Keeping it updated\n\nA dataset repository is git, so a new push is a commit and old versions remain reachable:\n\n```python\nds = load_dataset(\"your-name/support-tickets\", revision=\"v1.0\")\n```\n\nTag releases when a number you published was computed against a specific version. A model card that says \"trained on our dataset\" without a revision is a claim nobody can check, including you.\n\n## Very large uploads\n\n`push_to_hub` handles gigabytes. For hundreds of gigabytes, shard before uploading, use `upload_large_folder`, and expect to resume: a four-hour upload will be interrupted. The chunk-level storage behind the Hub deduplicates, so re-pushing a dataset where one shard changed does not resend everything.\n\n## Do this now\n\nTake a folder of your own files (notes, tickets, transcripts, photos) and make it a private dataset with typed features and a source column. Then load it back with `load_dataset` and confirm the types survived. That round trip is the whole skill, and it takes ten minutes."},"check":{"question":"Why is `ClassLabel` preferable to storing labels as plain integers in a dataset you intend to share?","options":[{"text":"It compresses the column, since integer categories are stored as dictionary-encoded values rather than full integers.","correct":false,"tempting":"Encoding efficiency is a side effect and not the reason to reach for it."},{"text":"It enforces a closed set of values, so rows with an unexpected label are rejected at write time.","correct":false,"tempting":"Validation is a real benefit but not the one that survives contact with other people."},{"text":"The class names travel with the data, so nobody downstream has to guess what 2 meant.","correct":true},{"text":"It enables stratified splitting, which cannot be performed on an untyped integer column.","correct":false,"tempting":"Stratification benefits from it but can be done on any column with discrete values."}]},"prev":{"slug":"the-viewer-and-sql","title":"Query a dataset with SQL before downloading it","url":"https://addaly.com/learn/hugging-face/the-viewer-and-sql"},"next":{"slug":"dataset-cards-and-provenance","title":"The card is where a dataset stops being a file","url":"https://addaly.com/learn/hugging-face/dataset-cards-and-provenance"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Turning your files into a dataset', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/building-your-own-dataset (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Turning your files into a dataset","courseTitle":"Hugging Face, End to End","module":{"index":5,"slug":"datasets-and-the-work-before-the-model","title":"Datasets, and the work that happens before any model"},"answer":null,"keyIdea":"Building a dataset is `from_dict`, `from_generator` or a folder convention plus explicitly typed `Features`, and publishing is one call; so the decisions that matter are the ones before the push: personal data, the right to redistribute, a licence, and starting private because public cannot be undone.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"from-whatever-you-have","runs":[{"text":"From whatever you have"}]},{"t":"code","lang":"python","label":"Python","text":"from datasets import Dataset, DatasetDict\n\nds = Dataset.from_dict({\"text\": texts, \"label\": labels})\nds = Dataset.from_pandas(df)\nds = Dataset.from_list([{\"text\": \"...\", \"label\": 1}, ...])"},{"t":"p","runs":[{"text":"For anything too large to hold in memory, use a generator so rows stream in:"}]},{"t":"code","lang":"python","label":"Python","text":"def gen():\n    for path in pathlib.Path(\"notes\").glob(\"*.txt\"):\n        yield {\"id\": path.stem, \"text\": path.read_text(encoding=\"utf-8\")}\n\nds = Dataset.from_generator(gen)"},{"t":"p","runs":[{"text":"For media, the folder conventions do the labelling:"}]},{"t":"code","lang":"python","label":"Python","text":"ds = load_dataset(\"imagefolder\", data_dir=\"photos\")   # photos/cat/*.jpg -> label 'cat'\nds = load_dataset(\"audiofolder\", data_dir=\"clips\")"},{"t":"p","runs":[{"text":"A "},{"text":"metadata.csv","code":true},{"text":" in the folder, with a "},{"text":"file_name","code":true},{"text":" column, attaches captions, transcripts or extra fields to each file."}]},{"t":"h2","id":"set-the-types-deliberately","runs":[{"text":"Set the types deliberately"}]},{"t":"code","lang":"python","label":"Python","text":"from datasets import Features, Value, ClassLabel\n\nfeatures = Features({\n    \"text\": Value(\"string\"),\n    \"label\": ClassLabel(names=[\"neg\", \"neu\", \"pos\"]),\n    \"source\": Value(\"string\"),\n})\nds = Dataset.from_dict(data, features=features)"},{"t":"p","runs":[{"text":"ClassLabel","code":true},{"text":" stores the names with the data, so nobody downstream has to guess what 2 means. Without it your labels are bare integers and the meaning lives in a message somebody sent you in March. This is the same class of mistake as a model with "},{"text":"LABEL_0","code":true},{"text":" outputs, and it is equally cheap to prevent."}]},{"t":"p","runs":[{"text":"Include a "},{"text":"source","code":true},{"text":" or "},{"text":"provenance","code":true},{"text":" column from the start. Six months later, every argument about a surprising result begins with where a row came from."}]},{"t":"h2","id":"publishing","runs":[{"text":"Publishing"}]},{"t":"code","lang":"python","label":"Python","text":"ds.push_to_hub(\"your-name/support-tickets\", private=True)"},{"t":"p","runs":[{"text":"That creates the repository if needed, writes parquet shards, and commits. Loading it back is "},{"text":"load_dataset(\"your-name/support-tickets\")","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"Start private.","b":true},{"text":" Making a private dataset public later is one click; un-publishing something that has been indexed, cloned and mirrored is not possible. The asymmetry should decide the default."}]},{"t":"p","runs":[{"text":"Push a "},{"text":"DatasetDict","code":true},{"text":" and the splits are preserved:"}]},{"t":"code","lang":"python","label":"Python","text":"DatasetDict({\"train\": tr, \"validation\": va, \"test\": te}).push_to_hub(repo)"},{"t":"p","runs":[{"text":"You can also push a configuration alongside others, which is how one repository holds several variants:"}]},{"t":"code","lang":"python","label":"Python","text":"ds_hi.push_to_hub(repo, config_name=\"hindi\")\nds_en.push_to_hub(repo, config_name=\"english\")"},{"t":"h2","id":"before-you-push-anything","runs":[{"text":"Before you push anything"}]},{"t":"p","runs":[{"text":"This list is short and every item on it has ruined somebody’s week:"}]},{"t":"ol","start":1,"items":[[{"text":"Personal data.","b":true},{"text":" Names, emails, phone numbers, addresses, account numbers, and the sentence where somebody names their employer. Run an automatic redaction pass, then read a sample by hand, because automatic redaction misses context. If the data is about people and you cannot redact it, the dataset stays private."}],[{"text":"Do you have the right to publish it?","b":true},{"text":" Scraped content, customer data, anything under a contract, anything with a licence that forbids redistribution. Being able to download something is not permission to republish it."}],[{"text":"Credentials in the rows.","b":true},{"text":" API keys pasted into support tickets are extremely common. Grep for them."}],[{"text":"A licence field.","b":true},{"text":" Decide before publishing, not after. No licence is not the same as permissive."}],[{"text":"The intended use and the known gaps","b":true},{"text":", in the card."}]]},{"t":"h2","id":"the-upload-that-silently-changes-your-data","runs":[{"text":"The upload that silently changes your data"}]},{"t":"p","runs":[{"text":"One behaviour surprises people. "},{"text":"push_to_hub","code":true},{"text":" writes the dataset as it currently exists, including every transformation you applied in the session. If you tokenised and then pushed, you have published token ids rather than text, and nobody else can use it with a different model."}]},{"t":"p","runs":[{"text":"Publish the raw form. Keep preprocessing in a script in a repository beside it, so somebody with a different tokenizer can run your pipeline rather than inherit your choices. The rule generalises: "},{"text":"publish the thing, not your view of the thing.","b":true}]},{"t":"h2","id":"keeping-it-updated","runs":[{"text":"Keeping it updated"}]},{"t":"p","runs":[{"text":"A dataset repository is git, so a new push is a commit and old versions remain reachable:"}]},{"t":"code","lang":"python","label":"Python","text":"ds = load_dataset(\"your-name/support-tickets\", revision=\"v1.0\")"},{"t":"p","runs":[{"text":"Tag releases when a number you published was computed against a specific version. A model card that says “trained on our dataset” without a revision is a claim nobody can check, including you."}]},{"t":"h2","id":"very-large-uploads","runs":[{"text":"Very large uploads"}]},{"t":"p","runs":[{"text":"push_to_hub","code":true},{"text":" handles gigabytes. For hundreds of gigabytes, shard before uploading, use "},{"text":"upload_large_folder","code":true},{"text":", and expect to resume: a four-hour upload will be interrupted. The chunk-level storage behind the Hub deduplicates, so re-pushing a dataset where one shard changed does not resend everything."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Take a folder of your own files (notes, tickets, transcripts, photos) and make it a private dataset with typed features and a source column. Then load it back with "},{"text":"load_dataset","code":true},{"text":" and confirm the types survived. That round trip is the whole skill, and it takes ten minutes."}]}],"check":{"question":[{"text":"Why is "},{"text":"ClassLabel","code":true},{"text":" preferable to storing labels as plain integers in a dataset you intend to share?"}],"options":[{"text":[{"text":"It compresses the column, since integer categories are stored as dictionary-encoded values rather than full integers."}],"correct":false,"tempting":[{"text":"Encoding efficiency is a side effect and not the reason to reach for it."}]},{"text":[{"text":"It enforces a closed set of values, so rows with an unexpected label are rejected at write time."}],"correct":false,"tempting":[{"text":"Validation is a real benefit but not the one that survives contact with other people."}]},{"text":[{"text":"The class names travel with the data, so nobody downstream has to guess what 2 meant."}],"correct":true,"tempting":null},{"text":[{"text":"It enables stratified splitting, which cannot be performed on an untyped integer column."}],"correct":false,"tempting":[{"text":"Stratification benefits from it but can be done on any column with discrete values."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"the-viewer-and-sql","title":"Query a dataset with SQL before downloading it","minutes":8},"next":{"slug":"dataset-cards-and-provenance","title":"The card is where a dataset stops being a file","minutes":8}}}