{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"datasets-without-downloading","title":"Do not download the dataset","url":"https://addaly.com/learn/hugging-face/datasets-without-downloading","position":38,"of":75,"minutes":9,"answer":"Inspect a dataset in the viewer and stream it with streaming=True rather than downloading, and treat licence obligations as surviving fine-tuning until a lawyer in your country tells you otherwise.","keyIdea":"Inspect a dataset in the viewer and stream it with `streaming=True` rather than downloading, and treat licence obligations as surviving fine-tuning until a lawyer in your country tells you otherwise."},"body":{"format":"markdown","text":"## The reflex that costs you an afternoon\n\nSomeone finds a dataset that looks right, calls `load_dataset`, and watches a progress bar for two hours to discover that the column they needed is empty for the language they care about. Everything in this lesson exists to stop that.\n\n## Look before you fetch\n\nEvery public dataset on the Hub gets an automatic viewer: a paged table with the real rows, the column names and the inferred types, plus per-column statistics. Public datasets are also converted automatically to Parquet, a columnar format that can be read a column and a chunk at a time over the network.\n\nThose two facts together mean you can answer most of your questions in a browser, on a phone, before downloading a byte. How many rows. What the label distribution looks like. Whether the text is actually the transcript or a URL pointing at one. Whether your language is present in any quantity or is three hundred rows out of eleven million.\n\nWhen the viewer is not enough, query the Parquet directly. DuckDB reads Parquet over HTTP and fetches only the ranges it needs:\n\n```sql\nINSTALL httpfs;\nLOAD httpfs;\n\nSELECT language, count(*)\nFROM 'hf://datasets/<owner>/<name>/**/*.parquet'\nGROUP BY language\nORDER BY 2 DESC;\n```\n\nSupport for the `hf://` prefix depends on your DuckDB version; if it complains, the plain `https://huggingface.co/...` URL of a single Parquet file works too. A grouped count over a large corpus pulls a few megabytes of one column, not the corpus. If SQL is unfamiliar, [/learn/data-and-sql](/learn/data-and-sql) is the shorter road than learning it from a dataset this size.\n\n## Loading, and streaming\n\nWhen you do want rows in Python:\n\n```bash\npip install -U datasets\n```\n\n```python\nfrom datasets import load_dataset\n\nds = load_dataset(\"stanfordnlp/imdb\", split=\"train\")\nprint(ds[0][\"text\"][:200])\n```\n\nThat downloads and caches everything, which is right for 80 MB and wrong for 800 GB. For anything large, add one argument:\n\n```python\nds = load_dataset(\"HuggingFaceFW/fineweb\", split=\"train\", streaming=True)\n\nfor row in ds.take(5):\n    print(row[\"text\"][:200])\n```\n\n`streaming=True` returns an iterable dataset instead of a materialised one. It fetches Parquet row groups over HTTP as you iterate and never writes a complete copy to disk. You lose random indexing and `len()`. You keep `.take()`, `.skip()`, `.filter()`, `.map()` and `.shuffle()` with a buffer. This is the difference between exploring a web-scale corpus on a laptop with 30 GB free and not exploring it.\n\nOne compatibility note: recent versions of `datasets` removed the old mechanism where a dataset repository shipped a Python loading script. Since datasets 4.0 (July 2025), passing `trust_remote_code=True` to `load_dataset`, as older tutorials do, does nothing except log an error, and any dataset that still depends on a loading script stops with \"Dataset scripts are no longer supported\". Almost all significant datasets now ship as plain data files, which is both safer and faster.\n\n## The licences that survive fine-tuning\n\nHere is where people get badly caught, and where the honest answer is uncomfortable.\n\nA dataset carries a licence just as a model does, and the obligations do not obviously evaporate when the data passes through gradient descent. Three that recur:\n\n- **Non-commercial** data does not become commercial because you trained on it. Whatever your view of the underlying law, no dataset card has ever said *unless you fine-tune, in which case never mind*.\n- **Share-alike** terms are designed to propagate to derivatives, and whether a set of weights is a derivative of its training data is exactly the question nobody has settled.\n- **Attribution** obligations under CC-BY do not disappear because you no longer ship the rows.\n\nThis area is unsettled and it differs by country. Courts in different jurisdictions are actively disagreeing about whether training is reproduction, whether it falls under text-and-data-mining exceptions, and what a model's weights are in legal terms. That is the shape of it; a lawyer in your country is the person who answers it for your case. What you can do without any lawyer is keep a record of which datasets went into which training run, because the question you cannot answer later is *what was in it*. [/learn/fine-tuning](/learn/fine-tuning) covers the training side.\n\n## The other half of the card\n\nRead the dataset card's sections on collection and on personal information. Was this scraped or contributed. Did the people in it know. Is there a takedown or opt-out route. Face datasets and voice datasets in particular have been assembled from material whose subjects never agreed, and several well-known ones have been withdrawn for that reason after other people had already built on them.\n\nIf you are assembling your own dataset from a community, the consent question is not a formality you satisfy afterwards. It is the design.\n\n## Do this now\n\nOpen the viewer of a dataset in your own field and check one thing you assumed: the number of rows in your language, or whether the label you need is populated. It takes a minute and saves the two hours."},"check":{"question":"A researcher wants to count how many rows of a 400 GB text corpus are in Marathi, on a laptop with 40 GB free. What is the sound approach?","options":[{"text":"Download a random shard of the dataset and extrapolate the proportion from it, since a sample is representative of the whole.","correct":false,"tempting":"Trades an exact answer for an estimate when the exact answer was cheaply available, and assumes shards are randomly composed."},{"text":"Query the automatically converted Parquet files over HTTP with DuckDB, or iterate with `streaming=True`, so only the language column is transferred.","correct":true},{"text":"Call `load_dataset` with a filter on the language column, because filtering happens before anything is written to disk.","correct":false,"tempting":"Believes `.filter()` on a normal dataset is applied during download rather than after the full copy is cached."},{"text":"Ask for the count in the dataset's Community tab, since only the maintainers can compute statistics over a corpus that size.","correct":false,"tempting":"Assumes large-scale statistics require the owner's infrastructure, when the columnar format makes them cheap for anyone."}]},"prev":{"slug":"debugging-a-space","title":"It built, it started, it is red","url":"https://addaly.com/learn/hugging-face/debugging-a-space"},"next":{"slug":"the-datasets-library","title":"A dataset that behaves like a list and is not one","url":"https://addaly.com/learn/hugging-face/the-datasets-library"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Do not download the dataset', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/datasets-without-downloading (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Do not download the dataset","courseTitle":"Hugging Face, End to End","module":{"index":5,"slug":"datasets-and-the-work-before-the-model","title":"Datasets, and the work that happens before any model"},"answer":null,"keyIdea":"Inspect a dataset in the viewer and stream it with `streaming=True` rather than downloading, and treat licence obligations as surviving fine-tuning until a lawyer in your country tells you otherwise.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-reflex-that-costs-you-an-afternoon","runs":[{"text":"The reflex that costs you an afternoon"}]},{"t":"p","runs":[{"text":"Someone finds a dataset that looks right, calls "},{"text":"load_dataset","code":true},{"text":", and watches a progress bar for two hours to discover that the column they needed is empty for the language they care about. Everything in this lesson exists to stop that."}]},{"t":"h2","id":"look-before-you-fetch","runs":[{"text":"Look before you fetch"}]},{"t":"p","runs":[{"text":"Every public dataset on the Hub gets an automatic viewer: a paged table with the real rows, the column names and the inferred types, plus per-column statistics. Public datasets are also converted automatically to Parquet, a columnar format that can be read a column and a chunk at a time over the network."}]},{"t":"p","runs":[{"text":"Those two facts together mean you can answer most of your questions in a browser, on a phone, before downloading a byte. How many rows. What the label distribution looks like. Whether the text is actually the transcript or a URL pointing at one. Whether your language is present in any quantity or is three hundred rows out of eleven million."}]},{"t":"p","runs":[{"text":"When the viewer is not enough, query the Parquet directly. DuckDB reads Parquet over HTTP and fetches only the ranges it needs:"}]},{"t":"code","lang":"sql","label":"SQL","text":"INSTALL httpfs;\nLOAD httpfs;\n\nSELECT language, count(*)\nFROM 'hf://datasets/<owner>/<name>/**/*.parquet'\nGROUP BY language\nORDER BY 2 DESC;"},{"t":"p","runs":[{"text":"Support for the "},{"text":"hf://","code":true},{"text":" prefix depends on your DuckDB version; if it complains, the plain "},{"text":"https://huggingface.co/...","code":true},{"text":" URL of a single Parquet file works too. A grouped count over a large corpus pulls a few megabytes of one column, not the corpus. If SQL is unfamiliar, "},{"text":"/learn/data-and-sql","href":"https://addaly.com/learn/data-and-sql"},{"text":" is the shorter road than learning it from a dataset this size."}]},{"t":"h2","id":"loading-and-streaming","runs":[{"text":"Loading, and streaming"}]},{"t":"p","runs":[{"text":"When you do want rows in Python:"}]},{"t":"code","lang":"bash","label":"Bash","text":"pip install -U datasets"},{"t":"code","lang":"python","label":"Python","text":"from datasets import load_dataset\n\nds = load_dataset(\"stanfordnlp/imdb\", split=\"train\")\nprint(ds[0][\"text\"][:200])"},{"t":"p","runs":[{"text":"That downloads and caches everything, which is right for 80 MB and wrong for 800 GB. For anything large, add one argument:"}]},{"t":"code","lang":"python","label":"Python","text":"ds = load_dataset(\"HuggingFaceFW/fineweb\", split=\"train\", streaming=True)\n\nfor row in ds.take(5):\n    print(row[\"text\"][:200])"},{"t":"p","runs":[{"text":"streaming=True","code":true},{"text":" returns an iterable dataset instead of a materialised one. It fetches Parquet row groups over HTTP as you iterate and never writes a complete copy to disk. You lose random indexing and "},{"text":"len()","code":true},{"text":". You keep "},{"text":".take()","code":true},{"text":", "},{"text":".skip()","code":true},{"text":", "},{"text":".filter()","code":true},{"text":", "},{"text":".map()","code":true},{"text":" and "},{"text":".shuffle()","code":true},{"text":" with a buffer. This is the difference between exploring a web-scale corpus on a laptop with 30 GB free and not exploring it."}]},{"t":"p","runs":[{"text":"One compatibility note: recent versions of "},{"text":"datasets","code":true},{"text":" removed the old mechanism where a dataset repository shipped a Python loading script. Since datasets 4.0 (July 2025), passing "},{"text":"trust_remote_code=True","code":true},{"text":" to "},{"text":"load_dataset","code":true},{"text":", as older tutorials do, does nothing except log an error, and any dataset that still depends on a loading script stops with “Dataset scripts are no longer supported”. Almost all significant datasets now ship as plain data files, which is both safer and faster."}]},{"t":"h2","id":"the-licences-that-survive-fine-tuning","runs":[{"text":"The licences that survive fine-tuning"}]},{"t":"p","runs":[{"text":"Here is where people get badly caught, and where the honest answer is uncomfortable."}]},{"t":"p","runs":[{"text":"A dataset carries a licence just as a model does, and the obligations do not obviously evaporate when the data passes through gradient descent. Three that recur:"}]},{"t":"ul","items":[[{"text":"Non-commercial","b":true},{"text":" data does not become commercial because you trained on it. Whatever your view of the underlying law, no dataset card has ever said "},{"text":"unless you fine-tune, in which case never mind","i":true},{"text":"."}],[{"text":"Share-alike","b":true},{"text":" terms are designed to propagate to derivatives, and whether a set of weights is a derivative of its training data is exactly the question nobody has settled."}],[{"text":"Attribution","b":true},{"text":" obligations under CC-BY do not disappear because you no longer ship the rows."}]]},{"t":"p","runs":[{"text":"This area is unsettled and it differs by country. Courts in different jurisdictions are actively disagreeing about whether training is reproduction, whether it falls under text-and-data-mining exceptions, and what a model’s weights are in legal terms. That is the shape of it; a lawyer in your country is the person who answers it for your case. What you can do without any lawyer is keep a record of which datasets went into which training run, because the question you cannot answer later is "},{"text":"what was in it","i":true},{"text":". "},{"text":"/learn/fine-tuning","href":"https://addaly.com/learn/fine-tuning"},{"text":" covers the training side."}]},{"t":"h2","id":"the-other-half-of-the-card","runs":[{"text":"The other half of the card"}]},{"t":"p","runs":[{"text":"Read the dataset card’s sections on collection and on personal information. Was this scraped or contributed. Did the people in it know. Is there a takedown or opt-out route. Face datasets and voice datasets in particular have been assembled from material whose subjects never agreed, and several well-known ones have been withdrawn for that reason after other people had already built on them."}]},{"t":"p","runs":[{"text":"If you are assembling your own dataset from a community, the consent question is not a formality you satisfy afterwards. It is the design."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Open the viewer of a dataset in your own field and check one thing you assumed: the number of rows in your language, or whether the label you need is populated. It takes a minute and saves the two hours."}]}],"check":{"question":[{"text":"A researcher wants to count how many rows of a 400 GB text corpus are in Marathi, on a laptop with 40 GB free. What is the sound approach?"}],"options":[{"text":[{"text":"Download a random shard of the dataset and extrapolate the proportion from it, since a sample is representative of the whole."}],"correct":false,"tempting":[{"text":"Trades an exact answer for an estimate when the exact answer was cheaply available, and assumes shards are randomly composed."}]},{"text":[{"text":"Query the automatically converted Parquet files over HTTP with DuckDB, or iterate with "},{"text":"streaming=True","code":true},{"text":", so only the language column is transferred."}],"correct":true,"tempting":null},{"text":[{"text":"Call "},{"text":"load_dataset","code":true},{"text":" with a filter on the language column, because filtering happens before anything is written to disk."}],"correct":false,"tempting":[{"text":"Believes "},{"text":".filter()","code":true},{"text":" on a normal dataset is applied during download rather than after the full copy is cached."}]},{"text":[{"text":"Ask for the count in the dataset’s Community tab, since only the maintainers can compute statistics over a corpus that size."}],"correct":false,"tempting":[{"text":"Assumes large-scale statistics require the owner’s infrastructure, when the columnar format makes them cheap for anyone."}]}],"order":[2,3,0,1],"why":null},"prev":{"slug":"debugging-a-space","title":"It built, it started, it is red","minutes":8},"next":{"slug":"the-datasets-library","title":"A dataset that behaves like a list and is not one","minutes":9}}}