{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"audio-and-image-datasets","title":"Media columns decode when you touch them","url":"https://addaly.com/learn/hugging-face/audio-and-image-datasets","position":45,"of":75,"minutes":8,"answer":"Audio and Image columns decode lazily on access, so cast_column can resample or defer decoding for free, and the silent failures in media work are a sampling rate that does not match the model.","keyIdea":"`Audio` and `Image` columns decode lazily on access, so `cast_column` can resample or defer decoding for free, and the silent failures in media work are a sampling rate that does not match the model, processor constants borrowed from another model, and augmentation frozen into a cache by `map`."},"body":{"format":"markdown","text":"## What is actually stored\n\n```python\nds = load_dataset(\"PolyAI/minds14\", \"en-US\", split=\"train\")\nprint(ds.features[\"audio\"])\n# Audio(sampling_rate=8000, mono=True, decode=True, id=None)\n\nrow = ds[0]\nprint(row[\"audio\"][\"array\"].shape, row[\"audio\"][\"sampling_rate\"])\n```\n\nThe stored column holds a path or the encoded bytes. The decoded waveform appears only when you access the row: `Audio` and `Image` are lazy decoding types. So a dataset of 100,000 photographs takes the disk of 100,000 compressed JPEGs, not of 100,000 raw arrays, and iterating decodes one at a time.\n\nTwo consequences worth holding:\n\n- **Never call `ds.map` over a media column carelessly.** If your function touches `row[\"image\"]`, every image decodes and the result is written back as decoded data, which can be fifty times the size. Use `set_transform` for augmentation, or return only the small derived columns you need.\n- **`ds.cast_column(\"image\", Image(decode=False))`** gives you paths and bytes without decoding, which is what you want for deduplication, hashing or copying files.\n\n## Sampling rate is the audio bug\n\nEvery speech model expects a specific rate. Whisper and most modern models want 16,000 Hz. Telephone corpora are often 8,000. Feed the wrong rate and you get no error and a much worse transcript, because the model hears something pitched and paced wrongly.\n\n```python\nfrom datasets import Audio\nds = ds.cast_column(\"audio\", Audio(sampling_rate=16_000))\n```\n\n`cast_column` resamples lazily on access. It costs nothing until the row is read, and it fixes the single most common silent failure in audio work. Check the model's expected rate in its processor config, not in its readme.\n\nNote that upsampling 8 kHz telephone audio to 16 kHz does not restore information that was never recorded. It makes the model's input well-formed; it cannot make a narrowband recording wideband. Expect a higher error rate on telephone audio and do not attribute it to the resampling.\n\n## Image preprocessing belongs to the processor\n\n```python\nfrom transformers import AutoImageProcessor\nproc = AutoImageProcessor.from_pretrained(name)\nprint(proc.image_mean, proc.image_std, proc.size)\n```\n\nEvery vision model has its own resize dimensions and normalisation constants. Using another model's constants produces a quiet accuracy drop with no error: the same shape of failure as the wrong sampling rate. Always load the processor from the same repository as the weights, exactly as with tokenizers.\n\n```python\ndef prep(batch):\n    batch[\"pixel_values\"] = proc(images=batch[\"image\"], return_tensors=\"pt\")[\"pixel_values\"]\n    return batch\n\nds.set_transform(prep)      # on access, not materialised\n```\n\n## Augmentation is per epoch, so it is a transform\n\nRandom crops, flips, colour jitter and noise must differ between epochs. `map` would freeze one random draw into a cached file and train on the same \"random\" augmentation every time: a subtle bug that shows up as a model that overfits despite augmentation. `set_transform` re-runs on every access, which is what augmentation means.\n\n## Streaming media\n\n```python\nds = load_dataset(\"mozilla-foundation/common_voice_17_0\", \"hi\",\n                  split=\"train\", streaming=True)\nfor row in ds.take(3):\n    print(row[\"sentence\"], row[\"audio\"][\"array\"].shape)\n```\n\nSpeech corpora are hundreds of gigabytes. Streaming is not an optimisation here, it is the only way most people can touch them at all. Many require accepting terms on the dataset page first, which is the gating mechanism from the tokens lesson.\n\n## Video, briefly\n\nVideo support exists and is younger. In practice most video work still means extracting frames and audio with `ffmpeg` and treating the result as an image dataset plus an audio dataset, which is free, scriptable, and avoids depending on decoder support that varies by environment.\n\n## What to check on any media dataset before training\n\n1. **Sampling rate** or **image size** distribution: are they uniform, and do they match the model?\n2. **Duration or dimension outliers.** One eight-hour recording in a corpus of ten-second clips will consume a whole batch and may exhaust memory.\n3. **Corrupt files.** Iterate once with a try/except and count failures. There are always some.\n4. **Silence and blank frames.** Cheap to detect, and they train the model on nothing.\n5. **Speaker or source overlap between splits.** The media equivalent of leakage: the same speaker in train and test makes your word error rate optimistic in a way that does not survive deployment.\n\n## Do this now\n\nLoad any audio dataset, print its sampling rate, and compare it against what your chosen model's processor expects. If they differ, you have just found the reason a transcript was worse than you expected."},"check":{"question":"A team applies random crops and flips with `ds.map` and finds the model overfits as if no augmentation were present. What is the mechanism?","options":[{"text":"`map` caches its output, so one random draw is frozen and every epoch trains on identical augmented images.","correct":true},{"text":"Augmentation applied before the image processor is undone by the processor's resize and normalisation step.","correct":false,"tempting":"Resizing changes geometry but does not reverse a crop or a flip."},{"text":"Lazy decoding means the augmentation function received encoded bytes rather than pixel arrays, so it silently did nothing.","correct":false,"tempting":"Accessing the column decodes it; a function operating on bytes would raise rather than pass through."},{"text":"Cached Arrow files store images losslessly, so the augmented copies are too similar to the originals to add variety.","correct":false,"tempting":"Attributes the failure to storage fidelity rather than to a single frozen random draw."}]},"prev":{"slug":"dataset-cards-and-provenance","title":"The card is where a dataset stops being a file","url":"https://addaly.com/learn/hugging-face/dataset-cards-and-provenance"},"next":{"slug":"data-quality-before-the-model","title":"Duplicates, leakage, balance, and label noise","url":"https://addaly.com/learn/hugging-face/data-quality-before-the-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Media columns decode when you touch them', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/audio-and-image-datasets (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Media columns decode when you touch them","courseTitle":"Hugging Face, End to End","module":{"index":5,"slug":"datasets-and-the-work-before-the-model","title":"Datasets, and the work that happens before any model"},"answer":null,"keyIdea":"`Audio` and `Image` columns decode lazily on access, so `cast_column` can resample or defer decoding for free, and the silent failures in media work are a sampling rate that does not match the model, processor constants borrowed from another model, and augmentation frozen into a cache by `map`.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-is-actually-stored","runs":[{"text":"What is actually stored"}]},{"t":"code","lang":"python","label":"Python","text":"ds = load_dataset(\"PolyAI/minds14\", \"en-US\", split=\"train\")\nprint(ds.features[\"audio\"])\n# Audio(sampling_rate=8000, mono=True, decode=True, id=None)\n\nrow = ds[0]\nprint(row[\"audio\"][\"array\"].shape, row[\"audio\"][\"sampling_rate\"])"},{"t":"p","runs":[{"text":"The stored column holds a path or the encoded bytes. The decoded waveform appears only when you access the row: "},{"text":"Audio","code":true},{"text":" and "},{"text":"Image","code":true},{"text":" are lazy decoding types. So a dataset of 100,000 photographs takes the disk of 100,000 compressed JPEGs, not of 100,000 raw arrays, and iterating decodes one at a time."}]},{"t":"p","runs":[{"text":"Two consequences worth holding:"}]},{"t":"ul","items":[[{"text":"Never call ","b":true},{"text":"ds.map","b":true,"code":true},{"text":" over a media column carelessly.","b":true},{"text":" If your function touches "},{"text":"row[\"image\"]","code":true},{"text":", every image decodes and the result is written back as decoded data, which can be fifty times the size. Use "},{"text":"set_transform","code":true},{"text":" for augmentation, or return only the small derived columns you need."}],[{"text":"ds.cast_column(\"image\", Image(decode=False))","b":true,"code":true},{"text":" gives you paths and bytes without decoding, which is what you want for deduplication, hashing or copying files."}]]},{"t":"h2","id":"sampling-rate-is-the-audio-bug","runs":[{"text":"Sampling rate is the audio bug"}]},{"t":"p","runs":[{"text":"Every speech model expects a specific rate. Whisper and most modern models want 16,000 Hz. Telephone corpora are often 8,000. Feed the wrong rate and you get no error and a much worse transcript, because the model hears something pitched and paced wrongly."}]},{"t":"code","lang":"python","label":"Python","text":"from datasets import Audio\nds = ds.cast_column(\"audio\", Audio(sampling_rate=16_000))"},{"t":"p","runs":[{"text":"cast_column","code":true},{"text":" resamples lazily on access. It costs nothing until the row is read, and it fixes the single most common silent failure in audio work. Check the model’s expected rate in its processor config, not in its readme."}]},{"t":"p","runs":[{"text":"Note that upsampling 8 kHz telephone audio to 16 kHz does not restore information that was never recorded. It makes the model’s input well-formed; it cannot make a narrowband recording wideband. Expect a higher error rate on telephone audio and do not attribute it to the resampling."}]},{"t":"h2","id":"image-preprocessing-belongs-to-the-processor","runs":[{"text":"Image preprocessing belongs to the processor"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import AutoImageProcessor\nproc = AutoImageProcessor.from_pretrained(name)\nprint(proc.image_mean, proc.image_std, proc.size)"},{"t":"p","runs":[{"text":"Every vision model has its own resize dimensions and normalisation constants. Using another model’s constants produces a quiet accuracy drop with no error: the same shape of failure as the wrong sampling rate. Always load the processor from the same repository as the weights, exactly as with tokenizers."}]},{"t":"code","lang":"python","label":"Python","text":"def prep(batch):\n    batch[\"pixel_values\"] = proc(images=batch[\"image\"], return_tensors=\"pt\")[\"pixel_values\"]\n    return batch\n\nds.set_transform(prep)      # on access, not materialised"},{"t":"h2","id":"augmentation-is-per-epoch-so-it-is-a-transform","runs":[{"text":"Augmentation is per epoch, so it is a transform"}]},{"t":"p","runs":[{"text":"Random crops, flips, colour jitter and noise must differ between epochs. "},{"text":"map","code":true},{"text":" would freeze one random draw into a cached file and train on the same “random” augmentation every time: a subtle bug that shows up as a model that overfits despite augmentation. "},{"text":"set_transform","code":true},{"text":" re-runs on every access, which is what augmentation means."}]},{"t":"h2","id":"streaming-media","runs":[{"text":"Streaming media"}]},{"t":"code","lang":"python","label":"Python","text":"ds = load_dataset(\"mozilla-foundation/common_voice_17_0\", \"hi\",\n                  split=\"train\", streaming=True)\nfor row in ds.take(3):\n    print(row[\"sentence\"], row[\"audio\"][\"array\"].shape)"},{"t":"p","runs":[{"text":"Speech corpora are hundreds of gigabytes. Streaming is not an optimisation here, it is the only way most people can touch them at all. Many require accepting terms on the dataset page first, which is the gating mechanism from the tokens lesson."}]},{"t":"h2","id":"video-briefly","runs":[{"text":"Video, briefly"}]},{"t":"p","runs":[{"text":"Video support exists and is younger. In practice most video work still means extracting frames and audio with "},{"text":"ffmpeg","code":true},{"text":" and treating the result as an image dataset plus an audio dataset, which is free, scriptable, and avoids depending on decoder support that varies by environment."}]},{"t":"h2","id":"what-to-check-on-any-media-dataset-before-training","runs":[{"text":"What to check on any media dataset before training"}]},{"t":"ol","start":1,"items":[[{"text":"Sampling rate","b":true},{"text":" or "},{"text":"image size","b":true},{"text":" distribution: are they uniform, and do they match the model?"}],[{"text":"Duration or dimension outliers.","b":true},{"text":" One eight-hour recording in a corpus of ten-second clips will consume a whole batch and may exhaust memory."}],[{"text":"Corrupt files.","b":true},{"text":" Iterate once with a try/except and count failures. There are always some."}],[{"text":"Silence and blank frames.","b":true},{"text":" Cheap to detect, and they train the model on nothing."}],[{"text":"Speaker or source overlap between splits.","b":true},{"text":" The media equivalent of leakage: the same speaker in train and test makes your word error rate optimistic in a way that does not survive deployment."}]]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Load any audio dataset, print its sampling rate, and compare it against what your chosen model’s processor expects. If they differ, you have just found the reason a transcript was worse than you expected."}]}],"check":{"question":[{"text":"A team applies random crops and flips with "},{"text":"ds.map","code":true},{"text":" and finds the model overfits as if no augmentation were present. What is the mechanism?"}],"options":[{"text":[{"text":"map","code":true},{"text":" caches its output, so one random draw is frozen and every epoch trains on identical augmented images."}],"correct":true,"tempting":null},{"text":[{"text":"Augmentation applied before the image processor is undone by the processor’s resize and normalisation step."}],"correct":false,"tempting":[{"text":"Resizing changes geometry but does not reverse a crop or a flip."}]},{"text":[{"text":"Lazy decoding means the augmentation function received encoded bytes rather than pixel arrays, so it silently did nothing."}],"correct":false,"tempting":[{"text":"Accessing the column decodes it; a function operating on bytes would raise rather than pass through."}]},{"text":[{"text":"Cached Arrow files store images losslessly, so the augmented copies are too similar to the originals to add variety."}],"correct":false,"tempting":[{"text":"Attributes the failure to storage fidelity rather than to a single frozen random draw."}]}],"order":[2,3,0,1],"why":null},"prev":{"slug":"dataset-cards-and-provenance","title":"The card is where a dataset stops being a file","minutes":8},"next":{"slug":"data-quality-before-the-model","title":"Duplicates, leakage, balance, and label noise","minutes":10}}}