Media columns decode when you touch them

What is actually stored

ds = load_dataset("PolyAI/minds14", "en-US", split="train")
print(ds.features["audio"])
# Audio(sampling_rate=8000, mono=True, decode=True, id=None)

row = ds[0]
print(row["audio"]["array"].shape, row["audio"]["sampling_rate"])

The stored column holds a path or the encoded bytes. The decoded waveform appears only when you access the row: Audio and Image are lazy decoding types. So a dataset of 100,000 photographs takes the disk of 100,000 compressed JPEGs, not of 100,000 raw arrays, and iterating decodes one at a time.

Two consequences worth holding:

• Never call ds.map over a media column carelessly. If your function touches row["image"], every image decodes and the result is written back as decoded data, which can be fifty times the size. Use set_transform for augmentation, or return only the small derived columns you need.
• ds.cast_column("image", Image(decode=False)) gives you paths and bytes without decoding, which is what you want for deduplication, hashing or copying files.

Sampling rate is the audio bug

Every speech model expects a specific rate. Whisper and most modern models want 16,000 Hz. Telephone corpora are often 8,000. Feed the wrong rate and you get no error and a much worse transcript, because the model hears something pitched and paced wrongly.

from datasets import Audio
ds = ds.cast_column("audio", Audio(sampling_rate=16_000))

cast_column resamples lazily on access. It costs nothing until the row is read, and it fixes the single most common silent failure in audio work. Check the model’s expected rate in its processor config, not in its readme.

Note that upsampling 8 kHz telephone audio to 16 kHz does not restore information that was never recorded. It makes the model’s input well-formed; it cannot make a narrowband recording wideband. Expect a higher error rate on telephone audio and do not attribute it to the resampling.

Image preprocessing belongs to the processor

from transformers import AutoImageProcessor
proc = AutoImageProcessor.from_pretrained(name)
print(proc.image_mean, proc.image_std, proc.size)

Every vision model has its own resize dimensions and normalisation constants. Using another model’s constants produces a quiet accuracy drop with no error: the same shape of failure as the wrong sampling rate. Always load the processor from the same repository as the weights, exactly as with tokenizers.

def prep(batch):
    batch["pixel_values"] = proc(images=batch["image"], return_tensors="pt")["pixel_values"]
    return batch

ds.set_transform(prep)      # on access, not materialised

Augmentation is per epoch, so it is a transform

Random crops, flips, colour jitter and noise must differ between epochs. map would freeze one random draw into a cached file and train on the same “random” augmentation every time: a subtle bug that shows up as a model that overfits despite augmentation. set_transform re-runs on every access, which is what augmentation means.

Streaming media

ds = load_dataset("mozilla-foundation/common_voice_17_0", "hi",
                  split="train", streaming=True)
for row in ds.take(3):
    print(row["sentence"], row["audio"]["array"].shape)

Speech corpora are hundreds of gigabytes. Streaming is not an optimisation here, it is the only way most people can touch them at all. Many require accepting terms on the dataset page first, which is the gating mechanism from the tokens lesson.

Video, briefly

Video support exists and is younger. In practice most video work still means extracting frames and audio with ffmpeg and treating the result as an image dataset plus an audio dataset, which is free, scriptable, and avoids depending on decoder support that varies by environment.

What to check on any media dataset before training

1. Sampling rate or image size distribution: are they uniform, and do they match the model?
2. Duration or dimension outliers. One eight-hour recording in a corpus of ten-second clips will consume a whole batch and may exhaust memory.
3. Corrupt files. Iterate once with a try/except and count failures. There are always some.
4. Silence and blank frames. Cheap to detect, and they train the model on nothing.
5. Speaker or source overlap between splits. The media equivalent of leakage: the same speaker in train and test makes your word error rate optimistic in a way that does not survive deployment.

Do this now

Load any audio dataset, print its sampling rate, and compare it against what your chosen model’s processor expects. If they differ, you have just found the reason a transcript was worse than you expected.