Hugging Face, End to End · Chapter 05
Do not download the dataset
2:05 · Narrated with a synthetic voice. Captions are available in the player.
Read the transcript
Before any model there is the data, and the first rule is not to download it. Public datasets on the Hub are converted to a columnar format automatically, and that format keeps its column statistics in the file footer. So a database engine can read the footer over the network, work out which row groups it needs, and fetch only those byte ranges. A grouped count over a fifty gigabyte corpus can transfer a few megabytes. Which means you can answer, from a phone, before downloading anything: how many rows are in my language, how long the texts are, whether there are duplicates, and whether the test split overlaps the training split. People routinely spend two hours on a progress bar to discover the column they needed is empty. Then four checks, in about an hour, that decide whether your result will mean anything. Duplicates, because four hundred copies of one automated message is a single row of information pretending to be four hundred. Leakage, which means splitting along the axis your deployment has to generalise across: by document, by customer, by speaker, or by date when you are predicting forward. Class balance, because at sixty one per cent in one class a model that always answers that class scores sixty one per cent and has learned nothing, and that is the first number anybody will quote. And label noise, which you find by reading a hundred rows rather than a summary. If two labellers agreed on seventy eight per cent of items, seventy eight is roughly your ceiling, and knowing it first saves weeks.