Turning your files into a dataset

From whatever you have

from datasets import Dataset, DatasetDict

ds = Dataset.from_dict({"text": texts, "label": labels})
ds = Dataset.from_pandas(df)
ds = Dataset.from_list([{"text": "...", "label": 1}, ...])

For anything too large to hold in memory, use a generator so rows stream in:

def gen():
    for path in pathlib.Path("notes").glob("*.txt"):
        yield {"id": path.stem, "text": path.read_text(encoding="utf-8")}

ds = Dataset.from_generator(gen)

For media, the folder conventions do the labelling:

ds = load_dataset("imagefolder", data_dir="photos")   # photos/cat/*.jpg -> label 'cat'
ds = load_dataset("audiofolder", data_dir="clips")

A metadata.csv in the folder, with a file_name column, attaches captions, transcripts or extra fields to each file.

Set the types deliberately

from datasets import Features, Value, ClassLabel

features = Features({
    "text": Value("string"),
    "label": ClassLabel(names=["neg", "neu", "pos"]),
    "source": Value("string"),
})
ds = Dataset.from_dict(data, features=features)

ClassLabel stores the names with the data, so nobody downstream has to guess what 2 means. Without it your labels are bare integers and the meaning lives in a message somebody sent you in March. This is the same class of mistake as a model with LABEL_0 outputs, and it is equally cheap to prevent.

Include a source or provenance column from the start. Six months later, every argument about a surprising result begins with where a row came from.

Publishing

ds.push_to_hub("your-name/support-tickets", private=True)

That creates the repository if needed, writes parquet shards, and commits. Loading it back is load_dataset("your-name/support-tickets").

Start private. Making a private dataset public later is one click; un-publishing something that has been indexed, cloned and mirrored is not possible. The asymmetry should decide the default.

Push a DatasetDict and the splits are preserved:

DatasetDict({"train": tr, "validation": va, "test": te}).push_to_hub(repo)

You can also push a configuration alongside others, which is how one repository holds several variants:

ds_hi.push_to_hub(repo, config_name="hindi")
ds_en.push_to_hub(repo, config_name="english")

Before you push anything

This list is short and every item on it has ruined somebody’s week:

1. Personal data. Names, emails, phone numbers, addresses, account numbers, and the sentence where somebody names their employer. Run an automatic redaction pass, then read a sample by hand, because automatic redaction misses context. If the data is about people and you cannot redact it, the dataset stays private.
2. Do you have the right to publish it? Scraped content, customer data, anything under a contract, anything with a licence that forbids redistribution. Being able to download something is not permission to republish it.
3. Credentials in the rows. API keys pasted into support tickets are extremely common. Grep for them.
4. A licence field. Decide before publishing, not after. No licence is not the same as permissive.
5. The intended use and the known gaps, in the card.

The upload that silently changes your data

One behaviour surprises people. push_to_hub writes the dataset as it currently exists, including every transformation you applied in the session. If you tokenised and then pushed, you have published token ids rather than text, and nobody else can use it with a different model.

Publish the raw form. Keep preprocessing in a script in a repository beside it, so somebody with a different tokenizer can run your pipeline rather than inherit your choices. The rule generalises: publish the thing, not your view of the thing.

Keeping it updated

A dataset repository is git, so a new push is a commit and old versions remain reachable:

ds = load_dataset("your-name/support-tickets", revision="v1.0")

Tag releases when a number you published was computed against a specific version. A model card that says “trained on our dataset” without a revision is a claim nobody can check, including you.

Very large uploads

push_to_hub handles gigabytes. For hundreds of gigabytes, shard before uploading, use upload_large_folder, and expect to resume: a four-hour upload will be interrupted. The chunk-level storage behind the Hub deduplicates, so re-pushing a dataset where one shard changed does not resend everything.

Do this now

Take a folder of your own files (notes, tickets, transcripts, photos) and make it a private dataset with typed features and a source column. Then load it back with load_dataset and confirm the types survived. That round trip is the whole skill, and it takes ten minutes.