{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"the-trainer-api","title":"Trainer, and what it saves you writing","url":"https://addaly.com/learn/hugging-face/the-trainer-api","position":58,"of":75,"minutes":9,"answer":"Trainer implements the loop, scheduling, mixed precision, checkpointing and resumption that are individually easy and collectively error-prone.","keyIdea":"`Trainer` implements the loop, scheduling, mixed precision, checkpointing and resumption that are individually easy and collectively error-prone, and the two arguments that most often decide whether a run is usable are `id2label` at construction and `resume_from_checkpoint` on restart."},"body":{"format":"markdown","text":"## The whole thing, for a classifier\n\n```python\nimport numpy as np, evaluate\nfrom transformers import (AutoModelForSequenceClassification, AutoTokenizer,\n                          TrainingArguments, Trainer, DataCollatorWithPadding)\n\ntok = AutoTokenizer.from_pretrained(\"distilbert-base-uncased\")\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    \"distilbert-base-uncased\", num_labels=3,\n    id2label={0: \"neg\", 1: \"neu\", 2: \"pos\"},\n    label2id={\"neg\": 0, \"neu\": 1, \"pos\": 2},\n)\n\nds = ds.map(lambda b: tok(b[\"text\"], truncation=True, max_length=256), batched=True)\n\nf1 = evaluate.load(\"f1\")\ndef metrics(p):\n    preds = np.argmax(p.predictions, axis=1)\n    return f1.compute(predictions=preds, references=p.label_ids, average=\"macro\")\n\nargs = TrainingArguments(\n    output_dir=\"out\",\n    learning_rate=2e-5,\n    per_device_train_batch_size=16,\n    num_train_epochs=3,\n    eval_strategy=\"epoch\",\n    save_strategy=\"epoch\",\n    load_best_model_at_end=True,\n    metric_for_best_model=\"f1\",\n    push_to_hub=True,\n    hub_model_id=\"your-name/reviews-classifier\",\n)\n\ntrainer = Trainer(\n    model=model, args=args,\n    train_dataset=ds[\"train\"], eval_dataset=ds[\"validation\"],\n    data_collator=DataCollatorWithPadding(tok),\n    compute_metrics=metrics,\n)\ntrainer.train()\ntrainer.push_to_hub()\n```\n\nThat is a complete, publishable fine-tune. Note `id2label` passed at construction: the thing whose absence produces `LABEL_0` outputs forever, and the cheapest fix in this course.\n\n## What Trainer is doing for you\n\nIt writes, correctly, the loop you would otherwise get subtly wrong: batching and shuffling, moving tensors to the device, the forward and backward passes, gradient accumulation, learning-rate scheduling and warmup, gradient clipping, mixed precision, evaluation at intervals, checkpoint saving and rotation, resumption, logging, and multi-GPU distribution through accelerate.\n\nNone of those is hard alone. Getting all of them right together, and keeping them right, is why nobody writes their own loop for ordinary work.\n\n## Metrics, and `evaluate`\n\n```bash\npip install evaluate\n```\n\n```python\nacc = evaluate.load(\"accuracy\")\nf1  = evaluate.load(\"f1\")\nwer = evaluate.load(\"wer\")          # speech\nrouge = evaluate.load(\"rouge\")      # summarisation\n```\n\n`evaluate` loads standard metric implementations from the Hub, which means everybody computing F1 is computing the same F1. That matters more than it sounds: `average=\"macro\"` and `average=\"micro\"` differ enormously on imbalanced data, and two teams quoting \"F1\" without saying which have not compared anything.\n\nFor a classifier on imbalanced classes, macro-F1 is usually the honest headline, with per-class precision and recall printed beside it.\n\n## Resuming, which you will need\n\n```python\ntrainer.train(resume_from_checkpoint=True)\n```\n\nOn free hardware the session will end. With `save_strategy=\"epoch\"` or `save_steps=200` and this one argument, a disconnection costs minutes rather than hours. Combine it with `push_to_hub=True` and checkpoints leave the machine as well, which is the module-three discipline applied to training.\n\n`save_total_limit=2` stops checkpoints filling the disk: each one is a full copy of the model, and three copies of a 7B model is 42 GB.\n\n## The evaluation loop, and its cost\n\n`eval_strategy=\"steps\"` with `eval_steps=100` runs the whole validation set every hundred steps. On a large validation set that can take longer than the training between evaluations, and people then conclude that training is slow when most of the wall-clock time is measurement.\n\nTwo fixes. Keep the validation set small: a thousand rows is plenty for watching a curve, and the real evaluation happens once at the end on a larger held-out set. And set `per_device_eval_batch_size` explicitly, because it inherits the training batch size by default and evaluation has different memory behaviour.\n\n```python\neval_dataset=ds[\"validation\"].select(range(1000)),\nper_device_eval_batch_size=32,\n```\n\n## Seq2seq and generation\n\n```python\nfrom transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments\n\nargs = Seq2SeqTrainingArguments(..., predict_with_generate=True, generation_max_length=128)\n```\n\n`predict_with_generate=True` makes evaluation actually generate text rather than scoring forced labels, which is what you need for ROUGE or word error rate. Without it your evaluation metric measures something other than what the model will do at inference, and looks better than reality.\n\n## When to write your own loop\n\nWhen you need a custom loss, an unusual optimisation schedule, or a training procedure with several models in it: reinforcement learning, adversarial setups, some distillation schemes. Subclassing `Trainer` and overriding `compute_loss` covers most of these and keeps everything else:\n\n```python\nclass WeightedTrainer(Trainer):\n    def compute_loss(self, model, inputs, return_outputs=False, **kw):\n        labels = inputs.pop(\"labels\")\n        out = model(**inputs)\n        loss = torch.nn.functional.cross_entropy(\n            out.logits, labels, weight=class_weights.to(out.logits.device))\n        return (loss, out) if return_outputs else loss\n```\n\nThat is class weighting for imbalanced data, which the data-quality lesson recommended, in eight lines.\n\n## Do this now\n\nTrain a classifier on one per cent of a dataset for one epoch. It will be bad, and that is not the point: the point is that the whole loop (data, metrics, checkpoints, push) runs end to end in a few minutes. Get the plumbing right on something worthless before you spend hours on something you care about."},"check":{"question":"A summarisation model reports strong ROUGE during training and produces poor summaries at inference. What is the most likely cause?","options":[{"text":"The evaluation set overlaps the training set, so the reported score reflects memorisation.","correct":false,"tempting":"Leakage produces inflated scores, but it would inflate any evaluation method rather than only the training-time one."},{"text":"`predict_with_generate` was left off, so evaluation scored forced reference tokens.","correct":true},{"text":"ROUGE is unsuitable for summarisation and should be replaced with a model-based metric.","correct":false,"tempting":"ROUGE has real weaknesses, but they would not create a gap between training-time and inference-time behaviour."},{"text":"The generation config at inference uses different sampling parameters from those used during evaluation.","correct":false,"tempting":"Worth checking and a real source of variance, but a smaller effect than scoring a different task entirely."}]},"prev":{"slug":"when-not-to-fine-tune","title":"Three cheaper things to try first","url":"https://addaly.com/learn/hugging-face/when-not-to-fine-tune"},"next":{"slug":"training-arguments-that-matter","title":"The six arguments worth understanding","url":"https://addaly.com/learn/hugging-face/training-arguments-that-matter"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Trainer, and what it saves you writing', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/the-trainer-api (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Trainer, and what it saves you writing","courseTitle":"Hugging Face, End to End","module":{"index":7,"slug":"training-with-the-stack","title":"Training and adapting, with the libraries the Hub is built on"},"answer":null,"keyIdea":"`Trainer` implements the loop, scheduling, mixed precision, checkpointing and resumption that are individually easy and collectively error-prone, and the two arguments that most often decide whether a run is usable are `id2label` at construction and `resume_from_checkpoint` on restart.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-whole-thing-for-a-classifier","runs":[{"text":"The whole thing, for a classifier"}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np, evaluate\nfrom transformers import (AutoModelForSequenceClassification, AutoTokenizer,\n                          TrainingArguments, Trainer, DataCollatorWithPadding)\n\ntok = AutoTokenizer.from_pretrained(\"distilbert-base-uncased\")\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    \"distilbert-base-uncased\", num_labels=3,\n    id2label={0: \"neg\", 1: \"neu\", 2: \"pos\"},\n    label2id={\"neg\": 0, \"neu\": 1, \"pos\": 2},\n)\n\nds = ds.map(lambda b: tok(b[\"text\"], truncation=True, max_length=256), batched=True)\n\nf1 = evaluate.load(\"f1\")\ndef metrics(p):\n    preds = np.argmax(p.predictions, axis=1)\n    return f1.compute(predictions=preds, references=p.label_ids, average=\"macro\")\n\nargs = TrainingArguments(\n    output_dir=\"out\",\n    learning_rate=2e-5,\n    per_device_train_batch_size=16,\n    num_train_epochs=3,\n    eval_strategy=\"epoch\",\n    save_strategy=\"epoch\",\n    load_best_model_at_end=True,\n    metric_for_best_model=\"f1\",\n    push_to_hub=True,\n    hub_model_id=\"your-name/reviews-classifier\",\n)\n\ntrainer = Trainer(\n    model=model, args=args,\n    train_dataset=ds[\"train\"], eval_dataset=ds[\"validation\"],\n    data_collator=DataCollatorWithPadding(tok),\n    compute_metrics=metrics,\n)\ntrainer.train()\ntrainer.push_to_hub()"},{"t":"p","runs":[{"text":"That is a complete, publishable fine-tune. Note "},{"text":"id2label","code":true},{"text":" passed at construction: the thing whose absence produces "},{"text":"LABEL_0","code":true},{"text":" outputs forever, and the cheapest fix in this course."}]},{"t":"h2","id":"what-trainer-is-doing-for-you","runs":[{"text":"What Trainer is doing for you"}]},{"t":"p","runs":[{"text":"It writes, correctly, the loop you would otherwise get subtly wrong: batching and shuffling, moving tensors to the device, the forward and backward passes, gradient accumulation, learning-rate scheduling and warmup, gradient clipping, mixed precision, evaluation at intervals, checkpoint saving and rotation, resumption, logging, and multi-GPU distribution through accelerate."}]},{"t":"p","runs":[{"text":"None of those is hard alone. Getting all of them right together, and keeping them right, is why nobody writes their own loop for ordinary work."}]},{"t":"h2","id":"metrics-and-evaluate","runs":[{"text":"Metrics, and "},{"text":"evaluate","code":true}]},{"t":"code","lang":"bash","label":"Bash","text":"pip install evaluate"},{"t":"code","lang":"python","label":"Python","text":"acc = evaluate.load(\"accuracy\")\nf1  = evaluate.load(\"f1\")\nwer = evaluate.load(\"wer\")          # speech\nrouge = evaluate.load(\"rouge\")      # summarisation"},{"t":"p","runs":[{"text":"evaluate","code":true},{"text":" loads standard metric implementations from the Hub, which means everybody computing F1 is computing the same F1. That matters more than it sounds: "},{"text":"average=\"macro\"","code":true},{"text":" and "},{"text":"average=\"micro\"","code":true},{"text":" differ enormously on imbalanced data, and two teams quoting “F1” without saying which have not compared anything."}]},{"t":"p","runs":[{"text":"For a classifier on imbalanced classes, macro-F1 is usually the honest headline, with per-class precision and recall printed beside it."}]},{"t":"h2","id":"resuming-which-you-will-need","runs":[{"text":"Resuming, which you will need"}]},{"t":"code","lang":"python","label":"Python","text":"trainer.train(resume_from_checkpoint=True)"},{"t":"p","runs":[{"text":"On free hardware the session will end. With "},{"text":"save_strategy=\"epoch\"","code":true},{"text":" or "},{"text":"save_steps=200","code":true},{"text":" and this one argument, a disconnection costs minutes rather than hours. Combine it with "},{"text":"push_to_hub=True","code":true},{"text":" and checkpoints leave the machine as well, which is the module-three discipline applied to training."}]},{"t":"p","runs":[{"text":"save_total_limit=2","code":true},{"text":" stops checkpoints filling the disk: each one is a full copy of the model, and three copies of a 7B model is 42 GB."}]},{"t":"h2","id":"the-evaluation-loop-and-its-cost","runs":[{"text":"The evaluation loop, and its cost"}]},{"t":"p","runs":[{"text":"eval_strategy=\"steps\"","code":true},{"text":" with "},{"text":"eval_steps=100","code":true},{"text":" runs the whole validation set every hundred steps. On a large validation set that can take longer than the training between evaluations, and people then conclude that training is slow when most of the wall-clock time is measurement."}]},{"t":"p","runs":[{"text":"Two fixes. Keep the validation set small: a thousand rows is plenty for watching a curve, and the real evaluation happens once at the end on a larger held-out set. And set "},{"text":"per_device_eval_batch_size","code":true},{"text":" explicitly, because it inherits the training batch size by default and evaluation has different memory behaviour."}]},{"t":"code","lang":"python","label":"Python","text":"eval_dataset=ds[\"validation\"].select(range(1000)),\nper_device_eval_batch_size=32,"},{"t":"h2","id":"seq2seq-and-generation","runs":[{"text":"Seq2seq and generation"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments\n\nargs = Seq2SeqTrainingArguments(..., predict_with_generate=True, generation_max_length=128)"},{"t":"p","runs":[{"text":"predict_with_generate=True","code":true},{"text":" makes evaluation actually generate text rather than scoring forced labels, which is what you need for ROUGE or word error rate. Without it your evaluation metric measures something other than what the model will do at inference, and looks better than reality."}]},{"t":"h2","id":"when-to-write-your-own-loop","runs":[{"text":"When to write your own loop"}]},{"t":"p","runs":[{"text":"When you need a custom loss, an unusual optimisation schedule, or a training procedure with several models in it: reinforcement learning, adversarial setups, some distillation schemes. Subclassing "},{"text":"Trainer","code":true},{"text":" and overriding "},{"text":"compute_loss","code":true},{"text":" covers most of these and keeps everything else:"}]},{"t":"code","lang":"python","label":"Python","text":"class WeightedTrainer(Trainer):\n    def compute_loss(self, model, inputs, return_outputs=False, **kw):\n        labels = inputs.pop(\"labels\")\n        out = model(**inputs)\n        loss = torch.nn.functional.cross_entropy(\n            out.logits, labels, weight=class_weights.to(out.logits.device))\n        return (loss, out) if return_outputs else loss"},{"t":"p","runs":[{"text":"That is class weighting for imbalanced data, which the data-quality lesson recommended, in eight lines."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Train a classifier on one per cent of a dataset for one epoch. It will be bad, and that is not the point: the point is that the whole loop (data, metrics, checkpoints, push) runs end to end in a few minutes. Get the plumbing right on something worthless before you spend hours on something you care about."}]}],"check":{"question":[{"text":"A summarisation model reports strong ROUGE during training and produces poor summaries at inference. What is the most likely cause?"}],"options":[{"text":[{"text":"The evaluation set overlaps the training set, so the reported score reflects memorisation."}],"correct":false,"tempting":[{"text":"Leakage produces inflated scores, but it would inflate any evaluation method rather than only the training-time one."}]},{"text":[{"text":"predict_with_generate","code":true},{"text":" was left off, so evaluation scored forced reference tokens."}],"correct":true,"tempting":null},{"text":[{"text":"ROUGE is unsuitable for summarisation and should be replaced with a model-based metric."}],"correct":false,"tempting":[{"text":"ROUGE has real weaknesses, but they would not create a gap between training-time and inference-time behaviour."}]},{"text":[{"text":"The generation config at inference uses different sampling parameters from those used during evaluation."}],"correct":false,"tempting":[{"text":"Worth checking and a real source of variance, but a smaller effect than scoring a different task entirely."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"when-not-to-fine-tune","title":"Three cheaper things to try first","minutes":8},"next":{"slug":"training-arguments-that-matter","title":"The six arguments worth understanding","minutes":9}}}