{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"the-auto-classes","title":"AutoModel gives you numbers, not answers","url":"https://addaly.com/learn/hugging-face/the-auto-classes","position":11,"of":75,"minutes":9,"answer":"AutoModel loads the network body and returns hidden states, while AutoModelForX attaches a task head, and when the checkpoint has no such head, transformers initialises it randomly and warns rather than failing.","keyIdea":"`AutoModel` loads the network body and returns hidden states, while `AutoModelForX` attaches a task head, and when the checkpoint has no such head, transformers initialises it randomly and warns rather than failing, so a classifier that returns confident noise is the expected outcome of loading a base checkpoint."},"body":{"format":"markdown","text":"## Three Auto classes, and only one of them is safe to guess\n\n```python\nfrom transformers import AutoConfig, AutoTokenizer, AutoModel\n\nname = \"distilbert-base-uncased\"\ncfg = AutoConfig.from_pretrained(name)\ntok = AutoTokenizer.from_pretrained(name)\nmodel = AutoModel.from_pretrained(name)\n```\n\nEach `Auto*` class reads `config.json`, finds `architectures`, and instantiates the matching implementation. That is all the magic is: a lookup table from a string to a Python class, so you do not have to import `DistilBertTokenizerFast` by name.\n\n`AutoConfig` and `AutoTokenizer` do what you expect. `AutoModel` does something that catches almost everyone.\n\n## The body without the head\n\n```python\nout = model(**tok(\"the food was excellent\", return_tensors=\"pt\"))\nprint(out.last_hidden_state.shape)   # torch.Size([1, 6, 768])\n```\n\nNo labels. No scores. A tensor of 6 positions × 768 numbers: one vector per token. `AutoModel` loads the **body** of the network and nothing else. It is the right class when you want representations to feed something of your own, and the wrong class when you want a prediction.\n\nFor a prediction you name the head:\n\n```python\nfrom transformers import AutoModelForSequenceClassification\n\nm = AutoModelForSequenceClassification.from_pretrained(\n    \"distilbert-base-uncased-finetuned-sst-2-english\"\n)\n```\n\nThe common heads, and what each is for:\n\n- `AutoModelForSequenceClassification`: one label for the whole input. Sentiment, intent, spam.\n- `AutoModelForTokenClassification`: one label per token. Named entities, part of speech.\n- `AutoModelForQuestionAnswering`: start and end positions of a span.\n- `AutoModelForCausalLM`: next-token prediction. Every chat and completion model.\n- `AutoModelForSeq2SeqLM`: encoder-decoder. Translation, older summarisers, T5.\n- `AutoModelForMaskedLM`: fill in a masked token. BERT-style pretraining.\n\n## The warning that means your classifier is random\n\nLoad a head that the checkpoint does not contain and transformers tells you, in a message people read as boilerplate:\n\n> Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']. You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n\nThose two tensors were filled with random numbers. The model will load, run, and return confident-looking scores that are pure noise. Nothing downstream will fail. You will get roughly 50% accuracy on a two-class problem and spend a day blaming the data.\n\nThe distinction is exactly this: `distilbert-base-uncased` is a **base** checkpoint with no classifier. `distilbert-base-uncased-finetuned-sst-2-english` has one. The warning is the only thing that tells them apart at load time, and it is worth treating as an error rather than a log line.\n\n## `id2label`, and why your labels say LABEL_0\n\nWhen output comes back as `LABEL_0` and `LABEL_1`, the uploader did not fill in the mapping:\n\n```python\nprint(m.config.id2label)   # {0: 'NEGATIVE', 1: 'POSITIVE'}\n```\n\nIf it prints `{0: 'LABEL_0', 1: 'LABEL_1'}`, the card is the only place the real meaning exists, and sometimes it is only in a discussion thread. Getting the polarity backwards because you assumed index 1 meant positive is a common and completely silent error. Check, do not assume.\n\n## The other Auto classes you will meet\n\n- `AutoProcessor`: the general wrapper for models that take more than text: it bundles a tokenizer with an image processor or a feature extractor. Vision-language models and Whisper use it.\n- `AutoImageProcessor`: resizing and normalisation for vision models. Normalisation constants differ per model and using the wrong ones degrades accuracy without any error.\n- `AutoModelForVision2Seq`, `AutoModelForSpeechSeq2Seq`: heads for captioning and transcription.\n\n## The rule that prevents most of this\n\n**Load the tokenizer and the model from the same repository identifier, always.** A tokenizer from `bert-base-uncased` with weights from a fine-tune of a different vocabulary produces integers the model was never trained on. No exception is raised, because integers are integers. The output is fluent nonsense, which is the hardest kind of bug to spot.\n\nWhen you must mix them, loading an adapter, say, check that `len(tokenizer)` equals `model.config.vocab_size` and stop if it does not.\n\n## Do this now\n\nLoad `distilbert-base-uncased` with `AutoModelForSequenceClassification` and read the warning in full. Then run it on ten sentences and look at the predictions. That is what a randomly initialised head looks like from the outside: confident, varied, and meaningless."},"check":{"question":"Someone loads `bert-base-uncased` with `AutoModelForSequenceClassification`, sees a warning, and gets about 50% accuracy on a balanced two-class task. What happened?","options":[{"text":"The base checkpoint was pretrained on masked language modelling, so its representations are unsuitable for classification without a different architecture.","correct":false,"tempting":"Blames the pretraining objective; those representations are exactly what fine-tuning uses successfully."},{"text":"The tokenizer and the model came from the same repository but the label mapping defaulted to LABEL_0 and LABEL_1, inverting the classes.","correct":false,"tempting":"An inverted mapping would produce accuracy near zero rather than near chance, and would not explain the warning."},{"text":"The classification head does not exist in the checkpoint, so it was created with random weights and never trained.","correct":true},{"text":"Sequence classification needs `AutoModelForMaskedLM` for BERT-family models, and the wrong head silently returns pooled embeddings instead of logits.","correct":false,"tempting":"Mismatches head to task; masked LM predicts vocabulary tokens, not class labels."}]},"prev":{"slug":"your-first-pipeline","title":"One line that hides five steps","url":"https://addaly.com/learn/hugging-face/your-first-pipeline"},"next":{"slug":"tokenizers-in-practice","title":"Your text is integers, and the count is not the word count","url":"https://addaly.com/learn/hugging-face/tokenizers-in-practice"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'AutoModel gives you numbers, not answers', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/the-auto-classes (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"AutoModel gives you numbers, not answers","courseTitle":"Hugging Face, End to End","module":{"index":2,"slug":"transformers-from-text-to-text","title":"Transformers, from characters to output"},"answer":null,"keyIdea":"`AutoModel` loads the network body and returns hidden states, while `AutoModelForX` attaches a task head, and when the checkpoint has no such head, transformers initialises it randomly and warns rather than failing, so a classifier that returns confident noise is the expected outcome of loading a base checkpoint.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"three-auto-classes-and-only-one-of-them-is-safe-to-guess","runs":[{"text":"Three Auto classes, and only one of them is safe to guess"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import AutoConfig, AutoTokenizer, AutoModel\n\nname = \"distilbert-base-uncased\"\ncfg = AutoConfig.from_pretrained(name)\ntok = AutoTokenizer.from_pretrained(name)\nmodel = AutoModel.from_pretrained(name)"},{"t":"p","runs":[{"text":"Each "},{"text":"Auto*","code":true},{"text":" class reads "},{"text":"config.json","code":true},{"text":", finds "},{"text":"architectures","code":true},{"text":", and instantiates the matching implementation. That is all the magic is: a lookup table from a string to a Python class, so you do not have to import "},{"text":"DistilBertTokenizerFast","code":true},{"text":" by name."}]},{"t":"p","runs":[{"text":"AutoConfig","code":true},{"text":" and "},{"text":"AutoTokenizer","code":true},{"text":" do what you expect. "},{"text":"AutoModel","code":true},{"text":" does something that catches almost everyone."}]},{"t":"h2","id":"the-body-without-the-head","runs":[{"text":"The body without the head"}]},{"t":"code","lang":"python","label":"Python","text":"out = model(**tok(\"the food was excellent\", return_tensors=\"pt\"))\nprint(out.last_hidden_state.shape)   # torch.Size([1, 6, 768])"},{"t":"p","runs":[{"text":"No labels. No scores. A tensor of 6 positions × 768 numbers: one vector per token. "},{"text":"AutoModel","code":true},{"text":" loads the "},{"text":"body","b":true},{"text":" of the network and nothing else. It is the right class when you want representations to feed something of your own, and the wrong class when you want a prediction."}]},{"t":"p","runs":[{"text":"For a prediction you name the head:"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import AutoModelForSequenceClassification\n\nm = AutoModelForSequenceClassification.from_pretrained(\n    \"distilbert-base-uncased-finetuned-sst-2-english\"\n)"},{"t":"p","runs":[{"text":"The common heads, and what each is for:"}]},{"t":"ul","items":[[{"text":"AutoModelForSequenceClassification","code":true},{"text":": one label for the whole input. Sentiment, intent, spam."}],[{"text":"AutoModelForTokenClassification","code":true},{"text":": one label per token. Named entities, part of speech."}],[{"text":"AutoModelForQuestionAnswering","code":true},{"text":": start and end positions of a span."}],[{"text":"AutoModelForCausalLM","code":true},{"text":": next-token prediction. Every chat and completion model."}],[{"text":"AutoModelForSeq2SeqLM","code":true},{"text":": encoder-decoder. Translation, older summarisers, T5."}],[{"text":"AutoModelForMaskedLM","code":true},{"text":": fill in a masked token. BERT-style pretraining."}]]},{"t":"h2","id":"the-warning-that-means-your-classifier-is-random","runs":[{"text":"The warning that means your classifier is random"}]},{"t":"p","runs":[{"text":"Load a head that the checkpoint does not contain and transformers tells you, in a message people read as boilerplate:"}]},{"t":"quote","runs":[{"text":"Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: [‘classifier.weight’, ‘classifier.bias’]. You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference."}]},{"t":"p","runs":[{"text":"Those two tensors were filled with random numbers. The model will load, run, and return confident-looking scores that are pure noise. Nothing downstream will fail. You will get roughly 50% accuracy on a two-class problem and spend a day blaming the data."}]},{"t":"p","runs":[{"text":"The distinction is exactly this: "},{"text":"distilbert-base-uncased","code":true},{"text":" is a "},{"text":"base","b":true},{"text":" checkpoint with no classifier. "},{"text":"distilbert-base-uncased-finetuned-sst-2-english","code":true},{"text":" has one. The warning is the only thing that tells them apart at load time, and it is worth treating as an error rather than a log line."}]},{"t":"h2","id":"id2label-and-why-your-labels-say-label0","runs":[{"text":"id2label","code":true},{"text":", and why your labels say LABEL_0"}]},{"t":"p","runs":[{"text":"When output comes back as "},{"text":"LABEL_0","code":true},{"text":" and "},{"text":"LABEL_1","code":true},{"text":", the uploader did not fill in the mapping:"}]},{"t":"code","lang":"python","label":"Python","text":"print(m.config.id2label)   # {0: 'NEGATIVE', 1: 'POSITIVE'}"},{"t":"p","runs":[{"text":"If it prints "},{"text":"{0: 'LABEL_0', 1: 'LABEL_1'}","code":true},{"text":", the card is the only place the real meaning exists, and sometimes it is only in a discussion thread. Getting the polarity backwards because you assumed index 1 meant positive is a common and completely silent error. Check, do not assume."}]},{"t":"h2","id":"the-other-auto-classes-you-will-meet","runs":[{"text":"The other Auto classes you will meet"}]},{"t":"ul","items":[[{"text":"AutoProcessor","code":true},{"text":": the general wrapper for models that take more than text: it bundles a tokenizer with an image processor or a feature extractor. Vision-language models and Whisper use it."}],[{"text":"AutoImageProcessor","code":true},{"text":": resizing and normalisation for vision models. Normalisation constants differ per model and using the wrong ones degrades accuracy without any error."}],[{"text":"AutoModelForVision2Seq","code":true},{"text":", "},{"text":"AutoModelForSpeechSeq2Seq","code":true},{"text":": heads for captioning and transcription."}]]},{"t":"h2","id":"the-rule-that-prevents-most-of-this","runs":[{"text":"The rule that prevents most of this"}]},{"t":"p","runs":[{"text":"Load the tokenizer and the model from the same repository identifier, always.","b":true},{"text":" A tokenizer from "},{"text":"bert-base-uncased","code":true},{"text":" with weights from a fine-tune of a different vocabulary produces integers the model was never trained on. No exception is raised, because integers are integers. The output is fluent nonsense, which is the hardest kind of bug to spot."}]},{"t":"p","runs":[{"text":"When you must mix them, loading an adapter, say, check that "},{"text":"len(tokenizer)","code":true},{"text":" equals "},{"text":"model.config.vocab_size","code":true},{"text":" and stop if it does not."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Load "},{"text":"distilbert-base-uncased","code":true},{"text":" with "},{"text":"AutoModelForSequenceClassification","code":true},{"text":" and read the warning in full. Then run it on ten sentences and look at the predictions. That is what a randomly initialised head looks like from the outside: confident, varied, and meaningless."}]}],"check":{"question":[{"text":"Someone loads "},{"text":"bert-base-uncased","code":true},{"text":" with "},{"text":"AutoModelForSequenceClassification","code":true},{"text":", sees a warning, and gets about 50% accuracy on a balanced two-class task. What happened?"}],"options":[{"text":[{"text":"The base checkpoint was pretrained on masked language modelling, so its representations are unsuitable for classification without a different architecture."}],"correct":false,"tempting":[{"text":"Blames the pretraining objective; those representations are exactly what fine-tuning uses successfully."}]},{"text":[{"text":"The tokenizer and the model came from the same repository but the label mapping defaulted to LABEL_0 and LABEL_1, inverting the classes."}],"correct":false,"tempting":[{"text":"An inverted mapping would produce accuracy near zero rather than near chance, and would not explain the warning."}]},{"text":[{"text":"The classification head does not exist in the checkpoint, so it was created with random weights and never trained."}],"correct":true,"tempting":null},{"text":[{"text":"Sequence classification needs "},{"text":"AutoModelForMaskedLM","code":true},{"text":" for BERT-family models, and the wrong head silently returns pooled embeddings instead of logits."}],"correct":false,"tempting":[{"text":"Mismatches head to task; masked LM predicts vocabulary tokens, not class labels."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"your-first-pipeline","title":"One line that hides five steps","minutes":8},"next":{"slug":"tokenizers-in-practice","title":"Your text is integers, and the count is not the word count","minutes":10}}}