When the answer is one of six labels

You may not need a generative model at all

A large share of what gets sent to a chat model is classification wearing a costume. Route this ticket to one of six queues. Is this review positive, negative or mixed. Does this sentence contain personal data. Is this transaction suspicious. The output space has six members, and you are paying a 70B decoder to write them out one token at a time.

An encoder model fine-tuned for classification does this better, and the gap is not small.

The arithmetic

Take ticket routing, six queues, 2,000 labelled examples.

• Prompting a hosted frontier model. Roughly 700 input tokens (instructions plus the ticket) and 5 output tokens. At typical 2026 pricing for a mid-tier hosted model that is on the order of $0.20 per thousand tickets, with a p95 latency near one second, and a rate limit that becomes your rate limit.
• Fine-tuned encoder. ModernBERT-base is 149M parameters; DeBERTa-v3-base is 184M. Training on 2,000 examples takes about eight minutes on a free Colab or Kaggle T4. Inference runs at roughly 5 to 15 milliseconds per item on a CPU, batched far faster, with no per-call cost and no vendor.

Accuracy usually favours the encoder too, once you have a thousand labels or more. The encoder sees the whole input bidirectionally and is optimised directly for the decision; the decoder is optimised to continue text and is being asked, indirectly, to make a decision.

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained(
    "answerdotai/ModernBERT-base", num_labels=6)

That num_labels=6 replaces the pretraining head with a fresh linear layer of 768 × 6 weights. Everything below it is the pretrained encoder, and the whole thing is trained: full fine-tuning of a 149M model fits in 6 GB of GPU memory comfortably, so none of the LoRA arithmetic in this course is needed.

What you give up, precisely

This is a real trade, not a free win.

• You need labels. A prompt works on day one with zero examples. An encoder needs several hundred before it is competitive and a thousand or two before it is clearly better. If a label costs a minute of somebody’s attention, 2,000 labels is about 33 hours of work.
• No new classes without retraining. Add a seventh queue and you retrain, cheap, but it is a step. A prompt takes a new class in one line.
• No explanation. The classifier emits a label and a probability. If your users need a reason, you either add a second call to a generative model or you accept a score.
• Brittleness to drift. An encoder trained on last year’s tickets does not know this year’s product names. It will not tell you so; it will just be wrong on the new ones. A frontier model degrades more gracefully because it reads the words.

The hybrid that most teams land on

Use the big model to make the labels, and the small model to serve them. Run a frontier model over 3,000 historical tickets, have a person check a sample of 200 of those labels to get an agreement figure, then train the encoder on all 3,000. You have bought a labelled dataset for a few dollars and a serving model that costs nothing per call. Where the encoder’s confidence is low, say the top probability is under 0.7, fall back to the expensive model for that item only. On real routing workloads that fallback fires on 5 to 15% of traffic.

Check your provider’s terms before doing this. Several model licences restrict using outputs to train models, and what counts as a competing model is not always clear; that is the subject of a later lesson in this module.

The signal that you are in this situation

You are classifying, not generating, if any of these are true: the output is drawn from a fixed list; you evaluate with accuracy, precision or recall rather than with a human reading the text; the prompt contains the phrase “answer with only one of”; or you are post-processing the model’s reply with a regex to extract the label. That regex is the clearest tell there is. It means you built a generative system and then spent effort hiding the generation.

The same logic covers extraction with a fixed schema (token classification, which is a per-token version of the same thing) and scoring (a regression head, one output instead of six). The family is called encoders, the library is transformers, and the whole path is free.