Soft prompts and prefixes

Training the prompt instead of the model

A prompt is a sequence of token embeddings. Those embeddings are looked up from a table, but nothing in the architecture requires them to correspond to real tokens. You can prepend a handful of learned vectors (not the embedding of any word, just free parameters) and train them with gradient descent while every weight in the model stays frozen.

That is prompt tuning. With 20 virtual tokens on a model with hidden size 4,096, you are training 20 × 4,096 = 81,920 parameters. The artefact is about 320 kilobytes. You could store one per customer in a database column.

Prefix tuning: the same idea, deeper

Prompt tuning only touches the input. Prefix tuning goes further: it prepends learned key and value vectors at every attention layer, so the adaptation is present throughout the network rather than only at the bottom.

This is more expressive and correspondingly larger, still tiny, typically under 1% of the model. During training the prefixes are usually produced by a small MLP rather than optimised directly, because optimising them directly is unstable; after training the MLP is discarded and only the resulting vectors are kept. P-tuning v2 is essentially this method applied to understanding tasks.

from peft import PrefixTuningConfig, PromptTuningConfig, get_peft_model

cfg = PromptTuningConfig(task_type="CAUSAL_LM", num_virtual_tokens=20,
                         prompt_tuning_init="TEXT",
                         prompt_tuning_init_text="Classify the support ticket:")

That prompt_tuning_init="TEXT" initialises the virtual tokens from the embeddings of a real sentence rather than randomly, and it matters more than it sounds: random initialisation of soft prompts converges slowly and sometimes not at all.

The three honest disadvantages

They eat your context. Twenty virtual tokens are twenty tokens of the window, at every request, forever. On a long-context task that is nothing; on a tight latency budget with a 512-token limit it is 4% of your input.

They cannot be merged. A LoRA can be folded into the weights so inference costs exactly what the base model costs. A soft prompt is always present at runtime, and the attention over those extra positions is real work.

They underperform at small scale. This is the substantive one. The original prompt-tuning result was that the method matches full fine-tuning, but only as model size grows past roughly ten billion parameters. Below that there is a real gap, and the gap widens the harder the task. On a 3B model with a difficult task, LoRA will beat a soft prompt, reliably.

The mechanism is intuitive enough: a soft prompt can only steer the computation the model already performs, by choosing what it conditions on. LoRA can change the computation. Big models have more behaviours available to steer towards; small ones need the change.

Where they are the right answer

Two situations, both about the artefact rather than the quality:

Many tenants, one model. If you need per-customer adaptation for a thousand customers, a thousand LoRAs is 160 GB and a thousand soft prompts is 320 MB. When a soft prompt is good enough, the storage and switching story is far simpler: you look up a tensor and concatenate it, with no adapter-swapping machinery in the serving stack.

Frozen-model constraints. Some deployment environments will not let you ship modified weights: a shared inference cluster, a compliance rule, a vendor endpoint. A prefix is data, not a model.

The thing to try before either

Write the prompt in words first. A soft prompt is an optimised version of an instruction, and if you have not yet found a good instruction by hand, you are asking gradient descent to search a space you have not explored yourself. The cases where soft prompting wins are the cases where a good hand-written prompt already gets close and you want the last few points, cheaply and with a very small artefact.

And if a good hand-written prompt already gets close, ask honestly whether the last few points are worth a training pipeline at all. That question is the spine of this whole course, and it applies to the cheapest method as much as to the most expensive.