Full fine-tuning, when it is genuinely right

Four cases where LoRA is not enough

Parameter-efficient methods are the default, not the rule. Four situations reliably want all the weights.

1. A new language or script. If the base model saw very little Marathi, or almost no Tamil script, the adaptation required is not a reweighting of existing representations: the representations are absent. Tokenisation is usually bad too (the next module covers why), and fixing it means resizing the embedding table, which is not something an adapter does. This case typically wants continued pretraining on raw text with everything unfrozen.

2. A large dataset. Above roughly 100,000 examples, the low-rank constraint becomes a ceiling rather than a helpful regulariser. There is measured evidence that LoRA both learns less and forgets less than full fine-tuning, which is a good trade when you have 3,000 examples and a bad one when you have 300,000 and actually want the model to change.

3. Small models. Full fine-tuning a 0.5B or 1.5B model is cheap: 1.5B at 16 bytes per parameter is 24 GB of state, which fits one 40 GB card, and with 8-bit AdamW it fits a 24 GB one. If full fine-tuning is affordable and better, the reason to use LoRA has evaporated.

4. Distillation into a small student. Related to the last two: you have a large synthetic dataset and a small model whose behaviour you want to change substantially. This is the archetypal full-fine-tuning job.

The memory, computed

Mixed-precision full fine-tuning with AdamW costs about 16 bytes per parameter of persistent state:

fp32 master weights   4
fp32 gradients        4
AdamW m and v         8
                     --
                     16 bytes/param

For 7B: 112 GB, before a single activation. Add activations and a logits tensor and you are near 140 GB. That is two 80 GB cards, minimum.

With 8-bit AdamW the state term drops from 8 to 2, giving 10 bytes per parameter: 70 GB for 7B, which starts to be one card plus care.

Sharding: ZeRO and FSDP

When the state does not fit one card, you split it across several. The standard scheme has three stages, and they are cumulative:

• Stage 1 shards the optimiser state across devices. Removes the 8-byte term from each card. Cheap, almost no communication cost.
• Stage 2 additionally shards gradients. Removes another 4 bytes per parameter per card.
• Stage 3 additionally shards the parameters themselves. Each card holds a slice, and layers are gathered just in time for the forward pass and released afterwards. Maximum memory saving, most communication: this is the one that needs fast interconnect to be efficient.

DeepSpeed calls these ZeRO stages; PyTorch’s native FSDP implements the same idea and is what accelerate will configure for you by default. They are equivalent in intent; pick whichever your stack supports and do not agonise.

CPU offload goes further: optimiser state, and optionally parameters, live in system RAM and are moved to the GPU as needed. This lets a 7B full fine-tune run on a single 24 GB card with 128 GB of system RAM. It is also roughly five to ten times slower, because you are moving tens of gigabytes across PCIe every step. It is a way to make something possible, not a way to make it practical.

A configuration that works

# accelerate config for FSDP full fine-tuning
compute_environment: LOCAL_MACHINE
distributed_type: FSDP
mixed_precision: bf16
num_processes: 8
fsdp_config:
  fsdp_sharding_strategy: FULL_SHARD      # equivalent to ZeRO-3
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer
  fsdp_state_dict_type: SHARDED_STATE_DICT

The fsdp_transformer_layer_cls_to_wrap line is the one that goes wrong. Get the class name wrong and FSDP wraps the whole model as a single unit, which defeats the sharding entirely and produces an out-of-memory error that looks like the sharding is not working, because it is not.

The thing to know before you commit

Full fine-tuning forgets. That is not a flaw to be engineered away; it is the direct consequence of every weight being free to move towards your objective, including the weights that encoded something else. The mitigations are a replay mixture in the data, a lower learning rate, fewer epochs, and a canary set evaluated at every checkpoint.

And the decision is reversible in one direction only. A LoRA that disappoints costs you a day. A full fine-tune that disappoints costs you the run, the cards, and a model you now have to store and serve at full size. Run the LoRA first, even when you are fairly sure you will need the full thing: it tells you whether the data is any good, which is the question that actually decides the project.