{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"full-fine-tuning-when-right","title":"Full fine-tuning, when it is genuinely right","url":"https://addaly.com/learn/fine-tuning/full-fine-tuning-when-right","position":26,"of":84,"minutes":9,"answer":"Full fine-tuning is right for a new language, a very large dataset, a small model or a distillation student, costs about 16 bytes per parameter in persistent state before activations.","keyIdea":"Full fine-tuning is right for a new language, a very large dataset, a small model or a distillation student, costs about 16 bytes per parameter in persistent state before activations, and forgets precisely because every weight is free to move, so run the LoRA first to find out whether the data is worth it."},"body":{"format":"markdown","text":"## Four cases where LoRA is not enough\n\nParameter-efficient methods are the default, not the rule. Four situations reliably want all the weights.\n\n**1. A new language or script.** If the base model saw very little Marathi, or almost no Tamil script, the adaptation required is not a reweighting of existing representations: the representations are absent. Tokenisation is usually bad too (the next module covers why), and fixing it means resizing the embedding table, which is not something an adapter does. This case typically wants continued pretraining on raw text with everything unfrozen.\n\n**2. A large dataset.** Above roughly 100,000 examples, the low-rank constraint becomes a ceiling rather than a helpful regulariser. There is measured evidence that LoRA both learns less and forgets less than full fine-tuning, which is a good trade when you have 3,000 examples and a bad one when you have 300,000 and actually want the model to change.\n\n**3. Small models.** Full fine-tuning a 0.5B or 1.5B model is cheap: 1.5B at 16 bytes per parameter is 24 GB of state, which fits one 40 GB card, and with 8-bit AdamW it fits a 24 GB one. If full fine-tuning is affordable and better, the reason to use LoRA has evaporated.\n\n**4. Distillation into a small student.** Related to the last two: you have a large synthetic dataset and a small model whose behaviour you want to change substantially. This is the archetypal full-fine-tuning job.\n\n## The memory, computed\n\nMixed-precision full fine-tuning with AdamW costs about **16 bytes per parameter** of persistent state:\n\n```\nfp32 master weights   4\nfp32 gradients        4\nAdamW m and v         8\n                     --\n                     16 bytes/param\n```\n\nFor 7B: 112 GB, before a single activation. Add activations and a logits tensor and you are near 140 GB. That is two 80 GB cards, minimum.\n\nWith 8-bit AdamW the state term drops from 8 to 2, giving 10 bytes per parameter: 70 GB for 7B, which starts to be one card plus care.\n\n## Sharding: ZeRO and FSDP\n\nWhen the state does not fit one card, you split it across several. The standard scheme has three stages, and they are cumulative:\n\n- **Stage 1** shards the optimiser state across devices. Removes the 8-byte term from each card. Cheap, almost no communication cost.\n- **Stage 2** additionally shards gradients. Removes another 4 bytes per parameter per card.\n- **Stage 3** additionally shards the parameters themselves. Each card holds a slice, and layers are gathered just in time for the forward pass and released afterwards. Maximum memory saving, most communication: this is the one that needs fast interconnect to be efficient.\n\nDeepSpeed calls these ZeRO stages; PyTorch's native FSDP implements the same idea and is what `accelerate` will configure for you by default. They are equivalent in intent; pick whichever your stack supports and do not agonise.\n\n**CPU offload** goes further: optimiser state, and optionally parameters, live in system RAM and are moved to the GPU as needed. This lets a 7B full fine-tune run on a single 24 GB card with 128 GB of system RAM. It is also roughly five to ten times slower, because you are moving tens of gigabytes across PCIe every step. It is a way to make something possible, not a way to make it practical.\n\n## A configuration that works\n\n```yaml\n# accelerate config for FSDP full fine-tuning\ncompute_environment: LOCAL_MACHINE\ndistributed_type: FSDP\nmixed_precision: bf16\nnum_processes: 8\nfsdp_config:\n  fsdp_sharding_strategy: FULL_SHARD      # equivalent to ZeRO-3\n  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP\n  fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer\n  fsdp_state_dict_type: SHARDED_STATE_DICT\n```\n\nThe `fsdp_transformer_layer_cls_to_wrap` line is the one that goes wrong. Get the class name wrong and FSDP wraps the whole model as a single unit, which defeats the sharding entirely and produces an out-of-memory error that looks like the sharding is not working, because it is not.\n\n## The thing to know before you commit\n\nFull fine-tuning forgets. That is not a flaw to be engineered away; it is the direct consequence of every weight being free to move towards your objective, including the weights that encoded something else. The mitigations are a replay mixture in the data, a lower learning rate, fewer epochs, and a canary set evaluated at every checkpoint.\n\nAnd the decision is reversible in one direction only. A LoRA that disappoints costs you a day. A full fine-tune that disappoints costs you the run, the cards, and a model you now have to store and serve at full size. Run the LoRA first, even when you are fairly sure you will need the full thing: it tells you whether the data is any good, which is the question that actually decides the project."},"check":{"question":"A team configures FSDP with FULL_SHARD across 8 GPUs for a 7B full fine-tune and still runs out of memory on every card. The transformer layer class name in the config is misspelled. Why does that single line cause the failure?","options":[{"text":"An unmatched class name disables mixed precision, so weights, gradients and optimiser state all revert to fp32.","correct":false,"tempting":"Couples two independent settings; the precision configuration is applied regardless of which modules the wrap policy matches."},{"text":"FSDP falls back to stage 1 sharding when the class is unknown, which leaves gradients and parameters replicated on every device.","correct":false,"tempting":"Invents a graceful degradation path; the failure is that no wrapping boundary is found at all, not a downgrade to a defined lesser stage."},{"text":"The policy matches no layer, so the model is wrapped as one unit and each card holds the whole state.","correct":true},{"text":"The misspelling prevents gradient checkpointing from attaching to the decoder layers, so activations for all 32 layers are retained.","correct":false,"tempting":"Points at a real memory consumer, but activations would not account for the failure on a run whose persistent state alone exceeds one card."}]},"prev":{"slug":"the-tiny-methods","title":"The tiny methods, and a gotcha in one of them","url":"https://addaly.com/learn/fine-tuning/the-tiny-methods"},"next":{"slug":"training-on-a-small-budget","title":"Training on a free or nearly free budget","url":"https://addaly.com/learn/fine-tuning/training-on-a-small-budget"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Full fine-tuning, when it is genuinely right', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/full-fine-tuning-when-right (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Full fine-tuning, when it is genuinely right","courseTitle":"Fine-Tuning, and When Not To","module":{"index":3,"slug":"the-peft-family","title":"Full, LoRA and the parameter-efficient family"},"answer":null,"keyIdea":"Full fine-tuning is right for a new language, a very large dataset, a small model or a distillation student, costs about 16 bytes per parameter in persistent state before activations, and forgets precisely because every weight is free to move, so run the LoRA first to find out whether the data is worth it.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"four-cases-where-lora-is-not-enough","runs":[{"text":"Four cases where LoRA is not enough"}]},{"t":"p","runs":[{"text":"Parameter-efficient methods are the default, not the rule. Four situations reliably want all the weights."}]},{"t":"p","runs":[{"text":"1. A new language or script.","b":true},{"text":" If the base model saw very little Marathi, or almost no Tamil script, the adaptation required is not a reweighting of existing representations: the representations are absent. Tokenisation is usually bad too (the next module covers why), and fixing it means resizing the embedding table, which is not something an adapter does. This case typically wants continued pretraining on raw text with everything unfrozen."}]},{"t":"p","runs":[{"text":"2. A large dataset.","b":true},{"text":" Above roughly 100,000 examples, the low-rank constraint becomes a ceiling rather than a helpful regulariser. There is measured evidence that LoRA both learns less and forgets less than full fine-tuning, which is a good trade when you have 3,000 examples and a bad one when you have 300,000 and actually want the model to change."}]},{"t":"p","runs":[{"text":"3. Small models.","b":true},{"text":" Full fine-tuning a 0.5B or 1.5B model is cheap: 1.5B at 16 bytes per parameter is 24 GB of state, which fits one 40 GB card, and with 8-bit AdamW it fits a 24 GB one. If full fine-tuning is affordable and better, the reason to use LoRA has evaporated."}]},{"t":"p","runs":[{"text":"4. Distillation into a small student.","b":true},{"text":" Related to the last two: you have a large synthetic dataset and a small model whose behaviour you want to change substantially. This is the archetypal full-fine-tuning job."}]},{"t":"h2","id":"the-memory-computed","runs":[{"text":"The memory, computed"}]},{"t":"p","runs":[{"text":"Mixed-precision full fine-tuning with AdamW costs about "},{"text":"16 bytes per parameter","b":true},{"text":" of persistent state:"}]},{"t":"code","lang":"","label":"Text","text":"fp32 master weights   4\nfp32 gradients        4\nAdamW m and v         8\n                     --\n                     16 bytes/param"},{"t":"p","runs":[{"text":"For 7B: 112 GB, before a single activation. Add activations and a logits tensor and you are near 140 GB. That is two 80 GB cards, minimum."}]},{"t":"p","runs":[{"text":"With 8-bit AdamW the state term drops from 8 to 2, giving 10 bytes per parameter: 70 GB for 7B, which starts to be one card plus care."}]},{"t":"h2","id":"sharding-zero-and-fsdp","runs":[{"text":"Sharding: ZeRO and FSDP"}]},{"t":"p","runs":[{"text":"When the state does not fit one card, you split it across several. The standard scheme has three stages, and they are cumulative:"}]},{"t":"ul","items":[[{"text":"Stage 1","b":true},{"text":" shards the optimiser state across devices. Removes the 8-byte term from each card. Cheap, almost no communication cost."}],[{"text":"Stage 2","b":true},{"text":" additionally shards gradients. Removes another 4 bytes per parameter per card."}],[{"text":"Stage 3","b":true},{"text":" additionally shards the parameters themselves. Each card holds a slice, and layers are gathered just in time for the forward pass and released afterwards. Maximum memory saving, most communication: this is the one that needs fast interconnect to be efficient."}]]},{"t":"p","runs":[{"text":"DeepSpeed calls these ZeRO stages; PyTorch’s native FSDP implements the same idea and is what "},{"text":"accelerate","code":true},{"text":" will configure for you by default. They are equivalent in intent; pick whichever your stack supports and do not agonise."}]},{"t":"p","runs":[{"text":"CPU offload","b":true},{"text":" goes further: optimiser state, and optionally parameters, live in system RAM and are moved to the GPU as needed. This lets a 7B full fine-tune run on a single 24 GB card with 128 GB of system RAM. It is also roughly five to ten times slower, because you are moving tens of gigabytes across PCIe every step. It is a way to make something possible, not a way to make it practical."}]},{"t":"h2","id":"a-configuration-that-works","runs":[{"text":"A configuration that works"}]},{"t":"code","lang":"yaml","label":"YAML","text":"# accelerate config for FSDP full fine-tuning\ncompute_environment: LOCAL_MACHINE\ndistributed_type: FSDP\nmixed_precision: bf16\nnum_processes: 8\nfsdp_config:\n  fsdp_sharding_strategy: FULL_SHARD      # equivalent to ZeRO-3\n  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP\n  fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer\n  fsdp_state_dict_type: SHARDED_STATE_DICT"},{"t":"p","runs":[{"text":"The "},{"text":"fsdp_transformer_layer_cls_to_wrap","code":true},{"text":" line is the one that goes wrong. Get the class name wrong and FSDP wraps the whole model as a single unit, which defeats the sharding entirely and produces an out-of-memory error that looks like the sharding is not working, because it is not."}]},{"t":"h2","id":"the-thing-to-know-before-you-commit","runs":[{"text":"The thing to know before you commit"}]},{"t":"p","runs":[{"text":"Full fine-tuning forgets. That is not a flaw to be engineered away; it is the direct consequence of every weight being free to move towards your objective, including the weights that encoded something else. The mitigations are a replay mixture in the data, a lower learning rate, fewer epochs, and a canary set evaluated at every checkpoint."}]},{"t":"p","runs":[{"text":"And the decision is reversible in one direction only. A LoRA that disappoints costs you a day. A full fine-tune that disappoints costs you the run, the cards, and a model you now have to store and serve at full size. Run the LoRA first, even when you are fairly sure you will need the full thing: it tells you whether the data is any good, which is the question that actually decides the project."}]}],"check":{"question":[{"text":"A team configures FSDP with FULL_SHARD across 8 GPUs for a 7B full fine-tune and still runs out of memory on every card. The transformer layer class name in the config is misspelled. Why does that single line cause the failure?"}],"options":[{"text":[{"text":"An unmatched class name disables mixed precision, so weights, gradients and optimiser state all revert to fp32."}],"correct":false,"tempting":[{"text":"Couples two independent settings; the precision configuration is applied regardless of which modules the wrap policy matches."}]},{"text":[{"text":"FSDP falls back to stage 1 sharding when the class is unknown, which leaves gradients and parameters replicated on every device."}],"correct":false,"tempting":[{"text":"Invents a graceful degradation path; the failure is that no wrapping boundary is found at all, not a downgrade to a defined lesser stage."}]},{"text":[{"text":"The policy matches no layer, so the model is wrapped as one unit and each card holds the whole state."}],"correct":true,"tempting":null},{"text":[{"text":"The misspelling prevents gradient checkpointing from attaching to the decoder layers, so activations for all 32 layers are retained."}],"correct":false,"tempting":[{"text":"Points at a real memory consumer, but activations would not account for the failure on a run whose persistent state alone exceeds one card."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"the-tiny-methods","title":"The tiny methods, and a gotcha in one of them","minutes":8},"next":{"slug":"training-on-a-small-budget","title":"Training on a free or nearly free budget","minutes":10}}}