{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"the-tiny-methods","title":"The tiny methods, and a gotcha in one of them","url":"https://addaly.com/learn/fine-tuning/the-tiny-methods","position":25,"of":84,"minutes":8,"answer":"IA³ adapts a model with three learned scaling vectors per layer, about 0.01% of parameters, merging exactly into the weights and suiting tiny datasets.","keyIdea":"IA³ adapts a model with three learned scaling vectors per layer, about 0.01% of parameters, merging exactly into the weights and suiting tiny datasets, while BitFit is nearly a no-op on Llama-style models because they have almost no bias terms to train."},"body":{"format":"markdown","text":"## Below LoRA, there is more\n\nLoRA trains about 0.5% of a model. There are methods that train a hundredth of that and still work on some tasks. They matter less for quality than for what they reveal: how little of a pretrained model needs to move for its behaviour to change.\n\n## IA³: learned scaling vectors\n\nIA³ (Infused Adapter by Inhibiting and Amplifying Inner Activations) introduces three learned vectors per layer, which *multiply* activations element-wise:\n\n- one scaling the attention keys,\n- one scaling the attention values,\n- one scaling the feed-forward intermediate activations.\n\nA vector, not a matrix. For an 8B model that is a few hundred thousand parameters in total, around **0.01%**, and the file is a couple of megabytes.\n\nTwo properties make it more interesting than the size alone suggests. First, it can be merged into the weights exactly, because scaling a row of a matrix is just scaling the matrix, so there is zero inference overhead. Second, it was designed for the few-shot setting and does unusually well with very small datasets, where a LoRA has enough capacity to memorise and IA³ does not.\n\n```python\nfrom peft import IA3Config\ncfg = IA3Config(task_type=\"CAUSAL_LM\",\n                target_modules=[\"k_proj\", \"v_proj\", \"down_proj\"],\n                feedforward_modules=[\"down_proj\"])\n```\n\nThe `feedforward_modules` argument must be a subset of `target_modules`; IA³ applies scaling differently for feed-forward layers, and getting this wrong is the usual configuration error.\n\n## BitFit, and the gotcha\n\nBitFit trains only the bias terms of the model and freezes everything else: about 0.08% of the parameters of a BERT-style model. On classification tasks it was shown to come remarkably close to full fine-tuning, which was a surprising result when it appeared.\n\nHere is the part that catches people out. **Most modern decoder LLMs have almost no biases to train.** Llama-family models use RMSNorm, which has a scale parameter and no bias, and their attention and feed-forward projections are configured without bias terms. Run BitFit on Llama-3-8B and you will find a few thousand trainable parameters (essentially the normalisation scales) and a run that goes almost nowhere.\n\nSo BitFit is a real method for encoder models with biases (BERT, DeBERTa, RoBERTa, where it is worth knowing) and close to a no-op on the architectures most of this course is about. It is a good example of why you should always print the trainable-parameter count rather than trusting that a named method is doing something.\n\n## Norm tuning\n\nTraining only the RMSNorm or LayerNorm scale parameters *is* available on modern decoders: roughly 0.003% of parameters. On its own it is weak. Combined with LoRA it occasionally adds a little, at negligible cost. Worth knowing, not worth a run of its own.\n\n## What these methods are for\n\nBe honest about the use cases, because \"0.01% of parameters\" is a marketing number and not a reason.\n\n1. **Extreme multi-tenancy.** Ten thousand customers, each with a 2 MB artefact, all served from one set of base weights. At that scale IA³'s size is the point.\n2. **Very small datasets.** Fifty to five hundred examples, where LoRA's capacity is a liability. IA³ has almost nothing with which to memorise, so what it learns tends to generalise.\n3. **Understanding.** These methods are a standing demonstration that adapting a pretrained model is a small intervention. That reframes the whole activity: you are not teaching the model, you are adjusting emphasis in something that already knows how to do the task approximately.\n\n## What they are not for\n\nAnything requiring new behaviour. If a full LoRA at rank 32 is not enough for your task, a method with a thousand times fewer parameters is not the answer. The ordering of capacity (IA³ < prompt tuning < LoRA < full fine-tuning) is monotonic, and every step down the list also steps down the ceiling.\n\nThe practical recommendation is dull: start with LoRA. Reach for IA³ only when the artefact size is a constraint, or when your dataset is small enough that LoRA visibly overfits, and verify both claims with numbers rather than assuming them."},"check":{"question":"Someone configures BitFit on Llama-3-8B and the trainable-parameter count prints in the low thousands, with a flat loss curve. What has happened?","options":[{"text":"The target module names do not match the architecture, so no layers were selected for training.","correct":false,"tempting":"Describes the usual cause of a zero count, but BitFit selects by parameter name pattern rather than module list and did select something."},{"text":"Llama-style models omit bias terms and use RMSNorm, so there are almost no biases to train.","correct":true},{"text":"BitFit requires a classification head, and on a causal language model it falls back to training only the output layer's bias.","correct":false,"tempting":"Invents a fallback behaviour; the method has no head requirement and the small count comes from the architecture's design, not a fallback."},{"text":"Quantisation removed the bias terms, since 4-bit formats store only weights and rescale biases into the block constants.","correct":false,"tempting":"Attributes the absence to quantisation, which leaves biases in higher precision and would not explain the result on an unquantised model."}]},"prev":{"slug":"soft-prompts-and-prefixes","title":"Soft prompts and prefixes","url":"https://addaly.com/learn/fine-tuning/soft-prompts-and-prefixes"},"next":{"slug":"full-fine-tuning-when-right","title":"Full fine-tuning, when it is genuinely right","url":"https://addaly.com/learn/fine-tuning/full-fine-tuning-when-right"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The tiny methods, and a gotcha in one of them', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/the-tiny-methods (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The tiny methods, and a gotcha in one of them","courseTitle":"Fine-Tuning, and When Not To","module":{"index":3,"slug":"the-peft-family","title":"Full, LoRA and the parameter-efficient family"},"answer":null,"keyIdea":"IA³ adapts a model with three learned scaling vectors per layer, about 0.01% of parameters, merging exactly into the weights and suiting tiny datasets, while BitFit is nearly a no-op on Llama-style models because they have almost no bias terms to train.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"below-lora-there-is-more","runs":[{"text":"Below LoRA, there is more"}]},{"t":"p","runs":[{"text":"LoRA trains about 0.5% of a model. There are methods that train a hundredth of that and still work on some tasks. They matter less for quality than for what they reveal: how little of a pretrained model needs to move for its behaviour to change."}]},{"t":"h2","id":"ia-learned-scaling-vectors","runs":[{"text":"IA³: learned scaling vectors"}]},{"t":"p","runs":[{"text":"IA³ (Infused Adapter by Inhibiting and Amplifying Inner Activations) introduces three learned vectors per layer, which "},{"text":"multiply","i":true},{"text":" activations element-wise:"}]},{"t":"ul","items":[[{"text":"one scaling the attention keys,"}],[{"text":"one scaling the attention values,"}],[{"text":"one scaling the feed-forward intermediate activations."}]]},{"t":"p","runs":[{"text":"A vector, not a matrix. For an 8B model that is a few hundred thousand parameters in total, around "},{"text":"0.01%","b":true},{"text":", and the file is a couple of megabytes."}]},{"t":"p","runs":[{"text":"Two properties make it more interesting than the size alone suggests. First, it can be merged into the weights exactly, because scaling a row of a matrix is just scaling the matrix, so there is zero inference overhead. Second, it was designed for the few-shot setting and does unusually well with very small datasets, where a LoRA has enough capacity to memorise and IA³ does not."}]},{"t":"code","lang":"python","label":"Python","text":"from peft import IA3Config\ncfg = IA3Config(task_type=\"CAUSAL_LM\",\n                target_modules=[\"k_proj\", \"v_proj\", \"down_proj\"],\n                feedforward_modules=[\"down_proj\"])"},{"t":"p","runs":[{"text":"The "},{"text":"feedforward_modules","code":true},{"text":" argument must be a subset of "},{"text":"target_modules","code":true},{"text":"; IA³ applies scaling differently for feed-forward layers, and getting this wrong is the usual configuration error."}]},{"t":"h2","id":"bitfit-and-the-gotcha","runs":[{"text":"BitFit, and the gotcha"}]},{"t":"p","runs":[{"text":"BitFit trains only the bias terms of the model and freezes everything else: about 0.08% of the parameters of a BERT-style model. On classification tasks it was shown to come remarkably close to full fine-tuning, which was a surprising result when it appeared."}]},{"t":"p","runs":[{"text":"Here is the part that catches people out. "},{"text":"Most modern decoder LLMs have almost no biases to train.","b":true},{"text":" Llama-family models use RMSNorm, which has a scale parameter and no bias, and their attention and feed-forward projections are configured without bias terms. Run BitFit on Llama-3-8B and you will find a few thousand trainable parameters (essentially the normalisation scales) and a run that goes almost nowhere."}]},{"t":"p","runs":[{"text":"So BitFit is a real method for encoder models with biases (BERT, DeBERTa, RoBERTa, where it is worth knowing) and close to a no-op on the architectures most of this course is about. It is a good example of why you should always print the trainable-parameter count rather than trusting that a named method is doing something."}]},{"t":"h2","id":"norm-tuning","runs":[{"text":"Norm tuning"}]},{"t":"p","runs":[{"text":"Training only the RMSNorm or LayerNorm scale parameters "},{"text":"is","i":true},{"text":" available on modern decoders: roughly 0.003% of parameters. On its own it is weak. Combined with LoRA it occasionally adds a little, at negligible cost. Worth knowing, not worth a run of its own."}]},{"t":"h2","id":"what-these-methods-are-for","runs":[{"text":"What these methods are for"}]},{"t":"p","runs":[{"text":"Be honest about the use cases, because “0.01% of parameters” is a marketing number and not a reason."}]},{"t":"ol","start":1,"items":[[{"text":"Extreme multi-tenancy.","b":true},{"text":" Ten thousand customers, each with a 2 MB artefact, all served from one set of base weights. At that scale IA³’s size is the point."}],[{"text":"Very small datasets.","b":true},{"text":" Fifty to five hundred examples, where LoRA’s capacity is a liability. IA³ has almost nothing with which to memorise, so what it learns tends to generalise."}],[{"text":"Understanding.","b":true},{"text":" These methods are a standing demonstration that adapting a pretrained model is a small intervention. That reframes the whole activity: you are not teaching the model, you are adjusting emphasis in something that already knows how to do the task approximately."}]]},{"t":"h2","id":"what-they-are-not-for","runs":[{"text":"What they are not for"}]},{"t":"p","runs":[{"text":"Anything requiring new behaviour. If a full LoRA at rank 32 is not enough for your task, a method with a thousand times fewer parameters is not the answer. The ordering of capacity (IA³ < prompt tuning < LoRA < full fine-tuning) is monotonic, and every step down the list also steps down the ceiling."}]},{"t":"p","runs":[{"text":"The practical recommendation is dull: start with LoRA. Reach for IA³ only when the artefact size is a constraint, or when your dataset is small enough that LoRA visibly overfits, and verify both claims with numbers rather than assuming them."}]}],"check":{"question":[{"text":"Someone configures BitFit on Llama-3-8B and the trainable-parameter count prints in the low thousands, with a flat loss curve. What has happened?"}],"options":[{"text":[{"text":"The target module names do not match the architecture, so no layers were selected for training."}],"correct":false,"tempting":[{"text":"Describes the usual cause of a zero count, but BitFit selects by parameter name pattern rather than module list and did select something."}]},{"text":[{"text":"Llama-style models omit bias terms and use RMSNorm, so there are almost no biases to train."}],"correct":true,"tempting":null},{"text":[{"text":"BitFit requires a classification head, and on a causal language model it falls back to training only the output layer’s bias."}],"correct":false,"tempting":[{"text":"Invents a fallback behaviour; the method has no head requirement and the small count comes from the architecture’s design, not a fallback."}]},{"text":[{"text":"Quantisation removed the bias terms, since 4-bit formats store only weights and rescale biases into the block constants."}],"correct":false,"tempting":[{"text":"Attributes the absence to quantisation, which leaves biases in higher precision and would not explain the result on an unquantised model."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"soft-prompts-and-prefixes","title":"Soft prompts and prefixes","minutes":8},"next":{"slug":"full-fine-tuning-when-right","title":"Full fine-tuning, when it is genuinely right","minutes":9}}}