The tiny methods, and a gotcha in one of them

Below LoRA, there is more

LoRA trains about 0.5% of a model. There are methods that train a hundredth of that and still work on some tasks. They matter less for quality than for what they reveal: how little of a pretrained model needs to move for its behaviour to change.

IA³: learned scaling vectors

IA³ (Infused Adapter by Inhibiting and Amplifying Inner Activations) introduces three learned vectors per layer, which multiply activations element-wise:

• one scaling the attention keys,
• one scaling the attention values,
• one scaling the feed-forward intermediate activations.

A vector, not a matrix. For an 8B model that is a few hundred thousand parameters in total, around 0.01%, and the file is a couple of megabytes.

Two properties make it more interesting than the size alone suggests. First, it can be merged into the weights exactly, because scaling a row of a matrix is just scaling the matrix, so there is zero inference overhead. Second, it was designed for the few-shot setting and does unusually well with very small datasets, where a LoRA has enough capacity to memorise and IA³ does not.

from peft import IA3Config
cfg = IA3Config(task_type="CAUSAL_LM",
                target_modules=["k_proj", "v_proj", "down_proj"],
                feedforward_modules=["down_proj"])

The feedforward_modules argument must be a subset of target_modules; IA³ applies scaling differently for feed-forward layers, and getting this wrong is the usual configuration error.

BitFit, and the gotcha

BitFit trains only the bias terms of the model and freezes everything else: about 0.08% of the parameters of a BERT-style model. On classification tasks it was shown to come remarkably close to full fine-tuning, which was a surprising result when it appeared.

Here is the part that catches people out. Most modern decoder LLMs have almost no biases to train. Llama-family models use RMSNorm, which has a scale parameter and no bias, and their attention and feed-forward projections are configured without bias terms. Run BitFit on Llama-3-8B and you will find a few thousand trainable parameters (essentially the normalisation scales) and a run that goes almost nowhere.

So BitFit is a real method for encoder models with biases (BERT, DeBERTa, RoBERTa, where it is worth knowing) and close to a no-op on the architectures most of this course is about. It is a good example of why you should always print the trainable-parameter count rather than trusting that a named method is doing something.

Norm tuning

Training only the RMSNorm or LayerNorm scale parameters is available on modern decoders: roughly 0.003% of parameters. On its own it is weak. Combined with LoRA it occasionally adds a little, at negligible cost. Worth knowing, not worth a run of its own.

What these methods are for

Be honest about the use cases, because “0.01% of parameters” is a marketing number and not a reason.

1. Extreme multi-tenancy. Ten thousand customers, each with a 2 MB artefact, all served from one set of base weights. At that scale IA³’s size is the point.
2. Very small datasets. Fifty to five hundred examples, where LoRA’s capacity is a liability. IA³ has almost nothing with which to memorise, so what it learns tends to generalise.
3. Understanding. These methods are a standing demonstration that adapting a pretrained model is a small intervention. That reframes the whole activity: you are not teaching the model, you are adjusting emphasis in something that already knows how to do the task approximately.

What they are not for

Anything requiring new behaviour. If a full LoRA at rank 32 is not enough for your task, a method with a thousand times fewer parameters is not the answer. The ordering of capacity (IA³ < prompt tuning < LoRA < full fine-tuning) is monotonic, and every step down the list also steps down the ceiling.

The practical recommendation is dull: start with LoRA. Reach for IA³ only when the artefact size is a constraint, or when your dataset is small enough that LoRA visibly overfits, and verify both claims with numbers rather than assuming them.