Parameter-efficient fine-tuning is a family of methods that freeze a pretrained model and train a small number of new weights alongside it. LoRA is the member everyone uses, and the reason it beat the others is a structural property rather than a quality one: its update can be added into the frozen weights and disappear.
The problem the family solves
Full fine-tuning of a 7B model updates all 7 billion parameters. In 16-bit that is 14 GB of weights, 14 GB of gradients and, with an Adam optimiser keeping two moments per parameter in 32-bit, another 56 GB of optimiser state — before activations. And the output is a new 14 GB checkpoint per task.
Ten tasks means ten checkpoints, ten deployments and ten copies of the same base model in memory. Every method here exists to make the per-task artefact small and the base model shared.
Bottleneck adapters, and their cost
