LoRA and Beyond:
Five LLM Fine-Tuning Techniques, Explained

Why fine-tuning an LLM is different

Classic fine-tuning updates every parameter. For an LLM that means holding much more than the model itself in GPU memory. A common rule of thumb for mixed-precision training with the Adam optimizer is about 16 bytes per parameter: 2 for the weights, 2 for the gradients, 4 for a full-precision master copy, and 8 for Adam's two running statistics. Activations come on top of that.

Storage is a second problem. Every fully fine-tuned variant is a complete copy of the model. Ten customers with ten custom models means ten copies of 140GB.

The one idea you need: rank

An LLM's weights are matrices. A single attention projection in a 7B-class model is roughly 4,096 by 4,096, about 16.8 million numbers. The rank of a matrix is how many independent directions it really contains. A matrix of low rank can be written exactly as the product of two thin matrices.

The insight behind LoRA, building on earlier research into the "intrinsic dimension" of fine-tuning, is that the change fine-tuning makes to a weight matrix has low rank, even when the matrix itself does not. So instead of learning a 4,096 by 4,096 update, you learn a 4,096 by 8 matrix and an 8 by 4,096 matrix and multiply them. That is 65,536 numbers instead of 16.8 million, 0.4% of the original.

1. LoRA: learn the change, not the weights

LoRA (Low-Rank Adaptation; Hu et al., Microsoft, 2021) places the pair B and A beside a frozen weight matrix. A starts random and B starts at zero, so on step one the model behaves exactly like the base model. Training adjusts only B and A, and the output is scaled by a constant α/r so you can change the rank without re-tuning the learning rate.

On GPT-3 175B, the paper reports 10,000 times fewer trainable parameters and 3 times less GPU memory than full fine-tuning, with quality on par or better. A checkpoint shrinks from 350GB to about 35MB. After training you can merge BA into W, so inference costs exactly the same as the base model. Or you can keep adapters separate and swap them per request.

The catch: freezing W saves gradients and optimizer state, but activations for the LoRA path still have to be stored for backpropagation. On long sequences, that becomes the memory bottleneck.

2. LoRA-FA: freeze A, train only B

LoRA-FA (Frozen-A; Zhang et al., 2023) targets that activation cost. To compute the gradient for A, LoRA must keep the full layer input, d numbers per token. If A never changes, only B needs a gradient, and B's gradient needs just A's output, r numbers per token. With r at 8 and d at 4,096, that is a big difference.

Training half the matrices sounds like a big sacrifice, but a random projection keeps most of the information, and B can learn to use it. The paper reports quality comparable to LoRA and full fine-tuning on benchmarks from GLUE to GSM8K and HumanEval, with significantly lower activation memory and compute. Use it when long sequences or big batches make activation memory the limit.

3. VeRA: share the matrices, train only vectors

VeRA (Vector-based Random Matrix Adaptation; Kopiczko et al., 2023) goes further. In LoRA, every adapted layer has its own B and A. VeRA creates one random pair, freezes it, and shares it across all layers. Each layer then learns two vectors: d scales the r inner dimensions and b scales the output. The update becomes Λ_b B Λ_d A, where Λ means a diagonal matrix built from a vector.

Because the random matrices can be regenerated from a seed, they do not even need to be saved. The paper reports about 10 times fewer trainable parameters than LoRA at comparable quality. For a 4,096 by 4,096 layer that is a few thousand numbers per layer. That matters when you store or serve thousands of personalized adapters. Since the matrices are random rather than learned, VeRA usually needs a higher rank than LoRA to match it.

4. Delta-LoRA: let the frozen weights move

Every technique so far leaves W untouched, which caps what a low-rank update can express. Delta-LoRA (Zi et al., 2023) updates W too, without paying for W's gradients. After each step it computes how much the product BA changed, Δ(BA) = B_t+1A_t+1 − B_tA_t, and adds a scaled copy of that difference to W.

The insight is that this difference already points in the direction the full gradient would push W. So W gets a useful update with no extra memory for gradients or optimizer state. The paper reports results ahead of LoRA and other low-rank methods at memory and compute costs comparable to LoRA. One practical point, our inference rather than the paper's: because W changes, it has to be kept at a precision you can write to, which does not fit naturally with a 4-bit QLoRA base.

5. LoRA+: train B faster than A

LoRA+ (Hayou, Ghosh and Yu, 2024) changes nothing in the architecture. It changes one hyperparameter. Standard LoRA trains B and A with the same learning rate. The authors show that for wide models this is mathematically inefficient, because the two matrices play different roles. B should learn much faster.

The fix is a single ratio: give B a learning rate λ times A's, with 16 a common default. The paper reports 1 to 2 percent better results and up to about 2 times faster fine-tuning at the same compute. It costs nothing extra and combines with almost every other technique here, so it is the easiest win on the list.

Two more worth knowing

Others you will meet in libraries: rsLoRA scales by α/√r so higher ranks keep learning, while PiSSA and OLoRA start B and A from a decomposition of W instead of random noise, which often converges faster.

Side by side

Trainable parameters for a single 4,096 by 4,096 layer at rank 8, compared with full fine-tuning:

VeRA's per-layer count excludes the shared random matrices, which are frozen and can be regenerated from a seed. In practice VeRA is run at a higher rank than LoRA, which adds only a few hundred numbers per layer.

Try it

Hugging Face's PEFT library implements most of these. A LoRA setup that adapts every linear layer:

from peft import LoraConfig, get_peft_model

config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                    target_modules="all-linear")
model = get_peft_model(base_model, config)
model.print_trainable_parameters()

From there, each variant is a small change. Set use_dora=True for DoRA or use_rslora=True for rsLoRA. Build the optimizer with create_loraplus_optimizer for LoRA+ or create_lorafa_optimizer for LoRA-FA. Use VeraConfig in place of LoraConfig for VeRA. For QLoRA, load the base model in 4-bit with bitsandbytes first.

Before you fine-tune at all

These techniques make fine-tuning cheap. They do not make it the right tool. Fine-tuning is good at teaching a model a format, a style, a domain vocabulary or a narrow skill. It is poor at teaching it facts that change, which belong in retrieval. Better prompting closes more of the gap than most teams expect.

Whatever you choose, build the evaluation set first: a few hundred real examples with a clear way to score them. Without it you cannot tell whether LoRA, DoRA or a better prompt actually helped, and every technique above is guesswork.

Our read

Sources: Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021); Zhang et al., LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning (2023, revised); Kopiczko, Blankevoort and Asano, VeRA: Vector-based Random Matrix Adaptation (2023); Zi et al., Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices (2023); Hayou, Ghosh and Yu, LoRA+: Efficient Low Rank Adaptation of Large Models (2024); Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023); Liu et al., DoRA: Weight-Decomposed Low-Rank Adaptation (2024); Hugging Face PEFT LoRA guide. Memory figures are rule-of-thumb estimates; actual use depends on sequence length, batch size and framework.