Low-Rank Adaptation (LoRA)

Definition

Low-Rank Adaptation (LoRA) fine-tunes a large model without changing a single one of its original weights. Every pre-trained matrix stays frozen, and beside each one you attach two thin matrices whose product represents the change you would otherwise have made. Only those two get trained. On GPT-3's query projection — a 12,288 x 12,288 matrix holding 150,994,944 numbers — a rank-4 LoRA trains 98,304 of them instead, a factor of 1,536. Everything else on this page is a consequence of that one substitution.

Three consequences are what turned it into an industry default. The frozen weights receive no gradient, so the optimizer's bookkeeping — the copies of every parameter that actually fill a GPU during training, not the model itself — collapses to almost nothing. The learned change has exactly the same shape as the matrix it adapts, so you can add it into the base weights before you serve and run at the speed of an ordinary model: Hu et al. (2021) write that doing so "guarantees that we do not introduce any additional latency during inference compared to a fine-tuned model by construction." And the artifact you keep is tens of megabytes rather than hundreds of gigabytes, so one copy of a base model in GPU memory can serve a hundred different fine-tunes at once.

LoRA is one member of a family called parameter-efficient fine-tuning (PEFT) — the general idea of freezing a pre-trained model and training a small number of new parameters beside it. Houlsby-style adapter modules and prefix tuning belong to the same family and get the same memory benefit, because the benefit comes from the freezing rather than from the shape of the trainable part. LoRA won on a detail the others do not have: its trainable part can be folded back into the frozen weights, so it is free at inference where a sequential adapter module is not. QLoRA is LoRA with the frozen base additionally held in 4-bit quantization, which is what you reach for when even the read-only copy of the model will not fit.

Getting this wrong costs real money in both directions. Reach for full fine-tuning where LoRA would have done and you rent an order of magnitude more GPU than the job needed. Reach for LoRA on a task whose update genuinely is not low-rank and you ship a model that quietly underperforms — with a training loss curve that looks perfectly healthy the entire time, because the loss is defined only over the examples you gave it.

How It Works

Take any weight matrix W0 of shape d x k. Ordinary fine-tuning computes a new matrix W0 + ΔW where ΔW is dense and unconstrained, which means it has d·k free numbers in it. LoRA constrains ΔW to be the product of two much thinner matrices — B of shape d x r and A of shape r x k, with r far smaller than either dimension — and trains only those:

W0 + ΔW = W0 + B·A        A: r x k,  B: d x r,  r ≪ min(d, k)

The parameter count falls from d·k to r(d + k). For a square matrix that ratio simplifies to d / 2r, which is worth committing to memory: on GPT-3's 12,288-wide projections, rank 4 is a 1,536x reduction and rank 64 is still 96x. The rank is the only knob in that formula, and it is unusually cheap — doubling it doubles the cost of something that was already a rounding error.

What the rank actually controls is how many independent directions the adapter may move the model in. A rank-4 update can shift the layer's behaviour along four directions and no more; anything the task needs that is not expressible as a combination of four directions cannot be learned there. The whole method rests on the hypothesis that this is enough. Hu et al. borrowed the idea from work showing that over-parameterised models "reside on a low intrinsic dimension", and hypothesised "the updates to the weights also have a low 'intrinsic rank' during adaptation."

Their own rank sweep on GPT-3 is the evidence, and it is more extreme than the hypothesis needed. Adapting the query and value projections, validation accuracy on WikiSQL was 73.4 at rank 1, 73.3 at rank 2, 73.7 at rank 4, 73.8 at rank 8 and 73.5 at rank 64 — a spread of 0.5 points across a 64-fold change in capacity, inside the +/-0.5% fluctuation the paper reports for that benchmark. The authors call it out themselves: "To our surprise, a rank as small as one suffices for adapting both Wq and Wv on these datasets." Rank is a real capacity limit, but on the tasks people actually fine-tune for, it is almost never the limit that binds.

Two initialisation details make the method behave. A starts from a random Gaussian and B starts at exactly zero, so B·A is the zero matrix at step one and the adapted model is bit-for-bit the base model before any training happens — there is no random perturbation to recover from. The update is then scaled by α/r, where α is a constant you fix once, so that changing the rank does not silently change the effective learning rate; the paper's advice is to "set α to the first r we try and do not tune it."

The memory saving is in the optimizer, not the weights

This is the part most explanations skip, and it is where the actual saving lives. Training with Adam in mixed precision, every trainable parameter needs five resident copies of itself: a bf16 weight (2 bytes), its bf16 gradient (2), an fp32 master copy (4), and Adam's two moment estimates in fp32 (4 + 4). That is 16 bytes per trainable parameter before a single activation is stored. A frozen parameter needs one copy — the 2-byte weight — because there is no gradient to hold and no moment to accumulate. So freezing removes 14 of those 16 bytes, or 87.5% of the per-parameter cost, and the total becomes:

2 bytes x (all parameters)  +  16 bytes x (adapter parameters only)

On GPT-3 175B with a rank-4 adapter on the query and value projections across all 96 layers, that is 350.5 GB of read-only base weights plus 302 MB of adapter and optimizer state — against 2,804 GB if you applied 16 bytes per parameter to the whole model. The adapter's optimizer state alone is 264 MB, which is the number worth staring at: the entire apparatus that makes training expensive has been reduced to something you could email.

Two honest caveats. First, the frozen weights still have to sit in memory, so the memory ratio is bounded at 16/2 = 8x no matter how small the adapter gets — the parameter ratio can be four orders of magnitude while the memory ratio is one. Second, real trainers are not the textbook. The paper's own measurement is 1.2 TB down to 350 GB, a factor of 3.4, and 1.2 TB across 175,255.8 million parameters works out at about 6.8 bytes per parameter — so their full-fine-tuning baseline was already using fewer resident copies than the naive recipe. The mechanism is identical; the exact ratio depends on your optimizer configuration, and quoting anyone's headline factor as if it were a law of nature will mislead you.

Merging, and the trade it forces

At deployment you compute W = W0 + B·A once and store the result. The served model is an ordinary model of exactly the original shape, running the original number of matrix multiplications — this is the "no additional inference latency by construction" property, and it is the structural reason LoRA displaced sequential adapter modules, which sit in the forward pass and must be executed on every token.

Merging is reversible: subtract B·A to recover W0 exactly, then add a different B'A' for another task. But a merged model is one model. The moment you want a hundred customers' adapters served from one GPU, you must leave the adapters unmerged and pay a small amount of extra compute per token, and the paper names this trade as its own limitation — "it is not straightforward to batch inputs to different tasks with different A and B in a single forward pass, if one chooses to absorb A and B into W."

QLoRA: quantizing the floor

Because the frozen base dominates the memory bill, the next lever is to shrink the base itself. QLoRA (Dettmers et al., 2023) stores it in a 4-bit data type called NormalFloat and dequantizes each block back to bf16 only when it is needed for a matrix multiply, backpropagating through that dequantization into ordinary bf16 LoRA adapters. The reported effect: fine-tuning a 65B model goes "from >780GB of GPU memory to <48GB without degrading the runtime or predictive performance compared to a 16-bit fully finetuned baseline."

The second trick is a piece of arithmetic worth following, because it shows how little slack is left at this level. 4-bit quantization needs a scaling constant per block of weights, and small blocks are what keep it accurate. With 32-bit constants and a block size of 64, those constants cost 32/64 = 0.5 bits per parameter — 12.5% overhead on top of the 4 bits you were trying to spend. QLoRA quantizes the constants too, to 8-bit floats in blocks of 256, giving 8/64 + 32/(64 x 256) = 0.127 bits per parameter: a saving of 0.373 bits per parameter, which the paper puts at approximately 3 GB on a 65B model. The third component, paged optimizers, uses NVIDIA unified memory so that a long-sequence mini-batch spikes into CPU RAM instead of crashing the run.

Real-World Applications

Guanaco is the demonstration that changed expectations. In the QLoRA paper, Dettmers et al. fine-tuned LLaMA models with 4-bit bases and LoRA adapters and reported the largest reaching "99.3% of the performance level of ChatGPT" on the Vicuna benchmark after 24 hours on a single professional GPU, with the 33B version trainable "in less than 12 hours on a single consumer GPU". Those are 2023 numbers against a 2023 ChatGPT on a benchmark that has since been superseded, and they should be read as an accessibility result rather than a quality ranking — the durable claim is that a model of that size became tunable by one person with one GPU, which had not been true before.

Image models are where the adapter ecosystem is most visible. Because Stable Diffusion's weights are downloadable, communities have built enormous libraries of LoRAs that each teach one shared base model a particular style, character or subject. Each is a few tens of megabytes, each is trained in minutes to hours, and users stack several at inference. The image generation page covers that ecosystem; the reason it can exist at all is the arithmetic above — a style is a small perturbation, and small perturbations are exactly what a low-rank update is good at.

Multi-adapter serving is the property with the largest commercial consequence. The LoRA paper does the sum in a footnote: storing 100 adapted models needs "350GB + 35MB * 100 ≈ 354GB as opposed to 100 * 350GB ≈ 35TB". That is the difference between per-customer fine-tuning being a product and being a rounding error on the pitch deck. Inference servers such as vLLM now load a base model once and attach many LoRA adapters, routing each request to the one it asked for, and it is the reason hosted "fine-tune your own model" offerings can price a private model like an API call rather than like a reserved GPU.

Key Concepts

• Rank r — how many independent directions the adapter may move the layer in, and the only term in r(d + k). It is the capacity ceiling, but on typical adaptation tasks it is rarely the binding constraint.
• α/r scaling — a fixed constant divided by the rank, applied to the adapter's output so that changing r does not change the effective step size. Set α once, do not sweep it alongside r.
• Target modules — which matrices get an adapter. On GPT-3 the paper adapted attention projections only and froze the MLP blocks; QLoRA found that reaching full fine-tuning quality on large bases needed "LoRA on all linear transformer block layers", which is roughly six times the trainable parameters.
• Merging — folding B·A into W0 before serving. Buys zero added latency, spends the ability to swap adapters per request.
• Base-model pinning — an adapter is a delta fitted to one exact checkpoint. It does not transfer to a newer base, and for QLoRA it was fitted to the quantized weights, so the serving stack has to quantize the same way.

Challenges

LoRA learns less than full fine-tuning, and the gap is measurable. Biderman et al. (2024), published in TMLR, compared the two on programming and mathematics across both instruction fine-tuning and continued pretraining, and found that "in the standard low-rank settings, LoRA substantially underperforms full finetuning." The mechanism they propose is the assumption itself: they measured full fine-tuning learning "perturbations with a rank that is 10-100x greater than typical LoRA configurations". The same paper is the strongest argument for LoRA on a different axis — it "better maintains the base model's performance on tasks outside the target domain", which is the catastrophic forgetting trade in one sentence. A low-rank update cannot learn everything, and that is also why it cannot destroy everything.

Where you put the adapters matters more than how big you make them. At a fixed budget of 18 million trainable parameters on GPT-3, the LoRA paper compared configurations that cost exactly the same: rank 8 on the query matrix alone scored 70.4 on WikiSQL, rank 4 on query and value scored 73.7, and rank 2 spread across all four attention matrices also scored 73.7. Concentrating the identical budget in one matrix cost 3.3 points. The intuition to unlearn is that rank is the quality dial — QLoRA states outright that "other LoRA hyperparameters, such as the projection dimension r, do not affect performance", and reports instead that breadth of coverage was the critical choice.

Merging and swapping are mutually exclusive, and teams discover this after they have built for one. A merged adapter is free at inference but is now a whole model; an unmerged one can be hot-swapped per request but adds work in the forward pass and constrains how you batch. Choosing wrong means either a serving fleet that cannot host a second customer without a second GPU, or a latency budget spent on a flexibility nobody uses.

Every adapter you train is pinned to one checkpoint. When the base model is deprecated or a better one ships, the adapter does not migrate — you re-run the data pipeline, the training and the evaluation to stand still. A fleet of a thousand per-customer adapters is a thousand migrations, and this recurring bill is the strongest reason to keep facts in a retrieval index and spend adaptation only on behaviour that is genuinely yours.

QLoRA adds a coupling that plain LoRA does not have. The adapter is trained against the dequantized 4-bit weights, not against the original bf16 ones, so it has partly learned to compensate for the quantization error of that specific scheme. Change the quantizer at serving time and you are adding a correction computed for weights that are no longer there.

Future Trends

Unmerged multi-adapter serving becomes an inference-stack primitive rather than a research demo. The economics — one base in memory, thousands of deltas on disk — only pay off if the runtime can batch requests bound for different adapters into a single forward pass. That is now a first-class feature in serving stacks rather than something teams build themselves, and it is what decides whether per-customer fine-tuning is a product line or a science project.

The variants worth watching attack the quality gap, not the memory. Memory was solved; the measured deficit against full fine-tuning was not. DoRA decomposes each weight into a magnitude and a direction and applies the low-rank update only to the direction; rsLoRA changes the scaling from α/r to something that stays stable as the rank grows, so that high-rank adapters actually benefit from their extra capacity instead of being damped by it. Both are attempts to buy back the expressiveness that the rank constraint spends.

Adapters are becoming composable objects. Because B·A is just a matrix added to the base, adapters can be added, subtracted and blended arithmetically — the same property that makes merging free makes task arithmetic possible. Expect the interesting work to be in routing and composition of many small deltas rather than in shrinking the delta further, because there is very little left to shrink.

Code Example

The arithmetic is small enough to check yourself, and checking it against the paper is worth more than trusting either. The only inputs are GPT-3's layer dimensions and the rank:

# GPT-3 175B, the model every headline number in the LoRA paper is measured on.
D_MODEL, LAYERS = 12288, 96
TOTAL = 175_255_800_000          # "175,255.8M", the full fine-tuning row of Table 4

def adapter_params(r, matrices_per_layer):
    # a rank-r adapter on a d x k matrix is A (r x k) plus B (d x r): r(d + k) numbers
    return LAYERS * matrices_per_layer * r * (D_MODEL + D_MODEL)

print(f"one W_q matrix, left frozen: {D_MODEL * D_MODEL:,} parameters\n")
print(f"{'configuration':<24}{'trainable':>14}{'of model':>11}{'fp16 size':>12}")
for r, n, label in [(1, 2, "r=1   W_q, W_v"),
                    (4, 2, "r=4   W_q, W_v"),
                    (8, 1, "r=8   W_q alone"),
                    (2, 4, "r=2   all four"),
                    (8, 2, "r=8   W_q, W_v"),
                    (64, 2, "r=64  W_q, W_v")]:
    p = adapter_params(r, n)
    print(f"{label:<24}{p:>14,}{p / TOTAL:>11.4%}{p * 2 / 1e6:>9.1f} MB")

# one W_q matrix, left frozen: 150,994,944 parameters
#
# configuration                trainable   of model   fp16 size
# r=1   W_q, W_v               4,718,592    0.0027%      9.4 MB
# r=4   W_q, W_v              18,874,368    0.0108%     37.7 MB
# r=8   W_q alone             18,874,368    0.0108%     37.7 MB
# r=2   all four              18,874,368    0.0108%     37.7 MB
# r=8   W_q, W_v              37,748,736    0.0215%     75.5 MB
# r=64  W_q, W_v             301,989,888    0.1723%    604.0 MB

Three of those rows are identical at 18,874,368 parameters, which is not a coincidence: it is the "parameter budget of 18M" the paper fixes so that its comparison of where to put the adapters is fair, and the reason rank 8 on one matrix and rank 2 on four matrices can be compared at all. The 4,718,592 and 37,748,736 rows reproduce the "4.7M" and "37.7M" LoRA entries in the paper's GPT-3 results table from the layer shapes alone. Now the memory, which is the decision the arithmetic actually drives:

GB, MB = 1e9, 1e6
adapter = 18_874_368            # r=4 on W_q and W_v, the paper's 18M budget

frozen_base = TOTAL * 2 / GB    # bf16, read-only: no gradient, no fp32 master, no moments
print(f"frozen base in bf16          {frozen_base:9.1f} GB")
print(f"adapter weights              {adapter * 2 / MB:9.1f} MB")
print(f"adapter optimizer state      {adapter * 14 / MB:9.1f} MB")   # grad 2 + master 4 + m 4 + v 4
print(f"LoRA total                   {frozen_base + adapter * 16 / GB:9.1f} GB")
print(f"textbook full fine-tune      {TOTAL * 16 / GB:9.1f} GB")
print()
print(f"100 tasks, one shared base   {(TOTAL * 2 + 100 * adapter * 2) / GB:9.1f} GB")
print(f"100 tasks, 100 checkpoints   {(100 * TOTAL * 2) / GB:9.1f} GB")

# frozen base in bf16              350.5 GB
# adapter weights                   37.7 MB
# adapter optimizer state          264.2 MB
# LoRA total                       350.8 GB
# textbook full fine-tune         2804.1 GB
#
# 100 tasks, one shared base       354.3 GB
# 100 tasks, 100 checkpoints     35051.2 GB

The last two lines are the paper's own footnote, recomputed: "350GB + 35MB * 100 ≈ 354GB as opposed to 100 * 350GB ≈ 35TB". The 350.8 against 2,804.1 is the training story, and the 354.3 against 35,051.2 is the serving story — and it is the second one that made LoRA an industry default rather than a memory trick.

Academic Sources

• "LoRA: Low-Rank Adaptation of Large Language Models" — Hu et al. (2021) — the original method, the rank sweep, and the no-added-latency merge.
• "QLoRA: Efficient Finetuning of Quantized LLMs" — Dettmers et al. (2023) — 4-bit NormalFloat, double quantization, paged optimizers, and Guanaco.
• "LoRA Learns Less and Forgets Less" — Biderman et al. (2024), TMLR — the measured quality gap against full fine-tuning, and the forgetting benefit that comes with it.
• "Parameter-Efficient Transfer Learning for NLP" — Houlsby et al. (2019) — adapter modules, the origin of the PEFT family LoRA belongs to.