Freeze the giant weight matrix; move it by training two tiny ones. The adapted weight is W' = W + (α/r)·B·A, where B is d×r, A is r×k, and the rank r ≪ min(d,k). The update ΔW lives in a rank-r subspace, so it is decided by only r·(d+k) numbers instead of d·k — and at initialization B=0, so W' = W exactly: the fine-tuned model starts identical to the base. Down the center, W and its adapters go in, the engine builds ΔW, the proven counts come out. The blue team builds and defends it; the red team tries to break it.
source Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, Chen, LoRA: Low-Rank Adaptation of Large Language Models (2021) — arxiv.org/abs/2106.09685. Rendered, not quoted.
The base weight W (d×k) is frozen — never touched. All learning happens in two small matrices whose product is the only thing that moves:
A (r×k) projects the input down to r dimensions; B (d×r) projects back up. Their product B·A is d×k but can have rank at most r. Scaled by α/r and added to W, it becomes the whole adaptation.
Live shapes & parameter counts for the toy dims:
| matrix | shape | params | trained? |
|---|
A full weight update is a d×k matrix. LoRA constrains it to a low-rank subspace — it is the-matrix-rank turned into a training method, sitting directly downstream of the dense linear layer W it patches.
The paper's wager hypothesis: the useful update for a task has low intrinsic rank, so a rank-r ΔW suffices. That is an empirical bet, not a theorem — which is why r is a knob. Each sphere is the next one's premise: rank in, adaptation out.
The blue team's live check: recompute the parameter economy, the rank bound on ΔW, and the "starts identical to base" guarantee. If red sets B≠0 at init, this badge is where it shows.
Three matrices go in. The frozen base W ∈ ℝd×k (d=8, k=6). The two trainable adapters: B ∈ ℝd×r and A ∈ ℝr×k, with rank r = 2 ≪ min(d,k) = 6.
| matrix | role | size |
|---|---|---|
| W | frozen base | d × k |
| B | trainable | d × r |
| A | trainable | r × k |
The scalar α is a fixed scaling; the effective update is (α/r)·B·A. That is the whole input — and it is what you feed the panel below.
TRAINED: B has learned values, so ΔW≠0 and W' has moved off the base. Flip to INIT to watch ΔW collapse to exactly zero.
Change any control — the counts, the rank, and every cell of ΔW are computed on the spot from B·A, never looked up.
What the machine produces, self-checked at boot: for the toy dims, r·(d+k) = 28 trainable numbers strictly beat the 48 of full W; rank(ΔW) ≤ r always; any rank-r target update is recovered exactly by factoring (‖B·A − ΔW‖ < 1e-9); and at init B=0 gives W' = W exactly.
The blue team's witness (left) confirms these live; the red team (right) tries to make them wrong.
And r, α, and which layers to adapt are heuristics, not derived — the "low intrinsic rank" premise is an empirical hope that holds for many tasks and can fail for others. LoRA is a cheap, often-sufficient patch on a frozen model, not a proof that the frozen model was almost right.
"LoRA matches full fine-tuning with no cost." Cut. On-par or better on many benchmarks in the paper — but rank-limited tasks can lag, and quality depends on r and layer choice. "Often as good," not "free."
"Like adapter layers, LoRA adds inference latency." Cut. B·A can be merged into W after training — zero extra layers, zero added latency. That is a headline advantage over adapters.
"Any weight update is inherently low-rank." Kept, corrected. Only the adaptation is constrained to rank r; that is an imposed structure, not a proven property of the optimal update.
The red team's move: initialize B to nonzero random values. Then ΔW≠0 at init, so W' ≠ W before a single training step — the "starts identical to base" guarantee is dead. Real LoRA zero-inits B for exactly this reason.
Set B≠0 at init and W' drifts off the base immediately — the witness (window 7) recomputes the guarantee, finds it violated, and turns red. Nothing is faked; the attack is real and it is caught.