LoRA Fine-Tuning: Custom LLMs on Consumer GPUs

L4-01 walked through how LLMs are trained. But what if you want to customize one for your domain — without a $1M budget?

Enter LoRA.

The Problem

Full fine-tuning of GPT-3 (175B) requires:

• 700GB+ VRAM (multi-GPU)
• ~$1000+ per run
• Each version: 350GB stored

For most people / companies — impossible.

LoRA’s Core Insight

When fine-tuning a large model, the weight delta $\Delta W$ is low-rank.

What does that mean concretely?

A full weight matrix might be 4096×4096 = 17M params. But its change during fine-tuning lives in a much smaller subspace.

Mathematically:

$$\Delta W = A \cdot B$$

Where $A \in \mathbb{R}^{d \times r}$ and $B \in \mathbb{R}^{r \times d}$, with $r \ll d$ (typically 4-64).

Full delta:  ΔW ∈ R^(4096 × 4096) → 17M params
LoRA delta:  ΔW = A · B
             A ∈ R^(4096 × 8) → 32K
             B ∈ R^(8 × 4096) → 32K
             total: 64K params

→ 256× fewer parameters trained!

Why This Works

Intuition:

• A pre-trained model already has rich knowledge baked in
• Fine-tuning adjusts behavior, not rebuilds knowledge
• The adjustment lives in a tiny subspace

Analogy: you don’t rewrite the OS to install a new app — you just add a small layer of customization on top.

How LoRA Works in Code

Standard fine-tuning

# All params trainable
for param in model.parameters():
    param.requires_grad = True

Memory needed: weights + gradients + Adam state = huge.

LoRA

# Freeze original weights
for param in model.parameters():
    param.requires_grad = False

# Add small trainable matrices
r = 8
A = nn.Parameter(torch.randn(d, r))   # train this
B = nn.Parameter(torch.zeros(r, d))   # train this (init to 0 to preserve original behavior)

# Forward
def forward(x):
    return x @ (W + A @ B)   # W unchanged + A·B provides delta

Only A and B are trained.

• Params: ~256× fewer
• Memory: original W frozen, no gradient storage
• Adam state: only for A, B → massive savings

Experimental Results

GPT-3 175B on multiple tasks:

Method	Trainable params	Performance (vs full FT)
Full Fine-tuning	175B (100%)	100%
Adapter	38M (0.02%)	98%
Prefix Tuning	0.04M	95%
LoRA	4.7M (0.003%)	101%

Least params + slightly outperforms full FT!

Why does LoRA sometimes beat full FT?

• Prevents overfitting (fewer params)
• More stable training
• The “constraint” acts as regularization

Engineering Wins

1. Massive VRAM savings

Fine-tune a 7B model with LoRA:

• No GPU offload needed — 24GB GPU is enough
• No multi-GPU — single RTX 4090 / A6000 works

2. Lightning-fast switching

A and B can be loaded/unloaded in seconds — one base model + N LoRA adapters:

Base Model (Llama-3-70B)
  ├── LoRA: Legal domain
  ├── LoRA: Medical domain
  ├── LoRA: Customer A style
  ├── LoRA: Customer B style
  └── ...

Multi-LoRA deployment — one model serves infinite customizations.

3. Storage friendly

Full 7B model → 14GB LoRA adapter → ~30MB

10,000 fine-tuned versions = 300GB (vs 140TB full) — 1000× savings.

4. Composable

In theory:

• LoRA-A: learned “code style”
• LoRA-B: learned “Chinese”
• A + B loaded together = “Chinese coding assistant”

In practice works, though not perfectly (cross-interference possible).

QLoRA: LoRA + Quantization

QLoRA (2023) combines LoRA with 4-bit quantization:

base = load_4bit_model("Llama-3-70B")  # base in 4-bit
lora_config = LoraConfig(r=16, lora_alpha=32)
model = get_peft_model(base, lora_config)
trainer.train()

70B model with ~40GB VRAM — fine-tune 70B on a single H100 (80GB)! Before LoRA: impossible.

See L7-04 on quantization.

The PEFT Family

LoRA inspired a whole “Parameter-Efficient Fine-Tuning” family:

Method	Idea	Note
LoRA	Low-rank matrix decomposition	Classic
DoRA (2024)	Decompose magnitude + direction	LoRA improvement
VeRA (2024)	Share LoRA matrices + scaling vector	Even fewer params
PiSSA (2024)	SVD init for LoRA	Faster training
(IA)³	Element-wise scaling	Minimal params

LoRA remains the de facto standard — others are incremental improvements.

Industrial Adoption

Open-source ecosystem

Almost all open-source fine-tuning uses LoRA:

• Civitai Stable Diffusion fine-tunes
• HuggingFace LLM fine-tunes
• Adobe / Canva “custom styles”
• OpenAI fine-tuning API (partially uses LoRA-class tech)

Salesforce’s case

Salesforce Einstein GPT — fine-tunes a LoRA per enterprise customer:

• Same base model (GPT-4 class)
• One LoRA adapter per customer (~30MB)
• Loaded dynamically at serve time

One base model = serving hundreds of enterprise customers.

Reflections

”Is LoRA a silver bullet?”

No — LoRA’s limits:

• Knowledge injection: not great for teaching new facts (use RAG instead)
• Large style shift: may need large r
• Specific tasks: full FT still slightly better in some cases

Start with LoRA + RAG — consider full FT only when those aren’t enough.

”Why did this take so long?”

LoRA’s math isn’t new — “low-rank matrix decomposition” is textbook linear algebra.

But applying it to LLM fine-tuning was Microsoft’s 2021 insight. Right idea at the right time — that’s how a simple thought becomes a revolution.

Using the PEFT Library

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

# Load base model
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B")

# Configure LoRA
config = LoraConfig(
    r=16,                              # rank
    lora_alpha=32,                     # scaling
    target_modules=["q_proj", "v_proj"],  # apply LoRA only to Q, V
    lora_dropout=0.1,
    bias="none",
)

# Apply LoRA
model = get_peft_model(model, config)
model.print_trainable_parameters()
# trainable params: 4M / 8B = 0.05%

# Train (just like normal fine-tuning)
trainer = Trainer(model=model, ...)
trainer.train()

# Save (just the LoRA, not the whole model!)
model.save_pretrained("./my_lora_adapter")  # only tens of MB

This is the standard fine-tuning flow in 2026 — minimal, efficient, cheap.

Next recommended: L4-06 In-Context Learning or L7-04 Quantization Deep Dive.