Inference Optimization: vLLM / Quantization / Speculative Decoding / KV Cache

L7-02 covered training across many GPUs. But your model only earns money when serving inference.

Inference economics:

• Train once: $1M-$100M
• Serve forever: $10K-$10M per month

Reducing inference cost by 30% → millions saved annually. This is the LLMOps engineer’s main playground.

What Slows LLM Inference

Three bottlenecks:

1. KV Cache (memory)

Every token generated requires:

• Compute the new K, V
• Store them in cache
• Use for future attention

KV cache grows linearly with sequence length. A 70B model on 32k tokens needs ~10GB just for KV cache.

2. Memory bandwidth (not compute)

Modern GPUs have way more compute than memory bandwidth:

• H100: 989 TFLOPs FP16
• H100 HBM3: 3.35 TB/s

When generating one token at a time, you transfer all weights from HBM to compute, do little math, transfer back. You’re memory-bound, not compute-bound.

3. Batching difficulty

Different requests have different lengths and arrive at different times. Naive batching wastes capacity.

The optimizations below target these three.

Optimization 1: KV Cache (already done by default)

We covered this in L7-01 / viz-kv-cache. Quick recap:

Without cache: O(n²) — recompute everything every step. With cache: O(n) — only compute new K, V each step.

This is default in every modern serving framework.

PagedAttention (vLLM’s contribution)

Standard KV cache: pre-allocate max-length buffer → wastes memory. PagedAttention: store KV in pages like OS virtual memory:

• Allocate only what’s used
• Share pages across requests with common prefix (e.g., same system prompt)

vLLM ships PagedAttention. 2-4× more requests served per GPU.

Optimization 2: Continuous Batching

Static batching:

T=0: Batch [req1, req2, req3] start
T=1: All three generate token 1
T=2: All three generate token 2
T=3: req2 finishes (short response). GPU idle on its slot.
T=4: req1 still going. req4 must wait.

Continuous batching (vLLM / TGI):

T=0: Batch [req1, req2, req3]
T=3: req2 done. Immediately slot in req4.
T=4: Batch [req1, req4, req3]
T=5: req3 done. Slot in req5.
T=6: Batch [req1, req4, req5]

GPU is always full. 5-10× throughput.

Optimization 3: Quantization

Default model: FP16 (16 bits per weight). Quantize to INT8 or INT4 → 2-4× memory savings + faster.

Format	Bits	Memory	Quality
FP16	16	100%	100% (baseline)
INT8	8	50%	99%
INT4	4	25%	95-98%
INT3	3	19%	85-90%
INT2	2	12%	60-70% (broken)

Popular quantization methods

• GPTQ (2022): per-layer quantization, calibrated
• AWQ (2023): activation-aware weight quantization
• GGUF (llama.cpp ecosystem): runs quantized models on CPU/GPU
• FP8: hardware-supported on H100+; minimal quality loss

See L7-04 for full quantization deep dive.

In practice: INT4 with AWQ is the standard for serving 70B models on 1× H100.

Optimization 4: Speculative Decoding

The clever idea (2023):

Use a small draft model to guess N tokens ahead. Verify all N with the big model in parallel.

Draft model (small, fast):  generates tokens [a, b, c, d, e]
Big model (slow, accurate): verifies all 5 in ONE forward pass
                            says: a ✓, b ✓, c ✓, d ✗, ...
                            keeps [a, b, c], starts over from d

The big model’s forward pass already computes probabilities for ALL prefix positions — so verification is almost free.

Acceptance rate

If draft is good (similar distribution to big model):

• 5 tokens guessed → 3-4 accepted on average
• → 3-4× speedup (one big-model forward = 3-4 tokens generated)

Tradeoffs

• Pro: 2-4× faster
• Con: needs maintaining 2 models
• Con: each big-model pass is slightly more expensive (computes more attention positions)

Used by: vLLM, TensorRT-LLM, all major serving stacks.

See /visualize/speculative for the interactive viz.

Optimization 5: FlashAttention

Already covered in L7-01. Quick recap:

Standard attention reads/writes intermediate matrices from HBM N times (O(N²) memory traffic). FlashAttention: tiles attention into chunks that fit SRAM, reads HBM once.

Same math, 2-5× faster on long contexts. Now standard in PyTorch (F.scaled_dot_product_attention).

Optimization 6: Tensor / Pipeline Parallelism for Inference

For massive models (175B+):

Tensor Parallel

Split each layer across GPUs (Megatron-style).

• All GPUs work on every token
• Heavy intra-card communication
• Pro: lowest latency
• Con: requires NVLink

Pipeline Parallel

Split layers across GPUs.

• Tokens flow through GPUs sequentially
• Pro: lower memory per GPU
• Con: higher latency due to pipeline

Inference: Tensor Parallel preferred (lower latency matters more than train).

Optimization 7: Caching at the Application Layer

Beyond model-level optimizations:

Prompt Cache

If many requests share the same prefix (system prompt, RAG context):

• Compute the prefix once
• Reuse the KV cache for all requests with that prefix

Anthropic introduced “Prompt Caching” API in 2024 — up to 90% cost savings for repeated prefixes.

Semantic Cache

If a question is “similar enough” to a recent one → return the cached answer.

# Cache by embedding similarity
if cosine(embed(new_q), embed(cached_q)) > 0.95:
    return cached_answer

Works for: customer support FAQs, common queries. Doesn’t work for: real-time data, personalized responses.

Real-World Stack

A production LLM serving setup:

[Load Balancer]
       ↓
[Semantic Cache] ← returns cached answer for ~30% of queries
       ↓
[Prompt Cache] ← reuses KV for shared prefixes
       ↓
[vLLM / TGI / TensorRT-LLM Server]
   - Continuous batching
   - PagedAttention
   - INT4 quantization (AWQ)
   - Speculative decoding
   - FlashAttention
       ↓
[GPU cluster (Tensor Parallel)]
       ↓
[Telemetry / Monitoring]

Net effect: 20-50× more queries/second per GPU vs naive PyTorch.

Cost Math Example

Let’s compare serving a 70B model with naive vs optimized stack:

Setup	Throughput	Cost/1M tokens
Naive PyTorch	50 tok/s	$5.00
+ vLLM	400 tok/s	$0.60
+ INT4 (AWQ)	600 tok/s	$0.40
+ Speculative decoding	1200 tok/s	$0.20
+ Prompt cache (70% hit)	4000 tok/s effective	$0.06

~80× cost reduction through software alone.

Frameworks Comparison

Framework	Best for	Notes
vLLM	Open-source default	UC Berkeley, broad support
TGI (HF Text Generation Inference)	Hugging Face ecosystem	Slightly less throughput than vLLM
TensorRT-LLM	NVIDIA NIM, max performance	Hard to set up
llama.cpp	CPU / consumer GPU	GGUF format, runs anywhere
Ollama	Personal use	Wraps llama.cpp
OpenAI SGLang	New (2024), promising	Smart caching

When to Optimize What

Priority order:

1. vLLM / TGI (default) — covers KV cache, batching, FlashAttention
2. Quantization (INT4 with AWQ) — when memory matters
3. Speculative decoding — when latency matters most
4. Prompt caching — when you have long system prompts
5. Semantic caching — when your traffic has high overlap
6. TP/PP — for 175B+ models only

Edge Cases

Streaming responses

Most user-facing chat is streaming (yield tokens one at a time). Latency = time to first token (TTFT) + per-token speed.

• TTFT: dominated by attention over prompt → use FlashAttention
• Per-token: dominated by per-token forward → use everything else

Long context (100k+ tokens)

Linear attention / sliding window may help. But mostly: bigger KV cache + more careful batching.

Multimodal

Image / audio tokens behave differently:

• Vision encoders: separate forward pass, cached
• Mixed-modality batching: complex, less mature

Next recommended: L7-04 Quantization Deep Dive or L7-05 Model Deployment.