Quantization Deep Dive: GPTQ / AWQ / FP8 / GGUF

A 70B model in FP16 = 140GB. A single H100 = 80GB. A 4090 = 24GB.

Without quantization, you can’t run 70B on consumer hardware. This piece explains how it actually works.

The Big Picture

Model weights are typically stored as FP16 (16-bit floats). Quantization reduces precision:

Format	Bits	Range	Use
FP32	32	±3.4 × 10³⁸	Training (legacy)
FP16	16	±65k	Training (default)
BF16	16	±3.4 × 10³⁸	Training (preferred)
FP8	8	±448	H100+ training, inference
INT8	8	-128 to 127	Inference
INT4	4	-8 to 7	Inference
INT2	2	-2 to 1	Inference (extreme)

Lower bits = less memory + faster + (usually) lower quality.

Why It Works

Surprising insight: LLM weights are highly redundant.

Most weights are near zero. The model uses fine-grained values, but the gradient is dominated by extremes. You can quantize most weights to coarse buckets without much quality loss.

Original FP16 weight: 0.0234567
INT4 quantized:       0  (rounded — close enough)

Original FP16 weight: -2.7
INT4 quantized:       -3  (rounded)

For 175B parameters, billions of small errors mostly cancel out.

Static vs Dynamic

Static quantization

Pre-compute scales / zero-points offline:

quantized_w, scale, zero_point = quantize(weight)
# scale = (max - min) / 2^bits
# zero_point = offset for asymmetric quantization

Fast at inference, but needs calibration data.

Dynamic quantization

Compute scales on-the-fly per layer / per token:

def forward(x):
    scale = compute_scale(x)
    x_quant = (x / scale).round().clamp(-127, 127)
    ...

Slightly slower, but adapts to input distribution.

Major Methods

GPTQ (Frantar et al. 2022)

Quantize one layer at a time, optimizing the rest to compensate.

for layer in model:
    # Quantize this layer's weights
    quantized = quantize_to_int4(layer.weight)
    # Compute error: original_output - quantized_output
    error = compute_error(...)
    # Adjust REMAINING layers to compensate
    next_layer.weight += correction

Key insight: layer-by-layer compensation prevents error accumulation.

Pros: high quality at INT4 (within 1-2% of FP16) Cons: slow to quantize (1-3 hours for 70B)

AWQ (Lin et al. 2023)

Activation-aware weight quantization.

Key insight: not all weights matter equally — the ones that interact with high-magnitude activations matter most. Keep those at higher precision, quantize the rest aggressively.

# Find important weight channels by activation magnitudes
important_channels = top_k(|activation|, k=1%)

# Quantize:
weights[important_channels] @ INT8
weights[other_channels] @ INT4

Pros: better quality than GPTQ Cons: needs calibration data

Most popular method for serving in 2024-2025.

FP8 (H100 native)

Hardware-level support on H100+ GPUs:

# H100 supports FP8 matrix multiplies natively
# 2× throughput of FP16, half memory

import torch
model.to(torch.float8_e4m3fn)  # FP8 format

E4M3: 4-bit exponent + 3-bit mantissa (range ±448, ~10⁻⁴ precision) E5M2: 5-bit exponent + 2-bit mantissa (wider range, less precision)

Used heavily in: DeepSeek V3 training, all modern frontier training.

GGUF (llama.cpp ecosystem)

A format, not a method. GGUF files contain quantized weights in various schemes:

• Q4_K_M — 4-bit with mixed scheme
• Q5_K_M — 5-bit
• Q8_0 — 8-bit
• IQ2_XXS — 2-bit experimental

Run on CPU or GPU, optimized for consumer hardware.

# Install llama.cpp
brew install llama.cpp  # macOS

# Download a GGUF model
huggingface-cli download bartowski/Llama-3.3-70B-Instruct-GGUF \
    --include "*Q4_K_M*"

# Run
llama-cli -m model.gguf -p "Hello"

Used by: Ollama, LM Studio, GPT4All, KoboldCpp.

NF4 (QLoRA, 2023)

Normal Float 4-bit — non-uniform quantization optimized for normally-distributed weights.

Used in QLoRA (fine-tuning quantized models).

Mixed Precision

Best of both worlds — quantize most, keep some at higher precision:

# Example: 4-bit base + 16-bit LoRA adapters
base_model = load_model_4bit("Llama-3-70B")
lora_adapter = load_lora_fp16("./my_lora")  # Adapter stays high precision

# Inference combines them

Heavy quantization for memory, high precision for the learning parts.

Quality / Speed Trade-offs

Real benchmarks (Llama-3-70B):

Setup	VRAM	Tokens/sec	Quality (MMLU)
FP16	140GB	25	79.5
INT8 (BitsAndBytes)	70GB	30	79.0
GPTQ INT4	35GB	45	78.3
AWQ INT4	35GB	50	78.7
GGUF Q4_K_M	40GB	35 (CPU)	78.5
FP8 (H100)	70GB	50	79.3

Sweet spot: AWQ INT4 — 4× memory savings, ~1% quality loss, 2× throughput.

When Quality Drops

Quantization can hurt:

1. Long-context tasks

Tasks needing 32k+ context attention sometimes degrade more.

2. Math / code

Numerical precision matters more — INT4 loses ~3-5% on these.

3. Multi-step reasoning

Errors compound across steps.

Rule of thumb: simpler the task, more quantization you can get away with.

Practical Recommendations

Inference, max VRAM efficiency

AWQ INT4 — best quality-per-bit.

Personal use, any hardware

GGUF (via Ollama / LM Studio) — runs on Mac, PC, anywhere.

Training a quantized model

QLoRA (NF4 + LoRA) — fine-tune 70B on 1× H100.

Production high-throughput

FP8 on H100+ — fastest, hardware-supported.

Tools

Tool	Use
BitsAndBytes	Easy INT8 / INT4 in HuggingFace
AutoGPTQ	GPTQ quantization
AutoAWQ	AWQ quantization
llama.cpp	GGUF + CPU inference
Ollama	Easiest local deploy
TensorRT-LLM	NVIDIA production inference
vLLM	Open-source serving with AWQ

Quantization Recipes

Quantize your own model

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = 'meta-llama/Llama-3-70B-Instruct'
quant_path = 'Llama-3-70B-Instruct-awq'
quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4}

# Load and quantize
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)

Takes 1-3 hours, fits 70B in 35GB.

Load and run

from awq import AutoAWQForCausalLM

model = AutoAWQForCausalLM.from_quantized("./Llama-3-70B-Instruct-awq")
output = model.generate(input_ids, max_new_tokens=200)

Future Directions

• Lower bits (1.58-bit, 1-bit models) — research shows it’s possible
• Better calibration — automated, no manual tuning
• Hardware — purpose-built AI chips with native low-bit support
• Mixed: per-channel, per-layer, per-token quantization

Next recommended: L7-05 Model Deployment or L7-03 Inference Optimization.