BERT vs GPT: Two Transformer Architectures Compared

After the Transformer (2017), two branches emerged:

• BERT (Google, 2018) — encoder-only
• GPT (OpenAI, 2018-) — decoder-only

For 4 years, BERT was the dominant paradigm. Then GPT-3 / ChatGPT changed everything.

Why is worth understanding.

Quick Reminder

L3-08 covered Transformer architecture:

• Encoder layers — bidirectional self-attention
• Decoder layers — masked (causal) self-attention + cross-attention

The original 2017 paper used both for translation. BERT and GPT each kept only one half.

BERT (Encoder-Only)

Architecture: stacked encoder blocks.

Training task: Masked Language Modeling (MLM).

Input:  "The [MASK] sat on the mat."
Target: predict that [MASK] = "cat"

Random 15% of tokens are masked. Model must recover them using bidirectional context (sees both left and right).

Output

For each input token, output a vector representation.

Use cases:

• Sentence classification (sentiment, topic)
• Token classification (NER, POS tagging)
• Question answering (find span in passage)
• Similarity / retrieval

Limitations

• Can’t generate text fluently
• Not “autoregressive” — outputs aren’t sequential text

GPT (Decoder-Only)

Architecture: stacked decoder blocks without cross-attention (since there’s no encoder).

Training task: Next Token Prediction.

Input:    "The cat sat on the"
Target:   predict "mat"

Model sees only previous tokens (causal mask). Predicts next.

Output

A probability distribution over next token. Sample / argmax to generate.

Use cases:

• Text generation
• Q&A (frame as continuation)
• Chat (user / assistant turns)
• Code generation
• Everything (with the right prompt)

Key insight

Same model, just prompt it differently:

Translate to French: "Hello world" → "Bonjour le monde"
Summarize: <article> → <summary>
Continue: <story> → <next paragraph>

This is few-shot / zero-shot learning — emerged from GPT-3 (L4-06).

Side by Side

Dimension	BERT	GPT
Architecture	Encoder only	Decoder only
Attention	Bidirectional	Causal (left-only)
Training	Masked language modeling	Next token prediction
Output	Per-token vectors	Probabilities for next token
Strength	Understanding / classification	Generation
Scaling	Bert-Large = 340M params	GPT-4 ≈ 1.7T params (MoE)
Use as foundation	Yes (RoBERTa, DeBERTa, etc.)	Yes (entire LLM era)

Why GPT Won

In 2020 BERT was still dominant. By 2023 GPT had largely won. Why?

1. Generation is more general than classification

A model that can generate text can also:

• Classify (output “positive” or “negative”)
• Translate (output translation)
• Answer questions (output answer)
• Reason (output reasoning + answer)

A pure encoder can’t generate. Its scope is fundamentally limited.

2. Scaling laws favor generative models

GPT-3 (175B) was BIG. Scaling laws (L4-01) showed that bigger generative models keep improving.

BERT-style models also scaled, but the payoff wasn’t as dramatic because their use cases were narrower.

3. ChatGPT product success

The breakthrough product was chat. Chat needs generation. GPT did generation.

By 2023, “use an LLM” meant “use a GPT-style model”.

4. Tool-use / Agents are natural with generation

An Agent needs to “output a tool call” — that’s generation. Decoder-only models do this naturally.

BERT Still Lives

BERT (and descendants) are still widely used for:

Embeddings

Sentence-BERT, mpnet — text → vector for similarity search, RAG. Better than decoder-only for this because bidirectional attention sees full context.

Classification at scale

If you need to classify millions of texts cheaply — BERT-base (110M params) is much smaller than GPT-4.

Specialized domains

BioBERT, SciBERT, FinBERT — domain-tuned BERTs still beat generic LLMs on specific NLP tasks.

ModernBERT (2024)

A 2024 revamp of BERT with 8x longer context and modern training tricks. Still relevant specifically for understanding tasks.

When to Use Which

Task	Better choice
Text classification	BERT (smaller, faster)
Token labeling (NER)	BERT
Embedding for search	BERT-family
QA over a fixed passage	BERT (extractive)
Open-ended QA	GPT
Translation	Either (GPT now dominates)
Summarization	GPT
Code generation	GPT
Chat / dialog	GPT
Reasoning	GPT (with CoT)
Tool use / Agents	GPT

Rule of thumb: use BERT for understanding (input → label), GPT for generation (input → text).

Hybrid Architectures

Recent work blurs the lines:

T5 (encoder-decoder)

Google’s T5 uses both. Slightly more parameters but flexible. Used in some translation / summarization settings.

Prefix LM

Special architecture where some tokens are bidirectional, rest are causal. Mixes BERT + GPT advantages.

Discriminative + generative training

Some models train on BOTH MLM and next-token prediction. Captures benefits of both.

A Concrete Comparison

Sentiment classification, same dataset:

# BERT approach
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
# Fine-tune on labeled examples: 1000 sentences with positive/negative labels
# Result: 95% accuracy, model is 110M params, runs fast on CPU

# GPT approach (zero-shot)
prompt = """Classify the sentiment of the following text as 'positive' or 'negative':

Text: "I love this product"
Sentiment: positive

Text: "Terrible experience, never again."
Sentiment: negative

Text: {user_input}
Sentiment:"""
# No training, just prompt + GPT-4 → ~92% accuracy, but each call = $0.001

Trade-offs:

• BERT: train once, run cheap forever
• GPT: no training, but every prediction has API cost

A Historical Note

In a 2022 talk, Yann LeCun (Meta’s chief AI scientist) said:

“Auto-regressive LLMs are an off-ramp. They’re a dead end.”

He preferred encoder-style or hybrid models. Many academics agreed.

3 years later, the entire field is auto-regressive LLMs.

LeCun’s argument wasn’t wrong (auto-regressive models have real limits), but the product success of ChatGPT made decoder-only the default.

This is a useful lesson:

Sometimes the “less elegant” approach wins because it’s good enough + product-friendly.

The Future

What’s next:

• Encoder-only: ModernBERT-class for specific NLP tasks
• Decoder-only: continues to dominate general LLMs
• Hybrid architectures: Mamba+Transformer, encoder+decoder for specific applications
• Beyond Transformer: state-space models, alternative attention — research ongoing

The “BERT vs GPT” debate is over for production use cases — but the architectural questions are still alive in research.

Next recommended: L4-01 LLM Training or L3-08 Transformer Architecture.