L4 Chapter 6 🐣 🕒 9 min

In-Context Learning: Why LLMs Learn From a Few Examples

The surprising GPT-3 capability — no fine-tuning, just a few examples and the model can do new tasks. How does it actually work?

Alai

8/1/2026

When GPT-3 launched in 2020, the AI community was stunned not by raw size (175B) but by a capability:

Show GPT-3 just 3 examples of a new task — and it does it. Without any retraining.

This is called In-Context Learning (ICL). This piece explains: what it is, why it works, when it doesn’t.

What is In-Context Learning

Q: Translate French → English

Examples:
"bonjour" → "hello"
"merci"   → "thank you"
"chat"    → "cat"

Now: "fromage" → ?

Model: "cheese"

Three examples, model does the task. No fine-tuning, no gradient updates, just prompting.

Why is this remarkable?

Traditional ML: feed 1000+ examples + epochs of training
ICL: 3 examples in the prompt, done in milliseconds

The model isn’t actually learning new weights — but behaves as if it learned.

Three Variations

Zero-Shot

No examples:

"Translate to English: fromage"

Works if the task is described in natural language.

One-Shot

1 example:

"bonjour" → "hello"
"fromage" → ?

Few-Shot

K examples (typically 3-8):

"bonjour" → "hello"
"merci"   → "thank you"
"chat"    → "cat"
"fromage" → ?

The more examples, the more accurate — but diminishing returns past ~8 examples.

Performance: GPT-3 Original Paper Results

The famous 2020 GPT-3 paper showed clear scaling:

Task	0-shot	1-shot	Few-shot
Translation	27 BLEU	33 BLEU	39 BLEU
Trivia QA	64%	68%	71%
Reading comp	65%	71%	75%

And crucially: bigger models benefit more from ICL. GPT-3 13B vs 175B — the 175B gained much more from few-shot examples.

This was the strong “scale unlocks new abilities” signal.

How Does ICL Actually Work

Hot research topic for 5 years. Three main theories:

Theory 1: Implicit Gradient Descent (2022)

Recent papers (Akyürek et al.) suggest the attention mechanism itself performs implicit gradient descent:

Each forward pass through attention layers approximates one step of optimization
Examples in the prompt act like “training data”
Output reflects the result of “training” on that mini-dataset

Evidence: small Transformers explicitly trained to mimic linear regression converge to ICL solutions that match real gradient descent.

Theory 2: Pattern Completion

The model learned during pre-training to complete patterns. ICL examples set up a pattern, the model fills it.

This is more “shallow” than gradient descent — but explains why ICL works for tasks the model has seen patterns of during training.

Theory 3: Task Inference

The model uses the examples to infer which task it should do, then applies pre-learned skills.

Examples → "Oh, this is translation!"
        → Apply pre-learned translation knowledge
        → Generate translated output

Different theories may apply to different tasks.

When ICL Works Well

✅ Format-following: copying patterns, JSON output, specific output style ✅ Style transfer: write in the style of these examples ✅ Simple classification: sentiment, intent ✅ Translation-like tasks: input → output mapping ✅ Tasks the model has seen during training

When ICL Fails

❌ Novel knowledge: model can’t learn new facts from examples (it doesn’t update weights) ❌ Long-tail facts: counting beyond what training showed ❌ Multi-step reasoning that’s too unique ❌ Numerical extrapolation: trained on 1-100, test on 1000 ❌ Genuinely new tasks not in training data

If the task is “produce content that doesn’t exist in the pre-training data” — ICL won’t save you. You need fine-tuning, RAG, or both.

Things That Hurt ICL

1. Wrong format

Bad:
"bonjour means hello"
"merci means thank you"
"fromage = ?"

Inconsistent format → model gets confused.

2. Inconsistent labels

If your examples don’t reliably show one mapping (some translations are wrong, etc.) — model picks up the wrong signal.

3. Example order matters (sometimes)

For some tasks, the order of examples affects accuracy by 10-20%. There are tricks (e.g., put the most “anchoring” example first), but no universal rule.

4. Position bias

Models often weight recent examples (those at the end of the prompt) more. Add a critical example near the end.

ICL vs Fine-Tuning vs RAG

When to use which?

Need	Use
Quick task adaptation	ICL
Format / style control	ICL
Large-scale custom knowledge	RAG
Style transfer + permanent	Fine-tune (LoRA)
Novel domain capability	Fine-tune or train
Real-time facts	RAG

Most production systems use combinations:

ICL for format
RAG for knowledge
Fine-tuning for behavior

Practical Tips

1. Number of examples

0-shot: only when task is well-described in natural language
1-3 shots: most cases
5-8 shots: harder tasks
More than 8: usually wasted tokens (diminishing returns)

2. Diverse examples

Don’t show only one type of input. Cover the variety in your test data:

Bad: 5 examples all about restaurants
Good: examples covering restaurants, hotels, attractions, transit

3. Hard examples

Include cases that are tricky — model learns “what to watch for”.

4. Be explicit about format

"Output format: JSON with keys 'sentiment' and 'confidence'.
sentiment ∈ {positive, negative, neutral}.
confidence ∈ [0,1] with 2 decimal places."

5. Avoid leakage

Don’t put test answers in your examples (sounds obvious but happens with auto-pipelines).

Modern Models: ICL “Built In”

Reasoning models (o1, o3, Claude 3.7 Sonnet, Gemini 2.5) do thinking internally before answering. The result: classical ICL examples matter less than they used to.

But ICL still wins for:

Format specification (impossible to “think your way” to a specific format)
Style transfer (examples set the tone)
Domain-specific patterns (your jargon, your conventions)

A Famous Failure Mode

ICL can be brittle. A 2024 study:

Same task, same examples — just reshuffled the order: Accuracy varied from 65% to 88%.

This is one reason production systems often build fine-tuned models for stable behavior.