Adaptive RAG: Let the LLM Decide Whether and How to Retrieve

L4-03 covered “classic RAG” — every query goes through embed → retrieve → stuff prompt → LLM. But in production, this naive pipeline has real problems:

• Asking “hi” still triggers retrieval = wasted embedding + DB call
• Asking a complex research question — one retrieval isn’t enough
• Asking sensitive topics — retrieval results may inject noise

Adaptive RAG: let the LLM judge first, then decide how to retrieve.

Naive RAG vs Adaptive RAG

Naive RAG (L4-03):
[Question] → embed → vector search → top-k → stuff prompt → LLM answers

Adaptive RAG:
[Question] → LLM routing → {
  No retrieval → answer directly
  Simple retrieval → classic RAG
  Complex retrieval → multi-hop / multi-source
  Need tools → call API (not just docs)
}

4 routing strategies

1. No-retrieval

For:

• Simple greetings: “hi” / “thanks”
• General common sense: “what is 1+1”
• Things the model already knows: “what is Python”

How to decide:

• Let the LLM say “do I need to look this up?”
• Or train a small classifier

Code:

def needs_retrieval(query: str) -> bool:
    prompt = f"""Decide if the user's question needs a knowledge base lookup.
Return yes or no.
- greeting / common sense → no
- company / product / specific fact → yes

Question: {query}
Answer:"""
    return "yes" in llm(prompt, max_tokens=5).lower()

Savings: in customer support, can cut “useless retrievals” by 30-50%.

2. Single-shot retrieval

Classic RAG. For:

• Most factual queries
• “What’s the X parameter for product Y”
• “What’s the refund policy”

3. Multi-hop retrieval

For complex queries requiring multiple retrievals + reasoning:

Q: "Is the product I bought last year still under warranty?"
↓
Retrieval 1: user's orders → "bought X in March last year"
↓
Retrieval 2: X's warranty policy → "2-year warranty"
↓
LLM reasons: now is June 2026, still under warranty

Implementation: ReAct-style loop

LLM outputs: thought + next retrieval query
Execute retrieval
LLM outputs: based on result, decide to continue or answer

4. Iterative refinement

For research / synthesis queries:

Q: "Compare GPT-5 vs Claude 4 on coding"
↓
Retrieval 1: GPT-5 coding benchmarks
Retrieval 2: Claude 4 coding benchmarks
Retrieval 3: comparison / user reviews
↓
LLM synthesizes

Not just multiple retrievals — also self-reflection:

• After answering, ask LLM “is this complete?”
• If not, supplement retrieval / answer

How routing decisions are implemented

A. LLM routing (expensive but accurate)

Use the big LLM directly:

ROUTING_PROMPT = """Based on the user's question type, output ROUTING:
- chat → simple greeting
- simple_rag → one retrieval is enough
- multi_hop → needs multiple retrievals + reasoning
- web_search → needs live web (not knowledge base)
- code_exec → needs to run code

Output only the label."""

Cost: extra LLM call per query. But savings from skipped retrieval far outweigh this routing call.

B. Small classifier (cheap but needs training)

Train a distilbert / mini LM classifier:

from sentence_transformers import SentenceTransformer
import torch.nn as nn

class RouteClassifier(nn.Module):
    # MiniLM → 5 classes
    ...

# Training data: 500-1000 hand-labeled (query, route_label) pairs
# Per-call cost: < 1ms, no LLM call

Good for high-QPS / cost-sensitive scenarios.

C. Rules + LLM fallback

Simplest:

• Length < 10 chars → likely chat
• Contains dates / numbers / proper nouns → likely RAG
• Otherwise → let LLM decide

Good for MVP — 3 days to ship.

Full Adaptive RAG architecture

[User question]
       ↓
[Router (LLM or classifier)]
       ↓
{
  chat → direct LLM
  simple_rag → embed → search → rerank → LLM
  multi_hop → ReAct loop (search + reason)
  web_search → live search API
  code_exec → Python sandbox
}
       ↓
[Output formatter (unified format)]

Real example: customer support

Requirements:

• User greets → answer directly
• User asks about product / policy → classic RAG
• User asks about “my order” → call order API
• User asks “compare A and B” → multi-retrieval + synthesis

async def adaptive_answer(user_msg: str, user_id: str) -> str:
    route = classify(user_msg)  # LLM or small model

    if route == "chat":
        return llm.chat(user_msg)

    if route == "order_query":
        order = await get_user_order(user_id)
        return llm.chat(f"User order: {order}\nQuery: {user_msg}")

    if route == "simple_rag":
        chunks = retrieve(user_msg, k=3)
        return llm.chat(f"Docs:\n{chunks}\nQuery: {user_msg}")

    if route == "comparison":
        sub_qs = decompose(user_msg)  # LLM splits question
        all_chunks = []
        for q in sub_qs:
            all_chunks.extend(retrieve(q, k=2))
        return llm.chat(f"All info:\n{all_chunks}\nQuery: {user_msg}")

Performance / cost comparison

Customer support, 1000 requests:

Strategy	Total retrievals	Total LLM tokens	Monthly cost
Naive RAG (always retrieve)	1000	5M	$150
+ simple chat routing	600	3.5M	$105
+ multi-hop (better on complex)	1200	4.5M	$135
+ small classifier (no LLM routing cost)	1200	4.2M	$125

Also: multi-hop improves complex-query accuracy by 30-50%.

Real open-source projects

Project	Approach
LlamaIndex Adaptive RAG	Built-in router + multi-hop
LangGraph	Express adaptive routing as a graph
DSPy	Prompt-as-programming for routing
Self-RAG (paper)	LLM judges whether retrieval is needed
Corrective RAG (paper)	After retrieval, judge “quality” and decide to supplement

When not to use Adaptive RAG

• MVP stage: get naive RAG working first
• Low traffic: <100 queries/day, savings aren’t worth dev cost
• DB is super cheap: when routing cost > retrieval cost, it’s negative ROI
• All queries are uniform: same type → no need to split

3 engineering tips

1. Start with 2-class routing

Don’t begin with 5 classes — start with chat vs rag, then refine.

2. Routing accuracy matters more than RAG accuracy

Wrong routing = whole pipeline wrong. Get routing accuracy > 95% first.

3. Log routing decisions for A/B

Always log (query, route, outcome) → useful for later training a classifier / improving prompts.

Recommended further reading

• HelloAI L4-03 RAG from 0 to 1
• HelloAI L4-08 LLM Evaluation
• HelloAI L4-15 Prompt Caching
• Paper: Self-RAG (Asai 2023)
• Paper: Corrective RAG (Yan 2024)

🚧 3 Common Pitfalls