Prompt Caching in Production: Save 90% of LLM Cost

In LLM cost structure, input tokens are usually the largest component — especially with big system prompts, RAG retrieval, or document analysis.

Prompt Caching is the most important cost optimization of the past two years:

Charge the same prefix once; subsequent hits drop to ~1/10 of the price.

Who pays this tax

Typical scenario:

Your system prompt = 5000 tokens
Each user request = 100 tokens of user input + 800 tokens output

Each billed call = 5000 + 100 input + 800 output

But the system prompt never changes —
why re-pay for it every time?

10,000 requests per day:

• Without cache: 5000 × 10000 = 50M input tokens
• With cache: 5000 once + 100 × 10000 = 6M input tokens → save 88%

Prompt Caching across three vendors

Anthropic (most explicit + most practical)

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    system=[
        {"type": "text", "text": "Short instruction"},
        {
            "type": "text",
            "text": LARGE_KNOWLEDGE_BASE,  # large chunk
            "cache_control": {"type": "ephemeral"}  # ← key
        }
    ],
    messages=[{"role": "user", "content": user_question}]
)

Effect:

• Cache write: 1.25× normal input price (first time)
• Cache read: 0.1× normal price = save 90%
• Cache TTL: 5 minutes (ephemeral); newer SKUs offer 1 hour

Up to 4 cache_control breakpoints in a single request.

OpenAI (automatic)

OpenAI has no explicit cache_control — it automatically detects repeated prefixes (>= 1024 tokens) and caches them:

# No special config needed
response = openai.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": LARGE_SYSTEM_PROMPT},
        {"role": "user", "content": user_q}
    ]
)
# Second identical request → automatic cache hit → 50% discount

Effect:

• Automatic detection, no code changes
• Cache hit: 50% discount (less aggressive than Anthropic’s 90%)
• Cache TTL: ~5-10 minutes

Gemini (context caching)

Requires explicit creation:

from google.generativeai import caching

cache = caching.CachedContent.create(
    model="gemini-2.5-pro",
    system_instruction=LARGE_SYSTEM_PROMPT,
    contents=[],
    ttl=datetime.timedelta(hours=1)
)

response = model.generate_content(
    "user question",
    cached_content=cache.name
)

Effect:

• Cache read: ~75% discount
• TTL controllable (up to several hours)
• Good for long-document RAG

The alignment principle

All vendors share a low-level logic:

Caching matches by token prefix — the prefix must be byte-identical.

Meaning:

Request A:
  system: "You are an assistant. Knowledge base: [10k tokens]"
  user:   "What is X?"

Request B:
  system: "You are an assistant. Knowledge base: [10k tokens]"
  user:   "What is Y?"
                              ↑
                10k tokens identical → cache hit → save money

Request C:
  system: "You are a helpful assistant. Knowledge base: [10k tokens]"
                          ↑
                One word different → no hit → full price

This is why you must be careful with variable content — dates, random IDs, usernames at the front break the cache.

A real-world template

def build_request(user_query: str, user_context: dict):
    return {
        "system": [
            # ① Cache layer 1: invariant instructions
            {"type": "text", "text": SYSTEM_INSTRUCTIONS,
             "cache_control": {"type": "ephemeral"}},

            # ② Cache layer 2: large knowledge base (invariant)
            {"type": "text", "text": KNOWLEDGE_BASE_10K,
             "cache_control": {"type": "ephemeral"}},

            # ③ NOT cached: per-request user context
            {"type": "text", "text": f"User profile: {json.dumps(user_context)}"},
        ],
        "messages": [{"role": "user", "content": user_query}]
    }

Order matters: variable content must come after the invariant prefix. Putting variables first invalidates every cache.

Common RAG mistake

# ❌ Wrong: retrieval first, question after
prompt = f"""
{retrieved_chunks}        ← Different each question
SYSTEM_INSTRUCTIONS
User: {user_query}
"""

# ✓ Right: invariant parts first
prompt = f"""
SYSTEM_INSTRUCTIONS       ← Cache this
RETRIEVAL_PROMPT_TEMPLATE
{retrieved_chunks}        ← Variable, not cached
User: {user_query}
"""

Can’t guarantee cache hits — but maximize hit rate.

Cache blind spots

1. ❌ Streaming responses may not show cache hits immediately

At the start of streaming, cache info may not be propagated yet. Read total token usage from the completed response.

2. ❌ Tool / function calling changes the prefix

When your tools list changes, the prefix changes too — don’t modify tools frequently.

3. ❌ Conversation memory breaks caches easily

# Multi-turn — full history sent each time
messages = [
    {"role": "system", "content": ...},   # cached
    {"role": "user", "content": "Q1"},
    {"role": "assistant", "content": "A1"},  # history
    {"role": "user", "content": "Q2"},
    {"role": "assistant", "content": "A2"},  # history
    {"role": "user", "content": "Q3"},   # new
]

Each new turn changes the prefix! Place cache_control immediately after the system segment, then let history grow.

Monitoring

OpenAI / Anthropic return cache usage in the response:

# Anthropic
print(response.usage)
# cache_creation_input_tokens: 5000  ← wrote 5000 tokens to cache
# cache_read_input_tokens: 0          ← 0 hits

# Second identical request
# cache_creation_input_tokens: 0
# cache_read_input_tokens: 5000       ← hit!

Log these two metrics and compute monthly savings:

def estimated_savings(cache_read_tokens, model="claude-sonnet"):
    # 90% off on cache reads
    full_price_per_M = 3.00  # input
    cached_price_per_M = 0.30
    saved_per_M = full_price_per_M - cached_price_per_M
    return cache_read_tokens / 1_000_000 * saved_per_M

Anti-patterns

Anti-pattern	Consequence
Putting current timestamp in system prompt	Cache invalidates every request
Putting user ID in the system prefix	Per-user cache = no shared savings
Frequently changing tool lists	Cache frequently invalidates
< 1024 token prompt expecting cache	OpenAI won’t cache
Splitting into too many breakpoints	Anthropic limits to 4

Pair with Semantic Cache

Prompt Caching saves when the prefix is byte-identical. Semantic Cache returns the previous answer when the question is semantically similar:

User A: "What's the capital of France?"  →  call LLM, cache result
User B: "France capital?"                 →  embedding match → return A's answer

The two are complementary — Prompt Caching saves input tokens; Semantic Cache skips the LLM call entirely.

Estimated savings

At 2026 pricing, 5000-token system prompt + 10k requests/day:

State	Monthly input cost
No cache	$4,500
OpenAI auto 50% off	$2,250
Anthropic 90% off	$450

A simple optimization, thousands of dollars saved per month.

Next: L4-12 LLM Cost Optimization or L7-03 Inference Optimization.

🚧 3 Common Pitfalls