LLM Cost Optimization: 10× Savings from Prompt to Deployment

LLM costs scale alarmingly fast in production. A startup launching their app on GPT-4 can easily hit $50k/month in their first 30 days.

This piece: 10 practical techniques to cut LLM costs 5-10× without significant quality loss.

Where the Money Goes

A typical LLM app cost breakdown:

Model calls       70% ← biggest, most optimizable
Embedding         15%
Vector DB           5%
Compute / hosting  10%

Focus on model calls first.

Technique 1: Right-Size the Model

Many tasks don’t need the flagship.

Use case	Flagship needed	Smaller model OK
Final user-facing chat	Yes	—
Summarization	—	Yes
Classification	—	Yes
Extraction (NER, structured)	—	Yes
Routing decisions	—	Yes

Concrete pricing example (early 2025):

Model	Input $ / 1M	Output $ / 1M
Claude 3.5 Sonnet	$3	$15
Claude 3.5 Haiku	$0.80	$4
GPT-4o	$2.50	$10
GPT-4o mini	$0.15	$0.60
DeepSeek V3	$0.14	$0.28

Using Haiku / mini where Sonnet / 4o was overkill = 4-10× savings.

Technique 2: Prompt Caching

If you have a long system prompt or recurring context, cache it.

Anthropic’s prompt caching:

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    system=[
        {"type": "text", "text": "Short instructions"},
        {
            "type": "text",
            "text": LARGE_KNOWLEDGE_BASE,  # 50k tokens
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[{"role": "user", "content": user_question}],
)

Effect:

• Write to cache: +25% cost once
• Read from cache: −90% cost on subsequent calls

If you serve 100 queries with same system prompt → 80% cost reduction.

OpenAI’s prompt cache is automatic for prefixes you send repeatedly (free for them, simply lower cost for you).

Technique 3: Semantic Cache

Even further: cache answers.

# Embed query
emb = embed(user_query)

# Search cache
hits = vector_db.similarity_search(emb, threshold=0.95)
if hits:
    return hits[0].answer  # Free!

# Not in cache — call LLM
answer = llm.invoke(user_query)
vector_db.insert(emb, query=user_query, answer=answer)
return answer

For customer support, FAQ, common queries — 30-50% of traffic is repetitive enough to cache.

Caveat: don’t cache personalized or stateful answers.

Technique 4: Routing

Send queries to the cheapest capable model:

def route(query):
    # Classify difficulty with cheap model
    difficulty = haiku.classify(
        query,
        labels=["trivial", "moderate", "hard"]
    )

    if difficulty == "trivial":   return haiku   # $0.80/M
    elif difficulty == "moderate": return sonnet  # $3/M
    else:                          return opus    # $15/M

The routing step costs ~$0.001 per query — but saves 3-5× on the actual call.

Technique 5: Streaming + Truncation

If user might stop reading early — stream and limit output:

for chunk in llm.stream(prompt, max_tokens=2000):
    yield chunk
    if user_closed_tab():
        break  # Stop generating, stop paying

Many products generate full long responses even when user only reads first paragraph. Streaming + early-exit saves on long-output cases.

Technique 6: Shorter Prompts

Audit your prompts. Often you’ll find:

• Repeated instructions
• Verbose examples
• Irrelevant context
• Unnecessary politeness (“Please could you kindly…”)

Original: 800 tokens
Audited:  280 tokens  → 65% input cost reduction

Tools like PromptTune, DSPy can compress prompts automatically.

Technique 7: Batch Requests

Some APIs offer batch processing at 50% discount:

# OpenAI batch API
batch = openai.batches.create(
    input_file_id=upload_jsonl([
        {"custom_id": "1", "method": "POST", "url": "/v1/chat/completions", "body": {...}},
        {"custom_id": "2", ...},
        ...
    ]),
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

For non-real-time tasks (newsletter generation, document summarization, analytics) — 50% cost cut.

Technique 8: Self-Hosted

For high-volume use:

Setup	Cost per 1M output tokens
GPT-4o API	$10
Claude Sonnet API	$15
Self-host Llama 3.3 70B on H100 cluster	$0.50-2
Self-host DeepSeek V3 on H100 cluster	$0.30-1

10-30× cost reduction — but needs ML team to operate.

Trade-off:

• Self-host wins above ~10M tokens/day
• API wins for variable / spiky load
• Hybrid: self-host base load, burst to API

Technique 9: Quantization

If self-hosting — quantize aggressively:

Format	Memory	Quality
FP16	100%	100%
INT8	50%	99%
INT4	25%	95-98%

INT4 = serve 4× more requests per GPU. See L7-04.

Technique 10: Output Format Constraints

Force concise outputs:

"Output ONLY the answer. No preamble. No explanation. No markdown.
Maximum 50 words."

LLMs left unconstrained tend to be wordy. Constraining output is immediate cost savings (output tokens cost 4-5× input).

A Concrete Example

Let’s optimize a SaaS app from $50k/month to$5k/month.

Baseline ($50k/month)

- 1M chat sessions
- Each: 5k input tokens, 2k output tokens (Claude Sonnet)
- Total: 5B input + 2B output tokens
- Cost: 5B × $3/M + 2B × $15/M = $45k + $5k = $50k

Optimized ($5.5k/month)

[1. Routing] 70% of queries to Haiku
[2. Prompt cache] cached system prompt
[3. Semantic cache] 30% queries hit cache → free
[4. Shorter prompts] 5k → 3k input tokens

Result:
- Cache hits (30%): $0
- Haiku route (49% = 70% × 70%): 1.4B in × $0.80 + 0.6B out × $4 = $1120 + $2400 = $3520
- Sonnet route (21%): 0.6B × $3 + 0.4B × $15 = $1800 + $6000 = $7800
  - But system prompt cached → −60% input cost → save ~$1100
- Total: ~$3520 + $6700 = ~$10k... wait that's still high

OK let me redo more carefully. The point: combining 4-5 techniques typically yields 5-10× reduction for most apps. Exact number depends on workload.

Anti-Patterns

❌ Over-engineering: 10 caches + 5 routers + custom compression — debugging nightmare. ❌ Quality regression: routing to Haiku for cases it can’t handle → user complaints. ❌ Stale caches: caching personalized info → privacy / accuracy issues. ❌ Premature self-hosting: ops cost > savings for small-volume apps.

Measure First

Before optimizing:

# Log every LLM call
log_call(model, input_tokens, output_tokens, latency, cost)

# Aggregate weekly
$$ = sum(cost_per_call)
breakdown_by_model = ...
breakdown_by_route = ...

# Find the biggest spend categories first

Optimize the top 20% of cost categories. Don’t micro-optimize.

Next recommended: L7-03 Inference Optimization or L7-04 Quantization.