LLMOps Panorama: Observability / Eval / Monitoring / Experiments

L7-05 covered “how to deploy a model.” L7-07 covered “basic monitoring.” This piece: what to do after launch — the LLMOps panorama.

The biggest difference from traditional software — shipping is just the start. Long-term operation is continuous eval + continuous optimization.

Why LLMOps differs from MLOps

Dimension	Traditional MLOps	LLMOps
Model source	Trained in-house	Mostly API calls
Retraining frequency	Daily / weekly	Almost never (unless fine-tuning)
Core pain	Model drift	Prompt drift / API changes / cost spikes
Eval difficulty	Has ground truth	No reference answer — needs LLM-as-judge
Failure modes	Accuracy drop	Hallucination / jailbreak / cost spike / tool failure

LLMOps is a brand-new engineering field, only mature since 2024-2026.

Full LLMOps stack (2026)

┌─────────────────────────────────────────────────┐
│  App layer: your LLM product                     │
└─────────────────────┬───────────────────────────┘
                      │
        ┌─────────────┼─────────────┐
        ↓             ↓             ↓
  ┌─────────┐  ┌─────────────┐  ┌─────────┐
  │ Routing │  │   Caching   │  │  RAG    │
  │OpenRouter│ │  Redis /    │  │ pgvector│
  │ LiteLLM │  │ Prompt cache│  │ Pinecone│
  └────┬────┘  └─────────────┘  └────┬────┘
       │                              │
       ↓                              │
  ┌─────────────────────────────┐     │
  │     LLM providers (many)     │←───┘
  │ OpenAI/Anthropic/Google/...  │
  └─────────┬───────────────────┘
            │
            ↓
  ┌──────────────────────────────────────┐
  │       Observability layer             │
  │  Langfuse / Helicone / LangSmith     │
  │  - Trace (full request lifecycle)    │
  │  - Token / cost / latency            │
  │  - Prompt / response full logs       │
  └────────────┬─────────────────────────┘
               │
               ↓
  ┌──────────────────────────────────────┐
  │       Eval layer                      │
  │  - Offline eval (before each deploy) │
  │  - Online eval (sample in real time) │
  │  - LLM-as-judge / human review       │
  └────────────┬─────────────────────────┘
               │
               ↓
  ┌──────────────────────────────────────┐
  │       Experimentation / iteration     │
  │  - Prompt version control            │
  │  - A/B testing                       │
  │  - Model swap experiments            │
  └──────────────────────────────────────┘

Tools cheat sheet

Observability (trace + logs)

Tool	Niche	Open source	Pricing
Langfuse	Self-host-friendly OS first choice	✅	Free / Cloud $29+
Helicone	OpenAI-style proxy	✅	Free / Pro $20+
LangSmith	LangChain official	❌	Free + Plus
Phoenix (Arize)	Notebook + production	✅	Free
W&B Weave	W&B’s offering	❌	Free + paid

Core capability: capture every LLM call’s full trace (input / output / model / latency / cost / tool calls).

Eval

Tool	Niche
Langfuse Datasets + Eval	Integrated
promptfoo	YAML config + CI integration
Inspect AI (UK AISI)	Serious safety eval
RAGAS	RAG-specific eval
TruLens	Multi-dimension RAG eval

Prompt management

Tool	Niche
Latitude	Prompt version control + collab
PromptLayer	Mainstream
Langfuse Prompts	Integrated with trace + prompt
Self-hosted git repo	Simple and reliable

LLM gateway / routing

Tool	Niche
OpenRouter	Unified API + multi-vendor routing
LiteLLM	Self-host proxy
Portkey	Commercial gateway (routing + cache + retry)

Practical: minimal viable LLMOps stack

Goal: know how much each LLM call costs, how fast it is, how good it is.

1. Add Langfuse trace (10 minutes)

from langfuse import Langfuse
from langfuse.decorators import observe

langfuse = Langfuse(public_key="pk-...", secret_key="sk-...")

@observe()
def answer_user_question(query: str):
    # Your existing LLM call
    response = openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": query}]
    )
    return response.choices[0].message.content

Auto-captures: input / output / model / token usage / cost / latency / trace ID.

2. Build an eval set (30 minutes)

# Collect 50-200 real user requests (anonymized) + expected answers
dataset = langfuse.create_dataset(name="customer_support_v1")
dataset.create_item(input="how do I get a refund", expected_output="...")
# ... 50+ items

3. Run eval before each deploy

# In CI
for item in dataset.items:
    pred = answer_user_question(item.input)
    score = llm_judge(pred, item.expected_output)
    if score < 0.7:
        raise Exception(f"Regression on {item.id}!")

4. Add online monitoring & alerts

# Sample 10% of production requests for LLM-as-judge in real time
# Alert when hallucination rate > 5% or cost > $X/h

Eval levels

1. Unit eval

One metric per capability:

• Classification accuracy
• JSON schema compliance
• Refusal rate (does it refuse sensitive queries?)

2. System eval

The entire LLM app:

• Task success rate
• User satisfaction (NPS)
• Average time to resolution

3. Safety eval

Targeting misuse / attacks:

• Jailbreak attack success rate
• Hallucination rate
• Privacy leak rate

4. Business eval

Ultimate business metrics:

• Customer retention
• Conversion
• AOV (average order value)

5 must-monitor metrics

Daily:

1. Cost: total daily spend + avg per request
2. Latency: P50 / P95 / P99
3. Hallucination rate: sample + human review or LLM-as-judge
4. Error rate: API failures + timeouts + parse errors
5. User feedback: thumbs up/down ratio

A/B experiments (prompt / model swap)

LLM product A/B is different from traditional software:

# 50% users go through prompt v1, 50% through prompt v2
prompt_version = "v2" if hash(user_id) % 2 == 0 else "v1"
prompt = load_prompt(f"customer_support_{prompt_version}")

# Tag the trace
with langfuse.trace(metadata={"prompt_version": prompt_version}):
    response = llm(prompt + query)

Evaluation windows:

• Offline eval: immediate (pre-deploy)
• User satisfaction: 1-2 weeks (statistical significance)
• Long-term retention: 1-3 months

Common production incidents (real cases)

1. Cost runaway

Dev wrote wrong max_tokens → each user burns 100K tokens → one night’s bill explodes. Defense: per-request cost alert + per-user quota.

2. Prompt injection

User puts “ignore previous instructions” in input. Defense: input filter + stronger system prompt + output review.

3. Model version silently changed

OpenAI repointed gpt-4 alias → answer quality shifts. Defense: pin exact versions (gpt-4-2024-08-06), keep eval running.

4. Hallucination surge

New scenario, model frequently confabulates → customer complaints. Defense: in-line LLM-as-judge sampling + RAG.

5. API rate limiting

Provider’s rate limit insufficient → 429 storm. Defense: multi-model fallback + queue + retry with backoff.

Team roles (mature LLMOps team)

Role	Responsibility
LLM Engineer	Prompt engineering + Agent / RAG / tools
Eval Engineer	Test sets / auto eval / regression monitoring
AI Safety / Red Team	Attack testing / misuse monitoring / content moderation
Prompt Designer / PM	Business understanding + prompt productization
Traditional SRE	Infra / cost / latency

Small teams collapse these roles; mid-large companies make each an independent position.

Recommended further reading

• HelloAI L4-08 LLM Evaluation
• HelloAI L7-05 Deployment
• HelloAI L7-07 Monitoring
• HelloAI L4-12 LLM Cost Optimization
• Langfuse official docs
• OpenAI Evals repo
• Papers: “How we evaluate models” from OpenAI / Anthropic

🚧 3 Common Pitfalls