Monitoring and Observability: LLM Application Production

L7-05 covered deployment. This piece: monitoring after deployment.

Traditional web monitoring isn’t enough for LLMs — most LLM-specific quality problems are invisible to standard tools.

No monitoring = black box = problems unknown until users complain. Production LLM apps need an LLM monitoring stack.

Three Dimensions of LLM Monitoring

Unlike traditional services:

Dimension 1: Technical Metrics (Basics)

Same as any web service:

• Latency (P50 / P95 / P99)
• Throughput (QPS)
• Error rate (5xx)
• Availability (uptime)

Dimension 2: LLM-Specific

Things LLMs particularly need:

• Token usage (input + output separately)
• Cache hit rate
• Cost per request
• Model mix usage
• Failure + retry counts

Dimension 3: Quality (Hardest)

• Output quality (user thumbs up/down)
• Hallucination rate
• Refusal rate (legitimate vs over-refusal)
• Style consistency
• Business KPIs (conversion / retention)

Third dimension is hardest — needs ongoing “quality eval” (see L4-08).

Logging Strategy

Required logs

Every LLM call should log:

log_entry = {
    # Basics
    "request_id": uuid(),
    "timestamp": now(),
    "user_id": user_id,
    "session_id": session_id,

    # LLM call
    "model": "gpt-4o",
    "endpoint": "/chat",
    "prompt_tokens": response.usage.prompt_tokens,
    "completion_tokens": response.usage.completion_tokens,
    "total_cost_usd": calculate_cost(...),

    # Performance
    "latency_ms": elapsed_ms,
    "ttft_ms": time_to_first_token,
    "cache_hit": cache_hit_bool,

    # Quality
    "prompt": prompt[:1000],  # Watch for PII
    "response": response[:1000],
    "fingerprint": response.fingerprint,  # OpenAI provides for consistency tracking

    # Errors
    "error": None,  # Or error details
    "retry_count": 0,

    # Business
    "feature": "summarization",
    "user_tier": "pro",
    "experiment_group": "B",
}

log_to_db(log_entry)

Comprehensive logging is the foundation for all downstream monitoring.

User feedback

@app.post("/feedback")
def feedback(request_id, rating, comment=""):
    log_feedback({
        "request_id": request_id,
        "rating": rating,  # 1-5 or thumbs
        "comment": comment,
        "timestamp": now(),
    })

Every request gets direct quality feedback — gold for analyzing model quality later.

Monitoring Tool Choices

Tool	Use	Recommendation
LangSmith (LangChain)	Full LLM app tracing	⭐⭐⭐⭐⭐
Helicone	Monitoring + cost	⭐⭐⭐⭐
Phoenix (Arize)	Open-source LLM observability	⭐⭐⭐⭐
Langfuse	Open-source tracing + eval	⭐⭐⭐⭐
Weights & Biases (Weave)	Experiment + LLM tracking	⭐⭐⭐
OpenAI Dashboard	Own token usage	⭐⭐⭐ (OpenAI only)
Datadog LLM Observability	Enterprise	⭐⭐⭐⭐

Starter recommendation: Langfuse (open-source, free) or LangSmith (more features).

Example with LangSmith

from langsmith import traceable

@traceable
def my_rag_pipeline(query):
    # 1. Retrieve
    docs = vector_db.search(query)

    # 2. Build prompt
    prompt = build_prompt(query, docs)

    # 3. LLM
    response = llm.invoke(prompt)

    return response

# Call = auto-traced
result = my_rag_pipeline("How does AI work?")

LangSmith UI automatically shows:

• Each step’s input / output
• Each step’s latency
• Each step’s cost
• Total quality score

Monitoring Dashboards

Common components in production:

Real-time metrics panel

Current QPS:        42
Current P95 latency: 2.3s
Today's tokens:      12.5M
Today's cost:        $156
Cache hit rate:      48%
Active users:        890

Time series

• 24h QPS trend
• Error rate over time
• Cost over time
• P50 / P95 / P99 latency over time

Model usage breakdown

Model distribution:
├── gpt-4o-mini (70%)  → simple queries
├── gpt-4o (25%)        → complex tasks
└── claude-3-opus (5%)  → fallback

Cost distribution:
├── gpt-4o-mini (15%)
├── gpt-4o (60%)
└── claude-3-opus (25%)

> mini high volume but cheap, opus low volume but expensive.

Error categorization

Error type breakdown:
├── timeout (40%)
├── content filter (25%)
├── token limit (15%)
├── API rate limit (10%)
└── other (10%)

Targeted optimization — too many timeouts = shorten prompt / switch model.

Alerting

Monitoring isn’t watching — it’s automatic alerting:

Key alert rules

- name: high_error_rate
  condition: error_rate > 5% for 5 minutes
  severity: critical
  notify: pagerduty

- name: cost_spike
  condition: hourly_cost > $100
  severity: warning
  notify: slack

- name: latency_degradation
  condition: P95 > 5s for 10 minutes
  severity: warning
  notify: slack

- name: cache_dropout
  condition: cache_hit_rate < 20% for 30 minutes
  severity: info
  notify: email

- name: user_satisfaction_drop
  condition: thumbs_down_rate > 10% for 1 hour
  severity: critical
  notify: pagerduty

Every LLM service needs these.

Quality Monitoring

Hardest and most important — how to continuously monitor LLM output quality?

Method 1: User feedback

Most direct — but biased:

• Unhappy users more likely to thumbs-down
• Silently unhappy users don’t feedback
• Feedback latency (might be long after launch)

Method 2: Periodic sampling + human review

# Every 100 requests, sample 1 for ops review
def sample_for_review(request_id, response):
    if random() < 0.01:
        send_to_review_queue(request_id, response)

Suited for: high-value scenarios (customer service, medical, finance). Downside: expensive + slow.

Method 3: LLM-as-Judge

Auto-score with another LLM:

def auto_evaluate(prompt, response):
    judge_prompt = f"""
    Rate this response on:
    1. Helpfulness (1-5)
    2. Accuracy (1-5)
    3. Safety (1-5)

    Prompt: {prompt}
    Response: {response}

    Return JSON.
    """
    return gpt4_judge(judge_prompt)

Suited for: large-scale auto eval. Downside: judgment biases (see L4-08).

Method 4: Business KPIs

Ultimate — look directly at business:

• Customer service bot: CSAT scores, resolution rate
• Writing assistant: export rate, user save rate
• Translation: edit distance (how much users change it)

This is the “gold standard” of quality — but feedback cycle is long.

Production: combine four methods — they complement each other.

Security Monitoring

LLMs particularly need security monitoring:

Detect suspicious prompts

def check_suspicious_prompt(prompt):
    flags = []

    # Length anomaly
    if len(prompt) > 50000:
        flags.append("excessive_length")

    # Known jailbreak patterns
    if any(pattern in prompt.lower() for pattern in JAILBREAK_PATTERNS):
        flags.append("jailbreak_pattern")

    # PII leak
    if contains_pii(prompt):
        flags.append("pii_leak")

    return flags

Detect suspicious outputs

def check_output(response):
    # Detect harmful content
    if contains_harmful(response):
        log_alert("harmful_output", response)
        return None  # Refuse to return

    # Detect PII leak
    if contains_pii(response):
        return redact_pii(response)

    return response

User behavior patterns

Monitor:

• Same user calling frequently → potential abuse / scraping
• Same user activating many tools → possible Agent attack
• Prompt contains “system” or other LLM internal words → potential prompt injection

Auto-log all anomalies + alert.

An Engineering Template

A full “LLM monitoring stack”:

┌─────────────────────────────────────┐
│  User Request                        │
└─────────────┬───────────────────────┘
              ↓
┌─────────────────────────────────────┐
│  Request middleware (input check + log)│
└─────────────┬───────────────────────┘
              ↓
┌─────────────────────────────────────┐
│  LLM call (with tracing)             │
│  - LangSmith / Langfuse SDK         │
└─────────────┬───────────────────────┘
              ↓
┌─────────────────────────────────────┐
│  Response middleware (output check + log)│
└─────────────┬───────────────────────┘
              ↓
┌─────────────────────────────────────┐
│  Storage                             │
│  - Logs → BigQuery / ClickHouse     │
│  - Metrics → Prometheus             │
│  - Traces → Jaeger / OpenTelemetry  │
└─────────────┬───────────────────────┘
              ↓
┌─────────────────────────────────────┐
│  Analytics + Alerts                  │
│  - Grafana dashboard                 │
│  - AlertManager                     │
│  - Business dashboard                │
└─────────────────────────────────────┘

Reflections

”LLM monitoring is 10× more complex than web monitoring”

Why?

• Quality has no simple numeric (vs HTTP 200/500)
• Cost is per-request variable (vs fixed)
• State is multimodal + multi-model + multi-step
• Change model / prompt iterate frequently

LLMOps is a new profession — like DevOps for traditional web. But more complex.

”The cost of not monitoring”

• Errors not caught timely → user attrition
• Cost out of control → end-of-month bill heart attack
• Security incidents → media news
• Quality decline → silent attrition (invisible)

“Monitoring time saved early returns 10× as fix cost later”.

”Starting recommendation”

If you’re just starting an LLM product — Week 1 monitoring:

• Use Langfuse / LangSmith (free tier sufficient)
• Add basic alerts (error rate, latency)
• User thumbs up/down

Don’t wait until launched to realize you need monitoring — by then the data is gone.

Next recommended: L7-04 Quantization or L7-05 Model Deployment.