Model Deployment: From Trained to Production

The dream after training is “let’s ship it”. Reality: deployment is 80% of the engineering.

This piece covers what production LLM serving looks like.

What Production Demands

Need	Why
Latency	Users want answers in milliseconds, not seconds
Throughput	Many requests in parallel
Availability	99.9%+ uptime
Cost	Per-request economics that work
Scaling	Handle traffic spikes
Observability	Know when things break
Safety	Filter harmful content + abuse

Hitting all of these at once is hard.

Architectural Patterns

Pattern 1: API Wrapper

The simplest:

[Your app] → [OpenAI / Anthropic API] → [Their inference]

Pros: zero ML ops, fastest to ship Cons: rate limits, cost, vendor lock-in, latency Best for: prototypes, low-volume, MVPs

Pattern 2: Self-Hosted (Single GPU)

[Your app] → [vLLM / TGI] → [Your GPU(s)]

Pros: control, cost at scale, customization Cons: ops team needed, capacity planning, ongoing maintenance Best for: 10M+ tokens/day, sensitive data

Pattern 3: Hybrid

[Your app] → [Router] → ┌─→ [API for high-quality]
                       └─→ [Self-hosted for high-volume routine]

Pros: cost-efficient + quality Cons: more complex Best for: production at scale

Deployment Stack

A typical self-hosted setup:

┌─────────────────────────────────┐
│        Load Balancer             │
│       (NGINX / Cloudflare)       │
└──────────────┬───────────────────┘
               ↓
┌─────────────────────────────────┐
│        API Gateway               │
│   (auth, rate limit, logging)    │
└──────────────┬───────────────────┘
               ↓
┌─────────────────────────────────┐
│      Inference Servers           │
│   (vLLM / TGI / TensorRT-LLM)    │
│   - 4× GPU node                  │
│   - replicated for HA            │
└──────────────┬───────────────────┘
               ↓
┌─────────────────────────────────┐
│      Storage / Cache              │
│   - Redis for sessions           │
│   - Vector DB for RAG             │
└──────────────┬───────────────────┘
               ↓
┌─────────────────────────────────┐
│      Monitoring / Telemetry      │
│   (Prometheus / Grafana / Datadog) │
└─────────────────────────────────┘

Inference Server Choices

vLLM (most popular open source)

Berkeley’s. Easy + fast.

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3.3-70B-Instruct")
sampling_params = SamplingParams(temperature=0.7, max_tokens=200)
outputs = llm.generate(["Hello"], sampling_params)

Or as a server:

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.3-70B-Instruct \
    --tensor-parallel-size 4 \
    --port 8000

OpenAI-compatible endpoint. Drop-in replacement.

TGI (Text Generation Inference)

HuggingFace’s. Similar to vLLM:

docker run --gpus all --shm-size 1g \
    -p 8080:80 \
    ghcr.io/huggingface/text-generation-inference:latest \
    --model-id meta-llama/Llama-3.3-70B-Instruct

TensorRT-LLM

NVIDIA. Highest performance, hardest to set up.

SGLang

UC Berkeley’s newer effort. Smart caching, growing fast.

Ollama / LM Studio

For personal / dev use, not production.

Hardware Choices

For self-hosted serving:

Inference-optimized GPUs

GPU	VRAM	Best for
A10	24GB	Small models (7B) at low cost
L40S	48GB	Mid-size (13B) cost-efficient
A100 80GB	80GB	70B with quantization
H100	80GB	70B with quantization + FP8
H200	141GB	70B without quantization
B200	192GB	Frontier inference

Cloud providers

• AWS: SageMaker, EC2 P-instances
• GCP: Vertex AI, A2/A3 instances
• Azure: ML Studio
• CoreWeave / Lambda Labs: GPU-specialized, often cheaper
• Modal: serverless GPU
• Replicate / Together: managed inference

Scaling

Vertical scaling

Bigger GPUs. Eventually hit physical limits.

Horizontal scaling

Multiple servers behind a load balancer.

Challenge: each server needs the full model loaded — GBs of memory per replica.

Autoscaling

# Kubernetes HPA based on QPS
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Pods
    pods:
      metric:
        name: queue_depth
      target:
        type: AverageValue
        averageValue: "5"

Tricky: GPU instances cold-start in ~1 min (model load). Smooth bursts with buffer capacity.

Latency Optimization

Two latency metrics:

• TTFT (Time to First Token): how long before user sees anything
• TBT (Time Between Tokens): how fast tokens stream after first

TTFT: dominated by prompt processing

• Use FlashAttention
• Prompt caching (Anthropic / OpenAI / vLLM)
• Reduce prompt length

TBT: dominated by per-token forward

• Better hardware
• Speculative decoding
• Smaller model where possible

Streaming

Always stream user-facing responses:

# Server
@app.get("/chat")
async def chat(prompt: str):
    async def generate():
        async for chunk in llm.stream(prompt):
            yield f"data: {chunk}\n\n"
    return StreamingResponse(generate(), media_type="text/event-stream")

User sees the first word instantly, even if full response takes 5 seconds.

Reliability

Health checks

@app.get("/health")
def health():
    if model_loaded and gpu_responsive():
        return {"status": "ok"}
    raise HTTPException(503)

Graceful degradation

If primary GPU goes down, route to:

• Backup GPU
• Smaller model fallback
• API service

Circuit breakers

If LLM responses are erroring repeatedly, stop trying for N seconds.

Retry logic

Transient errors (timeouts, rate limits) → retry with backoff.

Cost Engineering

1. Right-size your fleet

Don’t over-provision. Monitor utilization:

• Target 60-80% average GPU utilization
• Below 30%: too many GPUs
• Above 90%: about to fall over on spikes

2. Spot / preemptible instances

Cloud spot GPUs are 70% cheaper. Risk: can be killed anytime.

Use for:

• Batch jobs (training, evals)
• Auxiliary services
• Not your primary inference (unless you handle the unreliability)

3. Caching

Semantic cache + prompt cache (see L7-03).

4. Quantization

INT4 → 4× more requests per GPU.

Safety in Production

Input filtering

Block:

• Known prompt injection patterns
• Adult content
• Personal info exfiltration
• Abuse / harassment

Output filtering

Don’t let the model:

• Generate harmful content
• Leak system prompts
• Reveal training data

Tools:

• Llama Guard (open-source classifier)
• Azure Content Safety
• Anthropic’s Constitutional Classifiers

Rate limiting

Per-user, per-IP, per-API-key:

@rate_limit("1000/hour")
def chat_endpoint():
    ...

Stops abuse + accidental cost blowups.

Monitoring

Track:

• Latency (p50, p95, p99)
• Throughput (QPS, tokens/sec)
• Error rate
• Cost (per request, daily)
• GPU utilization (memory, compute)
• Cache hit rate
• User satisfaction (thumbs up/down)
• Output quality samples (LLM-as-judge sampling)

Tools: Prometheus + Grafana, Datadog, Langfuse, custom.

A Concrete Deployment Story

A startup deploys their LLM chatbot:

Week 1: OpenAI API + Vercel = working MVP Month 1: 10k users, $5k/month bill, OpenAI rate limits hit **Month 2**: switch to Anthropic + Cloudflare Workers, caching =$2k/month Month 3: 100k users, $20k/month, decide to self-host **Month 4**: 4× A100 cluster + vLLM =$5k/month at 100k users Month 6: 1M users, multi-region deployment, fall-back to API for burst Year 1: 10M users, mature ops, mixed routing, ~$50k/month

The journey from API to self-hosted has clear inflection points around volume.

Next recommended: L7-06 Advanced Training or L7-07 Monitoring.