Structured Output: JSON Mode / Grammars / Pydantic

In production LLM work, 80% of the time the LLM needs to output structured data (not chat text):

• Extract {name, email, skills[]} from a resume
• Classify a user question into {category, priority, sentiment}
• Have an Agent decide next: {action, args}

But LLMs love to “free-form” — they return "{ 'name': 'John', ... }" wrapped in a markdown ```json fence with an extra explanation sentence.

This piece covers: how to make LLMs reliably return structured data.

Three tiers

From lightest to heaviest:

Approach	Strength	Compatibility
Prompt convention	Weak (trust the model)	Any LLM
JSON Mode	Medium (guaranteed JSON, schema not enforced)	OpenAI / Anthropic / Gemini
Schema constraint / Grammars	Strong (schema-conformant)	OpenAI Structured Outputs / some open-source frameworks

1. Prompt convention (baseline)

Simplest: declare the format in the prompt.

prompt = """Extract the following from the resume:
{
  "name": string,
  "email": string,
  "skills": list of strings
}

Output ONLY the JSON, no markdown, no explanation.

Resume: {text}
"""

Problems:

• Model may add a markdown ```json fence
• Fields may be missing
• May add a “Here is the JSON:” sentence

You’ll need post-processing (regex extraction, try/except retry loops).

Good for: prototypes, low-reliability requirements.

2. JSON Mode (guaranteed JSON)

OpenAI, Anthropic, etc. provide response_format:

OpenAI

response = openai.chat.completions.create(
    model="gpt-4",
    messages=[...],
    response_format={"type": "json_object"}
)

Effect: model is forced to return valid JSON. No markdown fence, no extra text.

But: schema is not enforced — field names may drift (emailAddress instead of email), fields may be missing.

You still declare the expected schema in the prompt, but parsing won’t fail.

Anthropic

Claude reaches the same effect via tool_use (treat “output” as “call a function”):

response = anthropic.messages.create(
    model="claude-3-5-sonnet",
    messages=[...],
    tools=[{
        "name": "extract_resume",
        "input_schema": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "email": {"type": "string"},
                "skills": {"type": "array", "items": {"type": "string"}}
            }
        }
    }],
    tool_choice={"type": "tool", "name": "extract_resume"}
)
result = response.content[0].input  # dict directly

tool_choice forces Claude to call this tool — equivalent to enforcing structured output.

3. Strict schema (most reliable)

OpenAI Structured Outputs (2024+)

from pydantic import BaseModel

class Resume(BaseModel):
    name: str
    email: str
    skills: list[str]

response = openai.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[...],
    response_format=Resume
)
result: Resume = response.choices[0].message.parsed

Core idea: OpenAI uses constrained decoding internally — at each token sampling step, only tokens consistent with the schema are allowed.

Effect:

• 100% schema-conformant (vs JSON Mode which only guarantees syntax)
• Field names, types, required-ness all enforced
• Supports nested objects, enums, arrays

Outlines / lm-format-enforcer (open source)

For self-hosted LLMs (vLLM, llama.cpp) you can also enforce strict schemas:

import outlines

model = outlines.models.transformers("Llama-3-8B")

@outlines.json(Resume)
def extract(text):
    return f"Extract resume info: {text}"

result = extract("...")  # guaranteed to be a Resume

Same underlying idea — mask out invalid tokens at each sampling step.

In practice: Pydantic patterns

Cleanest workflow: define schemas with Pydantic, combine with OpenAI Structured Outputs or the instructor library:

from pydantic import BaseModel, Field
from typing import Literal

class TicketClassification(BaseModel):
    category: Literal["billing", "technical", "general"]
    priority: Literal["low", "medium", "high", "urgent"]
    sentiment: Literal["positive", "neutral", "negative"]
    summary: str = Field(description="One sentence summary")
    needs_human: bool

class CustomerTicket(BaseModel):
    classification: TicketClassification
    suggested_response: str
    estimated_resolution_minutes: int

import instructor
from openai import OpenAI

client = instructor.from_openai(OpenAI())

result = client.chat.completions.create(
    model="gpt-4o",
    response_model=CustomerTicket,
    messages=[{"role": "user", "content": ticket_text}]
)
# result.classification.category → "billing"
# result.suggested_response → "..."

80% of “LLM does classification / extraction” production tasks use exactly this pattern.

Nested + lists + conditional

Complex schemas work too:

class Person(BaseModel):
    name: str
    age: int

class Event(BaseModel):
    title: str
    date: str
    attendees: list[Person]
    is_virtual: bool
    meeting_link: str | None = None  # only when is_virtual=True

class Schedule(BaseModel):
    week_of: str
    events: list[Event]

The LLM will automatically respect nested structure — far more robust than regex over free text.

Performance impact

Approach	Speed	Quality
Prompt convention	Fastest	Unstable, needs post-processing
JSON Mode	Fast	Stable, field names may drift
Strict schema	Slightly slower (+5-15%)	100% conformant

Why slower: constrained decoding has to filter candidate tokens at each step. Why worth it: eliminates the entire “parse failure” error class.

Anti-patterns

❌ Regex over free text

# Fragile — breaks if the model changes format slightly
match = re.search(r'name:\s*"([^"]+)"', llm_output)

❌ Retry-until-valid-JSON

for _ in range(5):
    out = llm.chat(...)
    try:
        return json.loads(out)
    except:
        continue
raise Error("Failed 5 times")

Don’t retry — use schema enforcement to get it right the first time.

❌ Schema designed with “free-form text” fields

class Bad(BaseModel):
    response: str  # ← LLM starts free-forming here

Break response further into structured fields like summary / action / details.

3 engineering tips

1. Use Literal instead of str for enums

priority: Literal["low", "medium", "high"]  # ✓
priority: str                                # ✗ model may output "very high"

2. Use Field to describe intent

summary: str = Field(description="One sentence summary, max 100 chars")

This description is passed to the LLM as part of the prompt.

3. Use Optional for “may not exist” fields

phone: str | None = None

Prevents the model from making up a phone number that doesn’t exist.

The 2026 landscape

By current state:

Scenario	Recommended
OpenAI users, want reliability	Structured Outputs + Pydantic
Claude users, want reliability	Tool Use + tool_choice forcing
Self-hosted LLM (vLLM / llama.cpp)	Outlines / instructor
Cross-vendor abstraction	LangChain JsonOutputParser / DSPy
Fast prototyping	Prompt + JSON Mode

Next: L4-15 Prompt Caching or L4-09 Tool Use engineering.

🚧 3 Common Pitfalls