Constrained Generation: Outlines, JSON Mode, and Structured Output That Works

Every LLM application eventually needs structured output. The agent that calls tools needs parseable JSON. The extraction pipeline needs consistent field names. The classification step needs one of exactly three strings. And every one of these applications, if built naively, eventually breaks in production on a malformed response that looked fine in development.

There are two fundamentally different approaches to getting structured output from an LLM: constrain the generation at the token level so invalid output is physically impossible, or prompt the model to produce the right format and fix it afterward. The second approach—call it “JSON mode”—is how most teams start. It fails often enough to cause real problems, in ways that are difficult to catch in testing and painful to debug in production.

This post covers how token-level constrained generation works, the tools available to do it, the failure modes of each, and the practical schema patterns that make structured output reliable regardless of which approach you use.

Why Prompt-Only JSON Fails

When you ask an LLM to “return valid JSON with fields X, Y, Z,” you are relying on the model having internalized a perfect JSON grammar during training and reliably applying it under all conditions. This works most of the time—which is the problem. “Most of the time” means:

• Trailing commas in objects and arrays (valid in JavaScript, invalid in JSON)
• Unescaped quotes inside string values
• Missing quotes around keys
• Truncated output when generation hits the token limit mid-structure
• Extra prose before or after the JSON block
• Nested objects where the schema expected a string
• Numbers formatted as strings or strings formatted as numbers
• null where the schema required a specific string, or vice versa

Measured failure rates from production systems:

• Prompt-only JSON extraction: 5–20% parse failure rate
• OpenAI JSON mode (server-side hint, not guaranteed): 2–5% parse failure rate
• OpenAI Structured Outputs (schema-constrained): <0.1% failure rate
• vLLM guided_json with XGrammar: <0.01% failure rate (schema violations are impossible by construction)

The variance in the prompt-only failure rate is large because it depends heavily on model size, the complexity of the schema, and whether the schema matches the model’s training distribution. A simple flat schema with three string fields fails rarely; a deeply nested schema with conditional requirements fails constantly.

Retry loops are the common mitigation. When the parse fails, send the error message back to the model and ask it to fix its output. This works—and it hides the failure rate from your error metrics while adding latency and cost to every failed call. A 10% failure rate with one retry becomes a 10% latency spike plus 1% double-cost on every request.

The correct fix is to use constrained generation so that invalid output cannot be produced.

How Token-Level Constrained Generation Works

An LLM generates text by computing a probability distribution over all vocabulary tokens at each step and sampling from that distribution. Constrained generation works by masking the distribution at each step: tokens that would violate the current constraint are set to probability zero before sampling. The model can only sample from the valid subset.

The constraint is compiled into a finite-state machine (FSM) before generation begins. The FSM tracks which characters are valid at each position in the output, given the characters already produced. At each generation step, the FSM’s current state maps to a set of valid next characters, which maps to a set of valid next tokens. All other tokens are masked.

Constrained generation for the pattern {"status": "<one of: ok|error|pending>"}

  After "{"status": "":
  FSM state: inside string value, first character must be one of [o, e, p]
  Valid tokens: any token whose text starts with "o", "e", or "p"
                (including "ok", "ok\"", "error", "err", "pending", "pe", etc.)
  Masked tokens: everything else — including tokens starting with whitespace,
                 digits, uppercase, other letters, punctuation

  After "{"status": "ok":
  FSM state: inside string value, next character must be closing quote
  Valid tokens: only tokens that begin with '"'
  Masked tokens: everything else

  After "{"status": "ok"":
  FSM state: object, after key-value pair, next must be "}" or ","
  ...and so on

This approach is lossless in a specific sense: the model’s output preferences among valid tokens are unchanged. The relative probabilities of "ok" vs "error" vs "pending" are exactly what the model assigned before masking—only the invalid tokens have been removed. There is no quality degradation for outputs that conform to the constraint; the model’s reasoning and language generation are unaffected.

Building the FSM

For a regular expression constraint, the FSM is built by standard regex-to-NFA-to-DFA compilation, then intersected with the tokenizer’s vocabulary to produce a token-level transition table. At each FSM state, the precomputed table maps directly to the set of valid next tokens.

For a JSON schema, the schema is compiled into an EBNF grammar, which is compiled into a pushdown automaton (PDA)—essentially a stack-augmented FSM that can handle recursive structures like nested objects and arrays. The PDA is then mapped to the tokenizer vocabulary.

The compilation step is expensive (milliseconds to seconds for complex schemas) but happens once per unique schema. The per-token masking step at inference time is fast—XGrammar achieves under 40 microseconds per token by precomputing bitmask tables for context-independent tokens (roughly 99% of the vocabulary) and handling the small context-dependent remainder at runtime.

FSM compilation pipeline:

  Constraint (regex / JSON Schema / EBNF)
        |
        v
  Grammar / NFA
        |
        v
  Deterministic FSM / PDA
        |
        v
  Token-level transition table (vocabulary intersection)
        |
        v
  Per-step logit mask (zero out invalid token logits)
        |
        v
  Sample from masked distribution

Outlines: The Reference Implementation

Outlines (dottxt-ai) is the most widely adopted Python library for constrained generation. It integrates with HuggingFace Transformers, vLLM, TGI, SGLang, and other backends, and supports four constraint types: regex, JSON schema, Pydantic model, and EBNF grammar.

Installation and Basic Usage

1

pip install outlines

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

import outlines
from pydantic import BaseModel
from typing import Literal
from enum import Enum

# Load a local model
model = outlines.models.transformers(
    "meta-llama/Llama-3.2-3B-Instruct",
    device="cuda",
)

# --- Regex constraint ---
# Match an IP address exactly
ip_generator = outlines.generate.regex(
    model,
    r"(\d{1,3}\.){3}\d{1,3}",
)
ip = ip_generator("What is the loopback address?")
# ip is guaranteed to be a valid IP address string

# --- Choice constraint ---
# One of a fixed set of strings
sentiment_generator = outlines.generate.choice(
    model,
    ["positive", "negative", "neutral"],
)
result = sentiment_generator("Classify: 'The food was decent but overpriced.'")
# result is exactly one of the three strings

# --- Pydantic model constraint ---
class ExtractionResult(BaseModel):
    company_name: str
    founded_year: int
    headquarters: str
    industry: Literal["software", "hardware", "services", "other"]

extractor = outlines.generate.json(model, ExtractionResult)
result = extractor(
    "Extract company info from: Stripe was founded in 2010 in San Francisco "
    "and provides payment processing software."
)
# result is a validated ExtractionResult instance
print(result.company_name)   # "Stripe"
print(result.founded_year)   # 2010

Regex for Pattern-Constrained Fields

Regex constraints are particularly useful for fields with fixed formats: phone numbers, ISO dates, identifiers, IP addresses, semantic version strings.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Extract a structured log entry
log_entry_pattern = (
    r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z "  # timestamp
    r"(DEBUG|INFO|WARN|ERROR) "                  # level
    r"\[[\w\-\.]+\] "                           # component
    r".{10,200}"                                # message (10-200 chars)
)

log_generator = outlines.generate.regex(model, log_entry_pattern)
entry = log_generator("Generate a sample ERROR log entry for the auth service.")

One subtlety: the regex must match the entire output, not just contain a match. Outlines compiles the regex as a full-string match (anchored at start and end). Design your regex to allow the complete expected output, including any surrounding structure.

EBNF Grammar Constraints

For structures that regular expressions cannot express—balanced parentheses, proper nesting, valid SQL, valid Python expressions—Outlines supports EBNF grammar constraints via a pushdown automaton.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Constrain output to a simple arithmetic expression grammar
arithmetic_grammar = """
    expr: term (("+" | "-") term)*
    term: factor (("*" | "/") factor)*
    factor: NUMBER | "(" expr ")"
    NUMBER: /\d+(\.\d+)?/
    %ignore /\s+/
"""

generator = outlines.generate.cfg(model, arithmetic_grammar)
expression = generator("Write the expression for compound interest: P*(1+r)^n")

Grammar-based constraints are more expensive to compile and slower per token than regex constraints (the PDA is more complex than an FSM). Use them when the output structure is recursive or context-sensitive.

XGrammar: The Production Backend

XGrammar is the structured generation backend shipped by default in vLLM (v0.6+), SGLang, and TensorRT-LLM. It implements the same FSM/PDA approach as Outlines but is engineered for throughput rather than API ergonomics.

The key optimization is vocabulary partition. XGrammar divides the tokenizer’s vocabulary into two sets:

• Context-independent tokens: tokens whose validity at a given FSM state depends only on the state, not on what characters follow. For most grammars, ~99% of tokens fall here. These are precomputed into bitmask tables indexed by FSM state.
• Context-dependent tokens: tokens that span a transition point in the grammar (e.g., a token that contains both the closing " of a string and the , starting the next key). These require runtime evaluation.

The result: per-token masking costs under 40 microseconds even for complex JSON schemas, with the bitmask table lookup being a CPU cache-resident O(1) operation.

vLLM Guided Decoding API

vLLM exposes constrained generation through the guided_* parameters in SamplingParams. These are passed through to XGrammar (or Outlines as fallback for unsupported constraints).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

from vllm import LLM, SamplingParams
import json

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")

# --- guided_choice: one of a fixed set ---
params = SamplingParams(
    guided_choice=["approved", "denied", "pending_review"],
    temperature=0,
)
output = llm.generate(
    ["Classify this loan application: applicant has 750 credit score, stable income."],
    params,
)[0].outputs[0].text
# output is exactly one of the three strings

# --- guided_regex: regex-constrained output ---
params = SamplingParams(
    guided_regex=r"\d{4}-\d{2}-\d{2}",   # ISO date
    temperature=0,
)
output = llm.generate(["What date was the Apollo 11 landing?"], params)[0].outputs[0].text
# output is a string matching YYYY-MM-DD

# --- guided_json: JSON schema constraint ---
schema = {
    "type": "object",
    "properties": {
        "name":       {"type": "string"},
        "age":        {"type": "integer", "minimum": 0, "maximum": 150},
        "occupation": {"type": "string"},
        "city":       {"type": "string"},
    },
    "required": ["name", "age", "occupation", "city"],
    "additionalProperties": False,
}

params = SamplingParams(
    guided_json=json.dumps(schema),
    temperature=0.3,
)
output = llm.generate(
    ["Extract: Dr. Sarah Chen, 42, cardiologist, practices in Boston"],
    params,
)[0].outputs[0].text
result = json.loads(output)   # guaranteed to succeed

OpenAI-Compatible API (vLLM Server)

When serving via vllm serve, the same constraints are available through the OpenAI-compatible endpoint:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

from openai import OpenAI
import json

client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")

schema = {
    "type": "object",
    "properties": {
        "sentiment":   {"type": "string", "enum": ["positive", "negative", "neutral"]},
        "confidence":  {"type": "number", "minimum": 0.0, "maximum": 1.0},
        "key_phrases": {"type": "array", "items": {"type": "string"}, "maxItems": 5},
    },
    "required": ["sentiment", "confidence", "key_phrases"],
    "additionalProperties": False,
}

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Analyze sentiment: 'The deployment went smoothly but the docs are terrible.'"}],
    extra_body={"guided_json": json.dumps(schema)},
    temperature=0,
)
result = json.loads(response.choices[0].message.content)

vLLM CLI with Structured Output

1
2
3
4
5

# Start vLLM server with guided decoding enabled (default in v0.6+)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --guided-decoding-backend xgrammar \  # or "outlines" for fallback
  --max-model-len 8192 \
  --port 8000

Instructor: OpenAI API Patching

Instructor is the right tool when you are using a provider API (OpenAI, Anthropic, Mistral, Cohere) rather than a self-hosted model. It patches the client to add Pydantic-validated structured output with automatic retry on validation failure.

1

pip install instructor

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

import instructor
from openai import OpenAI
from pydantic import BaseModel, Field
from typing import Optional, List

client = instructor.from_openai(OpenAI())

class ExtractedEvent(BaseModel):
    event_name: str = Field(description="Name of the event")
    date: Optional[str] = Field(description="Date in YYYY-MM-DD format, or None if not mentioned")
    location: Optional[str] = Field(description="City and country, or None if not mentioned")
    attendee_count: Optional[int] = Field(description="Approximate number of attendees, or None")
    topics: List[str] = Field(description="Main topics discussed", max_length=10)

event = client.chat.completions.create(
    model="gpt-4o",
    response_model=ExtractedEvent,
    messages=[{
        "role": "user",
        "content": (
            "Extract event info: KubeCon North America 2024 drew around 12,000 attendees "
            "to Chicago in November 2024, focusing on Kubernetes, service mesh, and GitOps."
        ),
    }],
)

print(event.event_name)       # "KubeCon North America 2024"
print(event.attendee_count)   # 12000
print(event.topics)           # ["Kubernetes", "service mesh", "GitOps"]

Instructor uses the provider’s native tool-calling mechanism when available (OpenAI function calling, Anthropic tool use) and falls back to JSON mode with Pydantic validation plus retry. When validation fails, it automatically re-sends the error message and asks the model to correct its output—up to a configurable max_retries.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# With retries and validation
from pydantic import field_validator

class StrictDate(BaseModel):
    date: str

    @field_validator("date")
    @classmethod
    def validate_iso_format(cls, v):
        from datetime import date
        try:
            date.fromisoformat(v)
        except ValueError:
            raise ValueError(f"Must be YYYY-MM-DD format, got: {v}")
        return v

# Instructor will retry up to 3 times if validation fails
result = client.chat.completions.create(
    model="gpt-4o",
    response_model=StrictDate,
    messages=[{"role": "user", "content": "When was Python 3.12 released?"}],
    max_retries=3,
)

Instructor vs Token-Level Constraints

Instructor’s retry approach is not the same as token-level constrained generation:

• Instructor works post-generation: produce output, validate, retry if wrong. The model can still generate invalid JSON; Instructor catches it. Works with any provider API, no server-side changes needed.
• Constrained generation works pre-generation: invalid tokens are masked before sampling. Invalid output is physically impossible. Requires a compatible inference backend (vLLM, llama.cpp, Outlines).

For production API usage where you do not control the inference server, Instructor is the right answer. For self-hosted inference where reliability and eliminating retry overhead matter, use vLLM guided decoding.

Schema Design for Reliable Structured Output

The schema you define matters as much as the generation technique. Poorly designed schemas fail even with token-level constraints—either because the constraint is unsatisfiable for the model’s understanding of the task, or because the schema includes features the engine does not support.

Use Flat Schemas When Possible

Every level of nesting multiplies the complexity of the FSM. A flat schema with simple scalar types compiles instantly and has near-zero masking overhead. A deeply nested schema with arrays of objects with conditional fields is slow to compile and creates more opportunities for the model to produce technically valid JSON that is semantically wrong.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Prefer this (flat)
class CustomerRecord(BaseModel):
    customer_id: str
    name: str
    email: str
    tier: Literal["free", "pro", "enterprise"]
    account_age_days: int

# Over this (nested, complex)
class CustomerRecord(BaseModel):
    customer: dict    # open-ended — no benefit from constraints
    metadata: dict    # same problem

Mark All Fields Required

Optional fields that are absent cause subtle bugs downstream: code checks result.field and gets None where it expected a string. Make fields required in the schema and use explicit sentinel values instead.

1
2
3
4
5

# Explicit sentinel rather than None
class ClassificationResult(BaseModel):
    category: str
    confidence: float
    reason: str = "unknown"   # default instead of Optional[str]

Close the Schema with additionalProperties: False

Without this, constrained generation may allow the model to append extra fields the parser does not expect. Set it explicitly.

1
2
3
4
5
6

from pydantic import BaseModel, ConfigDict

class StrictModel(BaseModel):
    model_config = ConfigDict(extra="forbid")   # Pydantic equivalent
    name: str
    value: float

The JSON Schema equivalent:

1
2
3
4
5
6
7
8
9

{
  "type": "object",
  "properties": {
    "name": {"type": "string"},
    "value": {"type": "number"}
  },
  "required": ["name", "value"],
  "additionalProperties": false
}

Enums Over Open Strings for Classification

When the output must be one of a fixed set, use Literal or Enum rather than str. This gives the constraint engine the smallest possible valid set at that position, maximizing masking efficiency and eliminating the model’s ability to produce creative variations (“positive”, “Positive”, “POSITIVE”, “pos”, “positively”).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

from typing import Literal
from enum import Enum

# Literal (simpler)
class SentimentResult(BaseModel):
    sentiment: Literal["positive", "negative", "neutral"]
    score: float

# Enum (reusable across models)
class Priority(str, Enum):
    low = "low"
    medium = "medium"
    high = "high"
    critical = "critical"

class TicketClassification(BaseModel):
    priority: Priority
    category: Literal["bug", "feature", "question", "billing"]
    requires_human: bool

Avoid Schema Features Without Engine Support

XGrammar (as of early 2026) does not support: minLength, maxLength, minItems, maxItems, pattern (regex within a property), oneOf/anyOf with complex schemas, $ref references. Using unsupported features causes the engine to fall back to prompt-only generation, silently removing your constraint guarantee.

Check your engine’s supported feature list before relying on these in production:

• XGrammar: JSON Schema basic types, enum, required, additionalProperties
• Outlines: JSON Schema + regex properties + EBNF grammars
• Guidance/llguidance: broadest JSON Schema support, including many constraint types XGrammar skips

Constrained Generation for Agent Tool Calls

Structured output is not just for extraction pipelines. Agent tool calls are the most critical application: when an LLM is supposed to emit {"tool": "search", "query": "current gold price"}, a malformed output means the tool is not called and the agent is stuck.

The standard approach with OpenAI-compatible tool calling uses provider-managed function calling, which is effectively constrained generation on the server side. For self-hosted models without first-class tool-calling support, you can implement it explicitly.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

from vllm import LLM, SamplingParams
from pydantic import BaseModel
from typing import Union, Literal
import json

# Define tool call schemas
class SearchTool(BaseModel):
    tool: Literal["search"]
    query: str

class CalculateTool(BaseModel):
    tool: Literal["calculate"]
    expression: str

class FinalAnswer(BaseModel):
    tool: Literal["final_answer"]
    answer: str

# Build a union schema that matches any tool call
tool_schema = {
    "oneOf": [
        SearchTool.model_json_schema(),
        CalculateTool.model_json_schema(),
        FinalAnswer.model_json_schema(),
    ]
}

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")

def run_agent_step(context: str) -> dict:
    params = SamplingParams(
        guided_json=json.dumps(tool_schema),
        temperature=0,
        max_tokens=256,
    )
    output = llm.generate([context], params)[0].outputs[0].text
    return json.loads(output)   # guaranteed parseable, validated against schema

# Agent loop
context = "Available tools: search(query), calculate(expression), final_answer(answer)\n\n"
context += "User: What is 15% of the current population of France?"

for step in range(5):
    tool_call = run_agent_step(context)
    print(f"Step {step}: {tool_call}")

    if tool_call["tool"] == "final_answer":
        print(f"Answer: {tool_call['answer']}")
        break

    # Execute tool and append result to context
    result = execute_tool(tool_call)   # your tool execution logic
    context += f"\nAssistant: {json.dumps(tool_call)}"
    context += f"\nTool result: {result}"
    context += "\nAssistant:"

The guarantee here is significant: the agent loop cannot crash due to a malformed tool call. Every output is valid JSON matching one of the defined tool schemas. Invalid tool calls are not attempted.

Latency Overhead

Constrained generation adds overhead at two points: schema compilation (once per unique schema) and per-token masking (every step).

Structured generation overhead (approximate, Llama 3.1 8B on A100):

  Method                   | Compilation | Per-token overhead | Notes
  -------------------------|-------------|--------------------|-----------------------
  No constraint (baseline) | 0 ms        | 0 µs               | Reference
  XGrammar JSON schema     | 5–50 ms     | 15–40 µs           | Cached after first use
  Outlines regex           | 50–500 ms   | 30–80 µs           | Depends on regex size
  Outlines JSON schema     | 100–2000 ms | 40–100 µs          | Complex schemas slower
  Outlines EBNF grammar    | 500–5000 ms | 80–200 µs          | PDA more expensive

Schema compilation is cached by both XGrammar and Outlines. In a serving context where the same schema is used repeatedly (every tool call uses the same format, every extraction uses the same Pydantic model), the compilation cost amortizes quickly.

Per-token overhead of 15–100 µs is negligible relative to the 5–50 ms per-token generation time at batch size 1. At high batch sizes, the masking work runs in parallel with the GPU compute and is effectively free.

The real latency cost of constrained generation is when the constraint is very restrictive and the model struggles—for example, constraining output to a 6-character alphanumeric code when the model has no strong prior for that format. The model may generate many tokens that are immediately masked, slowing convergence. This is a sign the prompt needs to provide more guidance, not that constrained generation is wrong.

When to Use Each Approach

Structured output approach decision guide:

  Scenario                               | Recommended approach
  ---------------------------------------|------------------------------------------
  Self-hosted inference, reliability     | vLLM guided_json (XGrammar backend)
  critical (agents, pipelines)           |
                                         |
  Provider API (OpenAI, Anthropic)       | Instructor + Pydantic (retry on failure)
  with complex retry logic needed        | or provider native structured outputs
                                         |
  Simple classification / choice         | guided_choice or Literal enum + vLLM
  (one of N strings)                     |
                                         |
  Pattern-constrained fields             | guided_regex or Outlines regex
  (dates, IDs, phone numbers)            |
                                         |
  Sub-3-bit JSON, complex nesting        | Outlines (broader schema support)
  with pattern/length constraints        | or Guidance/llguidance
                                         |
  Valid code generation (SQL, Python     | Outlines EBNF grammar constraint
  expressions, DSL)                      |
                                         |
  Rapid prototyping / any provider       | Instructor (easiest setup)
  API surface                            |
                                         |
  Agent tool calls, self-hosted          | vLLM guided_json with union schema

The common case—self-hosted inference with a defined output schema—is straightforwardly served by vLLM’s guided decoding. The configuration is minimal, the quality guarantee is absolute, and the overhead is negligible.

Practical Production Setup

vLLM with XGrammar (Self-Hosted)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Start server
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --guided-decoding-backend xgrammar \
  --max-model-len 8192

# Test structured output
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Extract: John Smith, 34, software engineer in Seattle"}],
    "guided_json": "{\"type\":\"object\",\"properties\":{\"name\":{\"type\":\"string\"},\"age\":{\"type\":\"integer\"},\"title\":{\"type\":\"string\"},\"city\":{\"type\":\"string\"}},\"required\":[\"name\",\"age\",\"title\",\"city\"],\"additionalProperties\":false}",
    "temperature": 0
  }'

Outlines with a Local Model

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

import outlines
from pydantic import BaseModel
from typing import Literal

model = outlines.models.vllm("meta-llama/Llama-3.1-8B-Instruct")

class IncidentReport(BaseModel):
    severity: Literal["P0", "P1", "P2", "P3"]
    service: str
    summary: str
    customer_facing: bool
    estimated_duration_minutes: int

generator = outlines.generate.json(model, IncidentReport)

report = generator(
    "Classify this incident: Payment processing is completely down, "
    "affecting all customers, started 15 minutes ago."
)
# report.severity == "P0", report.customer_facing == True, etc.

Instructor with OpenAI (Provider API)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

import instructor
from anthropic import Anthropic
from pydantic import BaseModel, Field
from typing import List

# Works with Anthropic too
client = instructor.from_anthropic(Anthropic())

class CodeReviewComment(BaseModel):
    file: str
    line: int
    severity: Literal["error", "warning", "suggestion"]
    message: str = Field(min_length=10, max_length=500)

class CodeReview(BaseModel):
    summary: str
    comments: List[CodeReviewComment]
    approved: bool

review = client.messages.create(
    model="claude-opus-4-7",
    max_tokens=1024,
    response_model=CodeReview,
    messages=[{
        "role": "user",
        "content": f"Review this code:\n\n```python\n{code}\n```",
    }],
)

Constrained generation removes an entire class of production failure. Parse errors in extraction pipelines, malformed tool calls in agent loops, inconsistent field names across requests—these are eliminated rather than mitigated. The tooling is mature, the configuration overhead is minimal, and the per-token overhead is negligible at any realistic serving load. The prompt-only approach should be treated as a prototype technique that gets replaced before production, not a feature to be maintained and debugged indefinitely.