How to Get Reliable JSON From LLMs: JSON Mode vs Structured Outputs vs Constrained Decoding

Short answer: The most reliable way is grammar-constrained decoding via a strict schema mode — OpenAI Structured Outputs, Gemini's responseSchema, or vLLM/Outlines/llama.cpp on self-hosted models. These mask invalid token logits at each decoding step, so the completed output is guaranteed to match your schema. Plain "JSON mode" only guarantees valid syntax, and prompt-only instructions guarantee nothing.

The three mechanisms

There are exactly three levers, in ascending order of reliability:

• Prompt-only JSON. You ask the model in the system prompt to "respond with valid JSON only, no markdown." Nothing enforces it. The sampler is free to emit any token.
• Provider JSON mode. A flag (response_format: {type: "json_object"}) that makes the provider validate the output is syntactically valid JSON before returning it. It does not know or check your fields, types, or required keys.
• Strict Structured Outputs / grammar-constrained decoding. You supply a JSON Schema (or a formal grammar). At every decoding step the engine builds a token mask that zeroes the probability of any token that would make the partial output un-parseable against the grammar. The output cannot violate the schema because invalid continuations are never sampled.

That last mechanism is the important one. OpenAI Structured Outputs, Gemini, and vLLM all implement it by masking invalid token logits so the completed output matches the schema. This is a decoding-time guarantee, not a post-hoc check.

How they compare

What each actually fixes

The everyday failures fall into a few buckets:

• Markdown fences (```json). Models wrap JSON in fences even when explicitly told not to. Prompt-only does not fix this. JSON mode and Structured Outputs both do — the raw response is bare JSON.
• Prose preamble / "Here is your JSON:". Fixed by JSON mode and above; not by prompting.
• JSONL / multiple objects streamed back to back. Prompt-only fails here constantly. JSON mode fixes syntax framing; Structured Outputs fixes it structurally since the grammar only permits one object.
• Wrapper objects ({"result": {...}} when you wanted {...}, or items renamed to data). This is the one JSON mode does not fix — the JSON is valid, just not yours. Only Structured Outputs enforces the exact shape.
• Blank content. Even constrained modes occasionally return empty content (~3.8% in certain tool-call sequences where the model chooses to call a tool or emits nothing). No decoding constraint saves you from an empty completion — you still need a fallback.

Recommended default per stack

• Hosted API (OpenAI/Gemini). Use Structured Outputs / responseSchema with strict: true. Do not settle for plain JSON mode unless the schema genuinely varies at runtime.
• Self-hosted vLLM. Use the guided_json / structured-outputs parameter (backed by xgrammar/Outlines). Same decoding-time guarantee as the hosted APIs, at essentially zero extra infra.
• On-device (llama.cpp / GGUF). Use GBNF grammars or a JSON-schema-to-grammar conversion. This is where constrained decoding matters most, because small local models are the worst at following prompt-only JSON instructions.

A defensive fallback parser

Even with a good mechanism, wrap parsing defensively for the blank-content and fallback-provider cases:

import json, re
 
def parse_llm_json(text: str):
    if not text or not text.strip():
        raise ValueError("empty completion")           # ~3.8% blank case
    # strip ```json ... ``` fences if present
    m = re.search(r"```(?:json)?\s*(.*?)```", text, re.S)
    candidate = m.group(1) if m else text
    # trim to the outermost object/array
    start = min([i for i in (candidate.find("{"), candidate.find("[")) if i != -1])
    end = max(candidate.rfind("}"), candidate.rfind("]"))
    return json.loads(candidate[start:end + 1])

Use this only when the schema-constrained mode is unavailable (older model, third-party gateway, tiny local model). If you control the model, constrain the decoder instead of cleaning up after it.

Reach for schema-constrained decoding first; keep a defensive parser only for the cases where you can't.

Related: more writing.