JSON Mode

JSON mode changes the decoder contract from free text to valid JSON text. In OpenAI Chat Completions, that means setting response_format to an object with type json_object. In Azure OpenAI, the same family of model APIs exposes the same response_format concept. In Gemini, the closest syntax-oriented setting is a JSON response MIME type such as application/json, while a response schema moves closer to structured outputs. For providers without a dedicated JSON flag, tool use or schema output is usually more reliable than prompt-only JSON.

The key word is syntax. The response should parse as JSON, but the object can still be semantically wrong for your application. It might be an array instead of an object, include a string where your code expects a number, or use confidence_label instead of confidence. JSON mode makes parsing possible; validation decides whether the parsed value is acceptable.

The API flag does not describe your business contract. You still need prompt instructions that say the response must be JSON and list the fields, allowed values, and constraints. OpenAI JSON mode is especially strict about this pattern: the request context should mention JSON, because otherwise the model may produce unhelpful whitespace until it hits the token limit or the API may reject the request on some surfaces.

A good prompt says more than return JSON. It describes the keys, types, enum values, missing-evidence behavior, and whether extra fields are allowed. This reduces schema drift, makes repair prompts easier, and gives the validator a clear target.

Structured outputs use a schema to constrain the generation more tightly. With strict structured output, the model is expected to produce data matching the schema: required fields, nested objects, arrays, enum values, and types. JSON mode only says the bytes should form valid JSON.

In an interview, this distinction is often the crux. If the task is low-risk extraction where you can tolerate occasional repair, JSON mode plus validation can be enough. If the output drives automation, payments, policy decisions, database writes, or user-visible state, structured outputs are usually the stronger default because the schema is part of the generation contract.

JSON mode cannot make an impossible token budget work. If max_tokens or max_output_tokens is too small, the model may be cut off in the middle of an object. Providers usually expose a finish reason such as length to indicate that generation stopped because the token limit was reached.

A robust client checks finish_reason before parsing. If the finish reason is length, treat the response as incomplete even if a partial prefix looks promising. Retry with a larger output budget, ask for a shorter object, or split the task into smaller calls. Do not pass a truncated object to a best-effort parser and hope it means the same thing.

After the call, parse with a real JSON parser and validate with a schema library. In Python, a Pydantic model can enforce required fields, field types, length limits, numeric ranges, and enum choices. In TypeScript, Zod can do the same before the result reaches application logic.

Validation should be explicit and observable. Log the prompt version, model, finish reason, parse error, validation error, and repair outcome. Then decide whether to retry, repair, fall back to a human-readable response, or fail closed. JSON mode is one layer in a reliability stack, not a substitute for typed boundaries.

Sometimes a model returns prose wrapped around JSON, a fenced-looking block from a prompt habit, or a partial object from truncation. Recovery options include stripping text before the first opening brace and after the last closing brace, retrying the original request with clearer instructions, or sending a repair prompt that asks the model to convert the bad output into one valid object.

Repair should be bounded. Allow a small number of attempts, validate the repaired object, and stop if it still fails. Unlimited repair loops hide product bugs, increase cost, and can turn a simple extraction feature into a slow, unpredictable workflow.

OpenAI JSON mode is commonly configured with response_format type json_object. Azure OpenAI follows the same pattern for supported models. Gemini supports JSON-oriented generation through response MIME type and can add response schemas for schema control. Anthropic workflows often use tool definitions or explicit prompting with validation rather than a syntax-only flag on every surface.

The portable design is to separate concerns: prompt for the object, request the strongest provider constraint available, parse with a standard parser, and validate in your code. This makes the application resilient when you switch models or providers.

JSON mode can return a valid object that silently changes your contract. A field named priority can become urgency, a numeric confidence can become high, or citations can move from an array to a comma-separated string. These failures are dangerous because they parse successfully and then break business logic later.

Schema validation catches drift at the boundary. Treat validation failures as first-class outcomes, not rare exceptions. The validator should produce clear errors that can be sent to a repair prompt or used to improve the original prompt.

Malformed output means the model finished but the text is not usable JSON or does not validate. Truncation means generation was forcibly stopped before the object completed. The recovery strategies differ. Malformed output can be repaired. Truncated output should usually be regenerated, because missing tail fields may change the meaning of the object.

Always inspect finish_reason. A partial object with a closing brace inserted by a repair routine may look valid, but it can hide missing evidence, incomplete arrays, or half-written strings. Increase the token budget or reduce the requested payload before trying semantic repair.

The best repair is the one you never need. List fields in the prompt in the same order your schema expects. Provide enum values directly. Tell the model what to do when information is missing, such as using null for optional fields or an explicit unknown label. Avoid asking for explanatory prose in the same response when downstream code expects only JSON.

Examples can help when the shape is unusual, but they also consume context. For beginner JSON mode tasks, a concise field contract and low temperature often matter more than many examples.

JSON mode does not stop prompt injection, data exfiltration attempts, or unsafe tool arguments. An attacker can still put malicious text inside a field value, or try to make the model output a dangerous action encoded as JSON. Treat model output as untrusted input until validated and authorized.

If the JSON object will trigger side effects, combine validation with allowlists, authorization checks, and human review for risky actions. Tool calling is often safer for action selection because the application owns the tool schema and execution policy.

Log the response_format or equivalent provider setting, prompt version, model version, finish reason, parse result, validation result, retry count, and repair result. Aggregate these by use case so you can see whether one prompt or model is responsible for most JSON failures.

Use a bounded retry policy. A common pattern is one regeneration for truncation with a larger token budget, one repair attempt for malformed or wrapped output, and then a safe fallback. The fallback should be explicit: return a validation_error status, route to manual review, or ask the user to narrow the input.

JSON mode usually has small overhead, but retries, repair calls, and overly large objects can double latency and token use. Keep JSON fields compact, avoid long natural-language explanations inside objects, and separate large summaries from strict extraction when needed.

Design the application so provider JSON mode is behind an adapter. The adapter can choose OpenAI response_format, Gemini response MIME type, Anthropic tool use, or structured outputs. Keep the application validator stable even if the provider feature changes.

This example uses OpenAI-style JSON mode for syntax, then Pydantic for the actual schema. The repair path is only used after parsing or validation fails, and the repaired result is validated again before use.

This helper layer is provider-neutral. It separates truncation handling, wrapper stripping, and schema validation so failures are observable instead of being hidden in one broad exception.