Output Guardrails & Validation

An LLM response is not a typed return value. It is sampled text shaped by the prompt, model weights, context, tools, and decoding settings. The response can be syntactically malformed, semantically wrong, unsafe, off-policy, or influenced by untrusted text that was included in the prompt.

This is why production systems validate before trust. If downstream code expects JSON, validate the JSON before parsing it into a database write. If the answer affects a user decision, validate claims and policy. If the response contains tool arguments, validate authorization and allowed values before execution. Guardrails convert a probabilistic component into a controlled workflow with explicit pass, repair, block, or handoff states.

Input guardrails inspect the user request and any retrieved or uploaded content before the model call. They detect abuse, toxicity, self-harm risk, PII exposure, prompt-injection attempts, policy violations, and requests the product should not answer. They can block, redact, rewrite, or route the request before expensive inference.

Output guardrails inspect the model response after generation. They catch malformed structure, unsafe completions, unsupported claims, forbidden business promises, leaking private data, or action arguments that should not be executed. Mature systems use both because an allowed input can still produce a bad output, and a bad input can sometimes be made safe through refusal, redaction, or constrained routing.

Schema validation is the first layer when software must consume the response. JSON Schema, Pydantic, and Zod can enforce required fields, types, enums, length limits, nullable values, arrays, nested objects, and versioned contracts. This prevents fragile parsing and stops malformed text from reaching business logic.

Schema validation is necessary but not sufficient. A response can be valid JSON and still contain a false claim, a forbidden recommendation, or an unsafe instruction. Treat schemas as the gate that says the output has the expected shape, then add content-safety, business-rule, and grounding checks for the meaning of the data.

Moderation and safety classifiers categorize text against policy areas such as harassment, hate, sexual content, violence, self-harm, dangerous instructions, PII, and other product-specific categories. Running them on input protects the model and the system from disallowed requests. Running them on output prevents the model from returning unsafe or policy-violating content even when the request looked acceptable.

Classifiers can be deterministic rules, provider moderation endpoints, custom ML models, or LLM-based judges. They should return a category, severity, confidence, and decision. The application should define what happens for each decision: allow, redact, refuse, escalate, or ask for clarification.

Business-rule validators apply constraints that the model should not be trusted to remember. They check allowed values, numeric ranges, date windows, customer eligibility, account references, inventory ids, geographic availability, authorization, and forbidden claims such as guaranteed refunds, medical diagnosis, legal advice, or unapproved pricing promises.

These validators are usually deterministic because product policy should be explicit. They are also the right place for referential checks against databases or services. A model may output customer_id as text, but the application must verify that the customer exists, belongs to the authenticated user, and is eligible for the requested action.

In RAG, the answer should be faithful to the retrieved context. A grounding validator checks whether required claims cite provided passages, whether every citation id exists, and whether the answer says I do not know when the context is insufficient. Stronger systems compare claims against source spans or use a judge model to score faithfulness.

Prompt-injection defense is part of the same guardrail design. Retrieved documents, web pages, emails, and user uploads are untrusted data even when they are placed in the context window. Input filters, role separation, tool permissions, retrieval sanitization, output validation, and action allowlists work together to prevent injected instructions from becoming model behavior.

Schema validation gives the application a crisp contract: either the object can be parsed and has the required fields, or it cannot proceed. This is especially important for extraction, routing, tool calls, and agent plans. JSON Schema is portable across languages, while Pydantic and Zod make typed validation convenient in Python and TypeScript services.

The tradeoff is that schema validity can create false confidence. A perfectly shaped object can still be wrong. For high-risk flows, schema validation should be followed by semantic checks such as allowed claims, source citations, and policy categories.

A safety classifier usually returns categories and confidence, but the product must decide what to do with them. A mild toxicity signal in a user quote may be allowed in a summarization workflow, while the same signal in a generated assistant reply may need rewriting or refusal. Self-harm content may require a supportive crisis-safe response instead of a generic block.

Run moderation on both sides because input and output have different risk profiles. Input moderation protects the model call and controls abuse. Output moderation protects the user and prevents the application from amplifying unsafe content.

The model should not be the source of truth for product rules. If the refund window is 30 days, the validator should check the order date. If an action requires an active subscription, the validator should call the entitlement service. If a response must not promise a cure, refund, lawsuit outcome, or investment return, the validator should block those claims.

This layer is often where real incidents are prevented. The model may sound confident and empathetic, but the system must enforce allowed actions, ranges, references, and commitments.

The simplest RAG grounding check requires at least one citation and verifies that every citation id appears in the retrieved context. A stronger check maps each claim to a source span. A still stronger check uses an LLM-as-judge or entailment model to decide whether the cited text actually supports the claim.

Choose strictness based on risk. Internal search can tolerate lighter checks with visible citations. Medical, legal, financial, or customer-facing policy answers need stronger faithfulness validation, clearer uncertainty behavior, and human review for low-confidence cases.

Deterministic validators are fast, cheap, auditable, and reliable for explicit constraints such as JSON shape, enum values, length limits, known ids, and numeric ranges. They are weak for fuzzy judgments such as whether a paragraph is faithful to a source or whether an answer is helpful enough.

LLM-as-judge validators handle semantic judgments and can explain failures, but they add cost, latency, model variance, and another prompt surface that needs evaluation. Use deterministic checks first, then reserve judge models for cases where rules cannot capture the quality dimension.

Prompt injection can enter through user text, retrieved documents, emails, web pages, or tool outputs. The model may be asked to ignore system rules, reveal hidden prompts, call tools, exfiltrate data, or change the output format. Prompt wording helps, but it is not enough.

Defensive systems combine input filtering, role hierarchy, context labeling, retrieval sanitization, tool allowlists, authorization checks, output validation, and audit logging. The key principle is to treat every external text source as data, not as instructions with authority.

Log validation outcomes, block reasons, validator versions, prompt version, model version, retry count, fallback type, latency, and policy category. Store enough metadata to reproduce a failure without retaining unnecessary sensitive text. Dashboards should show schema failures, moderation blocks, repair success rate, and false-positive reports.

Repair prompts are useful for malformed structure or missing fields, but retries must be bounded. Repeatedly asking the model to fix unsafe content can waste tokens and still fail. After the retry budget, return a safe canned response, ask for clarification, or route to human review. Fallback behavior should be product-specific and tested.

Every validator adds latency and sometimes model or classifier cost. Put cheap deterministic checks early, batch moderation calls where possible, cache static policy lookups, and only call expensive LLM-as-judge checks for cases that need semantic review. Define service-level objectives for both allowed and blocked paths.

Guardrail rules are product logic and should be versioned, reviewed, tested, and rolled out gradually. Maintain evaluation sets with normal, edge-case, adversarial, and false-positive examples. A guardrail update that blocks too much can damage user experience, while one that blocks too little creates safety and compliance risk.

This example treats the model response as raw text. It parses JSON, validates schema-like constraints and business rules, sends one repair prompt for recoverable failures, and returns a safe fallback when the retry budget is exhausted.

This small checker enforces a basic RAG rule: every cited source must come from the provided context, and every structured claim must point to an allowed source id. Production systems can add source-span matching or an LLM-as-judge faithfulness check after this deterministic gate.