AI guardrails: types, where they run, and how to test them

Explainer · Gateways

AI guardrails are checks that run before or after model and tool calls and block, modify, or flag content and actions that break a policy. A guardrail can be a regex that redacts card numbers, a classifier that flags prompt injection, or a rule that stops an unapproved tool call. This explainer covers guardrail types, where they run (in code, a cloud safety service, a security API, or an AI gateway), their trade-offs, and how to test them. It belongs to our AI gateways and LLM routing hub.

What are AI guardrails?#

AI guardrails are checks placed around a language model that validate, change, or block its inputs, outputs, and actions so an application stays within its safety, security, and policy limits.

"LLM guardrails" means the same thing. Guardrails exist because a system prompt asks a model to behave but enforces nothing; a guardrail moves the rule into code, or into a separate model whose decision the application acts on. Every guardrail has three parts:

  • A checkpoint: where it runs, such as on the user's message, a retrieved document, a tool call's arguments, or the response.
  • A detector: how it decides, from a regular expression or a JSON Schema validator to a classifier model or an LLM judge.
  • An action: block, redact or rewrite, ask the model to retry, route to a person, or only log. The same PII detector that breaks a support chatbot when it blocks can keep it working when it redacts.

Types of AI guardrails#

AI guardrails fall into three families by checkpoint. Input guardrails screen what reaches the model, output guardrails screen what the model returns, and action guardrails screen what an agent tries to do with tools. Each family mixes deterministic rules, classifier models, LLM judges, and schema validation, chosen by how precise, fast, and flexible the check must be.

Type Example checks Typical implementation Typical action
Input Prompt injection, jailbreaks Classifier (Prompt Guard 2, Prompt Shields) Block or flag
Input Personal data, secrets Regex, entity recognition, secret scanners Redact or block
Input Topic restrictions Topic classifier, dialog rules, LLM judge Fixed reply
Output Harmful content Safety classifier (Llama Guard 4) Block or rewrite
Output PII or secret leakage Regex, entity recognition, PII filters Redact
Output Grounding, hallucination Grounding classifier or LLM judge Block, flag, regenerate
Output Schema validity JSON Schema or Pydantic validation Reask or fail
Action Tool allowlists Rules on tool name and caller Deny
Action Argument validation Schema checks, business rules Deny or fix
Action High-impact actions Human-in-the-loop approval Pause for a person

Input guardrails#

Input guardrails inspect text before the model sees it, including retrieved documents and tool results, which can carry instructions too. Azure's Prompt Shields analyzes a userPrompt for direct attacks and up to five documents for hidden instructions, and can scan agent tool responses in Microsoft Foundry, in preview. Meta's Llama Prompt Guard 2 reads 512 tokens per call, so its model card advises splitting longer inputs and scanning the segments in parallel. Topic restrictions, which Amazon Bedrock Guardrails calls denied topics, keep a narrow assistant narrow.

Output guardrails#

Output guardrails inspect the response before a user or downstream system receives it. Grounding checks flag claims the retrieved passages do not support, the property the faithfulness metric scores offline. Bedrock's contextual grounding check scores grounding and relevance against thresholds you set, and Azure AI Content Safety's groundedness detection is a preview that Microsoft lists for retirement on April 30, 2027. Format checks validate output against a schema and reask on a mismatch; provider structured outputs prevent most format failures, but not wrong values inside a valid structure.

Action and tool guardrails#

Action guardrails inspect the tool an agent called, its arguments, and sometimes the result. Allowlists limit each agent to the tools its task needs, argument validation enforces rules such as a refund limit, and approval pauses payments or deletions until a person agrees; our tool calling explainer covers the surrounding loop. The OWASP entry on excessive agency (LLM06:2025) recommends enforcing authorization in downstream systems instead of relying on the model to decide whether an action is allowed.

Where AI guardrails run#

AI guardrails run in four places: inside the application as libraries and self-hosted classifier models, in a cloud provider's safety service, through a dedicated security API, or centrally in an AI gateway that every application calls. The locations are not exclusive, and most production systems layer two or more so that one missed check is not the only defense.

Four layers where guardrails run: application code, an AI gateway, external guardrail services, and the model provider's built-in filters, with the first two under your control. Request path from the application to the modelApplicationCode you write or importSchema checksTool allowlistsNeMo GuardrailsGuardrails AIAI gatewayShared rules for every appInput rulesOutput rulesMCP tool rulesRedactionGuardrail servicesCalled by app or gatewayBedrock GuardrailsContent SafetyModel ArmorLakera GuardModel providerFilters built into the APIContent filtersSafety trainingYour infrastructureVendor APIs
Figure 1: Guardrails can run at four layers. Checks in application code and at the gateway run on your infrastructure; guardrail services and provider filters run on vendor infrastructure.
Location Examples (November 2026) Strengths Limits
Application code NeMo Guardrails, Guardrails AI, Llama Guard 4 Full context: user, documents, tool permissions Repeated per service, no shared logs
Cloud service Bedrock Guardrails, Azure AI Content Safety, Model Armor Managed detectors, nothing to host Per-check charges, language limits
Security API Lakera Guard (Check Point AI Guardrails) Attack-focused detection Another vendor in the data path
AI gateway Bifrost, LiteLLM, Kong AI Gateway One policy for every app, redacted logs No view of application state

In the application: libraries and open guard models#

In-application guardrails run in your process or a sidecar you host, so they suit checks that depend on what the application knows: the user, the retrieved documents, and each tool's permissions.

NVIDIA's NeMo Guardrails is an Apache 2.0 Python library with five rail types: input, dialog, retrieval, execution (for tools), and output. Rails live in config.yml and in Colang, a dialog flow language. It runs in-process through LLMRails or as a server with a /v1/chat/completions endpoint, and its built-in rails include LLM self-checks, jailbreak detection, and NVIDIA's content safety models, with integrations for Presidio and Llama Guard. Version 0.24.1 was published on September 16, 2026. An abbreviated config.yml, adapted from the README and the streaming docs:

models:
  - type: main
    engine: openai
    model: gpt-4o-mini
 
rails:
  input:
    flows:
      - check jailbreak
  output:
    flows:
      - self check facts
    streaming:
      enabled: True
      chunk_size: 200
      context_size: 50
      stream_first: False

Open guard models run beside the application. Meta's Llama Guard 4 is a 12B multimodal classifier that labels prompts and responses against 14 hazard categories based on the MLCommons taxonomy, and Prompt Guard 2 labels input as benign or malicious. IBM's Granite Guardian models (Apache 2.0) check harm, jailbreaks, and groundedness, and Qwen3Guard has a Stream variant that checks output token by token.

Guardrails AI#

Guardrails AI is an Apache 2.0 Python framework that runs input and output guards around LLM calls and generates structured data from Pydantic models. A guard combines validators from Guardrails Hub, such as ToxicLanguage or RegexMatch, and each validator takes an on-fail action: reask, fix, filter, refrain, noop, exception, fix_reask, or custom. guardrails start runs it as a REST server that clients can call through the OpenAI SDK. From the README:

from guardrails import Guard, OnFailAction
from guardrails_ai.competitor_check import CompetitorCheck
from guardrails_ai.toxic_language import ToxicLanguage
 
guard = Guard().use(
    CompetitorCheck(["Apple", "Microsoft", "Google"], on_fail=OnFailAction.EXCEPTION),
    ToxicLanguage(threshold=0.5, validation_method="sentence", on_fail=OnFailAction.EXCEPTION)
)

On July 6, 2026, the project announced that validators were moving to standard PyPI packages and that hosted remote inference for validators would end, with a planned cutoff of August 25. On September 9, Harvey announced that it had acquired Guardrails AI, without describing plans for the open-source framework. Version 0.11.0 was published on August 14, 2026.

Cloud provider safety services#

Cloud safety services run managed detectors, so you configure policies instead of hosting models, and each can check text bound for any model.

  • Amazon Bedrock Guardrails offers content filters (including a prompt attack filter), denied topics, word filters, PII filters, contextual grounding checks, and Automated Reasoning checks. Its ApplyGuardrail API checks any text, with source set to INPUT or OUTPUT, without invoking a model.
  • Azure AI Content Safety scores text and images for hate, sexual content, violence, and self-harm by severity and detects prompt and document attacks with Prompt Shields, inside Microsoft Foundry or through standalone APIs.
  • Google Cloud Model Armor screens for injection and jailbreaks, sensitive data, malicious URLs, and harmful content, for AI deployed on Google Cloud or elsewhere. Its overview notes that it inspects each prompt and response as a single-turn request, without conversation history.

Dedicated security APIs#

Dedicated security APIs come from vendors that specialize in AI attacks. Lakera Guard is one example. Check Point acquired Lakera and now documents the product as Check Point AI Guardrails, served from api.lakera.ai. One POST /v2/guard call takes the conversation in OpenAI chat format and screens the latest user message as input, the latest assistant message as output, and tool messages and tool calls as untrusted content and agent actions. Our guardrails and AI security directory lists more vendors.

At an AI gateway#

A gateway runs guardrails once for every application and agent behind it, so a new service inherits the organization's PII, secret, and content policies without code changes, including on tool calls it proxies to MCP servers.

Bifrost, the open-source AI gateway from Maxim AI, includes guardrails in Bifrost Enterprise, built from two objects:

  • Rules decide when a check runs: a CEL expression over metadata such as model, virtual key, team, or user; a phase (input, output, or both); a sampling rate; and a target, LLM calls or MCP tool executions. The expression mcp_client == "github" && mcp_tool == "create_issue" guards one tool.
  • Profiles decide how content is checked. Bifrost runs Secrets Detection (Gitleaks rules), Custom Regex with a PII template, and Prompt Guardrails, an LLM judge. External profiles call AWS Bedrock Guardrails, Azure Content Safety, Google Model Armor, Microsoft Presidio, Azure AI Language PII, CrowdStrike AIDR, Gray Swan Cygnal, Patronus AI, Check Point's AI Agent Security (the Lakera Guard API), Repello Argus, Singulr AI, Noma Security, and A10 Guardrails.

A rule's profiles run in order, and the documentation recommends a cheap local profile, such as Custom Regex, before external ones. A block stops the remaining profiles and returns HTTP 446 with a guardrail_violation error; a provider error or timeout fails open.

Sequence: an application sends a request to an AI gateway, which checks the prompt with a guardrail API, calls the model provider, and checks the response before returning it. Applicationchat or agentAI gatewayrules, profilesGuardrail APIclassifier, judgeModel providerhosted model1user request2input check3allow or redact4checked prompt5response6output check7allow, redact, block8response or error
Figure 2: A gateway checks the prompt before the provider receives it and the response before the application does; a block at either step ends the request.

Bifrost's redaction modes set where a rewrite applies: runtime redacts live traffic and logs, logs_only forwards the original but stores placeholders in Bifrost logs and trace exports such as OpenTelemetry, and runtime_reversible uses placeholders that a permitted user can reveal in Bifrost logs; the reveal mapping is not exported. Logs need this care because they outlive requests. AWS notes that content blocked by Bedrock Guardrails appears as plain text in model invocation logs when those are enabled. Our guide to LLM observability covers keeping prompts private in traces.

Other gateways in our AI gateways directory, including LiteLLM, Kong AI Gateway, agentgateway, and Prisma AIRS AI Gateway (formerly Portkey), also run guardrail checks.

Trade-offs of AI guardrails#

Every guardrail spends latency, money, and some legitimate traffic to reduce a risk. The costs compound because checks run on both input and output, often several per phase, and streaming forces a choice between delaying text and checking it after users see it. Weigh each check against the failure it prevents before enabling it everywhere.

Latency and cost#

An input check delays the first token by its full duration, and sequential checks add up. Classifiers are fast; Meta's model card lists 19.3 ms per 512-token call for Prompt Guard 2 22M and 92.4 ms for 86M on an A100 GPU. LLM judges cost the most because each is a full model call; Bifrost's documentation flags the added latency and token cost of its judge calls and recommends sampling rules on high-traffic endpoints. Run cheap checks first, run independent ones in parallel, and set a timeout on each external call, then decide whether a timeout fails open (traffic passes unchecked) or closed (users get errors). For payment or deletion tools, closed is usually safer.

Services charge per check. On November 16, 2026, the Amazon Bedrock pricing page listed content filters and denied topics at $0.15 per 1,000 text units each and sensitive information filters at $0.10, where a text unit holds up to 1,000 characters. Each policy is charged separately, so three policies on a prompt and its response are each charged twice.

False positives and false negatives#

Every detector has a threshold, and moving it trades a blocked legitimate user (a false positive) for a missed attack (a false negative). Google recommends Model Armor's High confidence level for production apps that prioritize uninterrupted interactions, and suggests its lowest level, with caution, for high-stakes categories such as prompt injection. Placement also shifts errors. Meta's Llama Guard 4 card reports that in internal tests, input filtering raised the refusal rate more than output filtering did. The Prompt Guard 2 card reports 97.5% recall at a 1% false-positive rate for the 86M model on Meta's English evaluation set, and warns that adversaries may build attacks to bypass detection, so measure on your own traffic.

Streaming responses#

Streaming changes only output guardrails. A guardrail can buffer and check each chunk before release, which delays text, or stream immediately and check in the background, which lets some violating text reach the user before the stream stops.

Product Check before release Stream first, check after Note
Amazon Bedrock Guardrails Synchronous mode (default) Asynchronous streamProcessingMode Async mode cannot mask PII
Azure OpenAI in Foundry Default, in buffered chunks Asynchronous Filter Async signal within about 1,000 characters
NeMo Guardrails stream_first: False stream_first: True (default) Checks chunk_size tokens at a time
Bifrost Holds the stream if a rule can block Detect-only and monitor rules Redaction releases text by segment

Microsoft's content streaming documentation adds that even in default mode, the last segment can be returned before the policy is applied. In its test, a blocklisted term in the final 15 to 30 words reached the client.

Multilingual coverage#

Accuracy usually drops outside the languages a detector was tested on. As of November 2026, Bedrock Guardrails' Classic tier supports English, French, and Spanish; its Standard tier supports more. Azure AI Content Safety's models were trained and tested on eight languages, but groundedness and protected material detection are English only. Model Armor's safety and injection filters were tested on nine languages and Prompt Guard 2 on eight, with a multilingual AUC of 0.942 for the 22M model against 0.995 for 86M. Model Armor also does not decode Base64, hexadecimal, or URL encoding, so encoded payloads need their own check.

How to test AI guardrails#

Test guardrails the way you test any classifier: measure how often they catch what they should and how often they block what they should allow. That takes a red-team set of attacks, a labeled sample of real traffic, a shadow period before enforcement, and a regression suite that reruns whenever a model, threshold, rule, or guardrail version changes.

Our guide to LLM evaluation covers the datasets, graders, and CI gates these tests share with quality evals.

Build the test sets#

  1. Attack set: prompt injections, jailbreaks, exfiltration attempts, and harmful requests for your application, including injections planted in documents and tool results. Open-source scanners such as NVIDIA's garak, Microsoft's PyRIT, and promptfoo generate attacks automatically.
  2. Benign set: labeled production traffic with personal data masked, plus hard negatives such as security questions from real users, medical language, and code with secret-like strings. A golden dataset of these cases catches over-blocking.
  3. Expected actions: label each case allow, redact, block, or approve, so the test checks the action as well as the detection.

Measure the right rates#

Metric Definition What it tells you
Attack recall Share of attack cases blocked or flagged How much the check catches
False-positive rate Share of benign cases blocked or flagged How much legitimate traffic it costs
Live block rate Share of production requests blocked, by rule Drift and incidents after release
Added latency p50 and p95 time spent in checks Fit with the latency budget
Redaction accuracy Planted PII removed, other text intact Over- or under-matching

Report each rate by category and language, because a low overall false-positive rate can hide a high one for a single language or customer segment.

Shadow mode and regression suites#

Run a new guardrail in a mode that records decisions without acting, then compare findings with labeled traffic: Bifrost's monitor mode logs what output guardrails would have done, Model Armor offers inspect-only enforcement, and Check Point recommends starting in detect mode and raising sensitivity gradually. After launch, rerun both sets on every model, threshold, or policy change, fail the build when recall or the false-positive rate moves past a limit, and turn each incident into a test case.

Map guardrails to the OWASP Top 10 for LLM Applications#

The OWASP Top 10 for LLM Applications 2025 is the current edition, and it works as a coverage checklist for runtime guardrails and their tests.

OWASP 2025 risk Guardrails that address it What to test
LLM01 Prompt Injection Injection classifiers on input, documents, tool results Direct and indirect attacks
LLM02 Sensitive Information Disclosure PII and secret redaction in traffic and logs Planted PII and keys
LLM05 Improper Output Handling Schema validation, output encoding, URL checks Malformed JSON, malicious links
LLM06 Excessive Agency Tool allowlists, argument checks, approval Unauthorized tool calls
LLM07 System Prompt Leakage Output checks; no secrets in prompts Extraction attempts
LLM09 Misinformation Grounding checks against sources Unsupported claims
LLM10 Unbounded Consumption Gateway rate limits and token budgets Oversized inputs, request floods

LLM03, LLM04, and LLM08 (supply chain, data and model poisoning, and vector and embedding weaknesses) depend mostly on controls outside the request path. OWASP's prompt injection entry states that it is unclear whether fool-proof prevention exists, so it pairs filtering with least privilege, human approval, and adversarial testing. For agents, OWASP also published a Top 10 for Agentic Applications for 2026, dated December 9, 2025.

Where to go next: our explainer on what an AI gateway is covers the layer where shared guardrails usually live.

Frequently asked questions

What is NeMo Guardrails?

NeMo Guardrails is NVIDIA's Apache 2.0 Python library for adding programmable guardrails to LLM applications. It defines input, dialog, retrieval, execution, and output rails in YAML and the Colang language, calls safety models and detection APIs, and runs in-process or as a server with a chat completions endpoint. Version 0.24.1 was published on September 16, 2026.

What is Guardrails AI?

Guardrails AI is an Apache 2.0 Python framework that wraps LLM calls in input and output guards built from validators, and that generates structured data from Pydantic models. In July 2026, the project announced that its validators were moving to standard PyPI packages. Harvey announced its acquisition of Guardrails AI on September 9, 2026, without describing plans for the open-source framework.

What is the difference between NeMo Guardrails and Guardrails AI?

NeMo Guardrails controls a whole conversation, with rails at five checkpoints, dialog flows written in Colang, and calls to safety models such as Llama Guard. Guardrails AI validates individual inputs and outputs with validators and on-fail actions such as reask, fix, or exception. Both are Apache 2.0 Python libraries, and NeMo Guardrails lists Guardrails AI validators among its integrations.

Can guardrails stop prompt injection?

No guardrail stops every prompt injection. Classifiers such as Llama Prompt Guard 2, Azure Prompt Shields, and Google Model Armor catch many known attack patterns, but OWASP says it is unclear whether fool-proof prevention exists. Pair detection with least-privilege tools, output validation, and human approval for high-impact actions, so a missed injection does limited damage.

Do guardrails work with streaming responses?

Yes, with a trade-off. Input checks finish before streaming starts. For output, a guardrail either buffers and checks chunks before release, which delays text, or streams first and checks in the background, which can let some violating text through before the stream stops. Amazon Bedrock Guardrails, Azure OpenAI, and NeMo Guardrails document both modes.

Should guardrails run in the application or in an AI gateway?

Run checks that need application context in the application, such as validating a tool's arguments against the current user's permissions. Run organization-wide policies, such as PII redaction, secret detection, and content safety, in an AI gateway, so every app and agent gets the same rules and logs without code changes. Most production systems use both.

Terms in this guide

Cite this guide

The AI Engineering Editors. "AI guardrails: types, where they run, and how to test them." The AI Engineering, updated Nov 18, 2026. https://theaiengineering.com/articles/ai-guardrails

Title
AI guardrails: types, where they run, and how to test them
Author
The AI Engineering Editors
Publisher
The AI Engineering
Published
Updated
View as Markdown

The AI Engineering Editors ·