Why teams filter prompt guards at the APISIX gateway
A gateway checkpoint is often the first place a prompt can be screened before it leaves your network. When an AI feature starts sending user prompts to hosted models, security and compliance teams usually ask who inspects that text, where the inspection happens, and what evidence records it. APISIX ships several AI plugins that can sit on the same route. ai-prompt-guard checks request content against rules you configure. ai-aws-content-moderation and ai-aliyun-content-moderation send text to a cloud moderation service and act on its verdict. ai-lakera-guard calls the Lakera Guard service for screening. A team typically needs one of these when prompts may carry personal or confidential data, when responses must avoid certain policy categories, or when a control owner expects proof that every request passed a documented check. The full plugin set is listed in the APISIX plugin documentation.
The gateway is one layer among several, and it works best as the place where a single rule applies to every consumer and every model behind a route. APISIX runs these checks alongside ai-proxy and ai-proxy-multi, so the guard decides whether a request reaches a provider at all. A gateway check sees the HTTP request and the text in it, but it cannot judge what a model does with context it already holds, or whether a model would produce an unsafe answer from an innocent prompt. Those risks need model-level controls, system prompt design, output checks and monitoring. Before you enable a moderation plugin, confirm with the vendor which data leaves your network when the plugin calls its service, how long that service retains it, and whether your data terms allow that transfer.
Settle ownership before the build starts. Name who maintains prompt patterns, who sets moderation categories and thresholds, and who reviews blocked requests. Confirm that each plugin you choose is supported in the APISIX version you run, and read the project changelog before you pin a version, because the behavior of these plugins has changed between recent releases. Plan the evidence too: logs that show which check fired and why, plus a fixed set of test prompts that you rerun after every configuration change. DeployOpen can plan and configure this gateway layer and its test set under a fixed-scope engagement, with scope, deliverables and timelines agreed in writing before work starts.
Choosing between the guard and moderation plugins
ai-prompt-guard checks request content against allow and deny patterns that you define in the gateway. The check runs inside APISIX, so prompt text does not leave your network for this control and no provider terms apply. Its decisions are based on text your team can read, test and explain to an auditor, but it only catches what your patterns anticipate.
- What it inspects: the prompt messages in the request body, matched against deny patterns and, where you configure them, allow patterns.
- Upstream dependency: none beyond APISIX itself, so there are no external accounts, regions or retention questions.
- Typical gaps: paraphrased injections, other languages and encoded text can pass fixed patterns unless you add cases for them.
- Ownership: the patterns are configuration that your team must review, version and retest after every change.
- Scope: the plugin inspects the request side. Output screening needs a separate control.
Confirm the configuration keys and matching behavior for your APISIX version in the plugin documentation, then test the patterns against a sample set of prompts you expect to block and prompts you expect to pass.
ai-aws-content-moderation sends prompt text to an AWS moderation service and acts on the result. Since APISIX 3.18.0 it moderates the decoded prompt in the access phase and requires ai-proxy or ai-proxy-multi on the same route, so it only guards traffic that APISIX forwards to a model. Every checked request leaves your network for AWS, which makes the AWS account and its terms part of your design.
- What it inspects: the decoded prompt text, checked before the request reaches a model.
- Upstream dependency: an AWS account, credentials the gateway must hold, and a network call from the gateway to AWS on each checked request.
- Route dependency: the plugin depends on an AI proxy plugin on the route, so routes without one are not guarded by it.
- Latency and failure handling: each check adds a remote call, so timeouts and AWS service disruptions affect request handling. Set the failure behavior deliberately.
- Data path: prompts are processed in the AWS region you configure.
Confirm which AWS service and region the plugin calls, where prompt text is processed, whether AWS retains it, and how your AWS agreements and data processing terms cover that processing.
ai-aliyun-content-moderation sends text to the Alibaba Cloud content moderation service. From APISIX 3.18.0 it checks the latest user turn by default, so earlier turns in a conversation and model output need explicit handling if they matter to your policy. The service and its endpoints sit outside your infrastructure, which makes the Alibaba Cloud account, region and residency terms the first questions for regulated data.
- What it inspects: by default the latest user turn in the conversation, as documented for version 3.18.0 and later.
- Upstream dependency: an Alibaba Cloud account, credentials held by the gateway and outbound calls to the service on each checked request.
- Region and residency: the endpoint region determines where text is processed and stored, which can conflict with data residency rules in your sector or country.
- Scope gaps: checking only the latest user turn leaves earlier turns and responses unscreened unless you configure more coverage.
- Terms: the provider's service terms, retention and any cross-border transfer obligations apply to your prompts.
Confirm the service regions available to your account, where prompt text is processed and retained, and whether your legal and privacy teams accept those flows before you enable the plugin.
ai-lakera-guard passes prompt content to Lakera Guard, a hosted screening service, and acts on its verdict. It is the option that depends most on a vendor's own detection logic, so your team cannot inspect how a given prompt was classified in the way it can review a local rule. The plugin adds an external call on each checked request, and that call becomes part of your data flow diagram and your vendor review.
- What it inspects: the prompt content sent to the service. Confirm whether responses or only inputs are sent.
- Upstream dependency: a Lakera account, an API credential held by the gateway and outbound calls on each checked request.
- Detection scope: confirm which threat and content categories the service detects and how it reports results to APISIX.
- Hosting and terms: confirm hosting regions, retention, sub-processors and the contractual data terms that apply to your account.
- Failure behavior: confirm what APISIX does when the service is slow or unavailable, and set that behavior to match your risk appetite.
Before you rely on the plugin, ask the vendor for its current data processing terms, hosting locations and any self-hosted option that would keep prompt text inside your environment.
What changed in APISIX 3.18 for moderation scope and fail modes
Version 3.18.0 changed what three AI moderation settings cover, so review those routes before you upgrade. First, ai-aws-content-moderation now moderates the decoded prompt in the access phase, and it requires ai-proxy or ai-proxy-multi on the same route. A route that relied on the plugin running without one of those proxy plugins, or that expected it to see the raw request body, will behave differently. Second, ai-aliyun-content-moderation checks the latest user turn by default instead of the whole conversation. If your policy depends on earlier turns being screened, you must set that explicitly. The changelog entries are in the APISIX changelog, and they are the reference for exact wording.
Third, 3.18.0 added a fail_mode option for AI plugins. Some of these plugins use a skip default, which lets traffic that is not an AI request pass through the plugin untouched. That is convenient for mixed routes but unsafe where a guard is meant to be mandatory. Decide, per plugin and per route, whether a failure or a non-AI request should be blocked or allowed, and write that decision down. Confirm the default values for your installed version in the plugin documentation rather than assuming them from memory.
Before upgrading, export your route definitions from the Admin API and compare each route that uses a moderation plugin against the new access-phase, latest-turn and fail_mode behavior. Then replay representative prompts, including multi-turn conversations, oversized bodies and requests that should be rejected, on a non-production route with the same upstream and consumer setup. Check that blocked requests return the status and message your clients expect, and that allowed requests still reach the model. Record the results so your change review can show what was tested.
Configuration options and when each fits
This table compares the main ways to filter prompts and responses at the APISIX gateway, from in-gateway pattern rules to external moderation and guard services. Read each row for when it fits and what it costs in latency, data movement or false positives. Confirm exact configuration keys and current behavior in the plugin documentation before you build.
| Option | When it fits | Trade-off to weigh |
|---|---|---|
| ai-prompt-guard rule patterns and actions | Blocking or rejecting prompts that match known phrases, forbidden terms or formats you can describe in advance, with no third-party call | Runs inside the gateway, so no data leaves your network. Pattern lists need ongoing upkeep and can produce false positives or miss paraphrased attacks |
| ai-aws-content-moderation | Teams already on AWS that want a managed moderation classifier applied to the decoded prompt during the access phase | Prompt content is sent to AWS. Requires ai-proxy or ai-proxy-multi in the route since APISIX 3.18.0, and adds a network round trip to every checked request |
| ai-aliyun-content-moderation | Deployments serving users in regions where Alibaba Cloud moderation is the approved service, or where the account already exists | Prompt content is sent to Alibaba Cloud. Since 3.18.0 it checks the latest user turn by default, so earlier turns in a conversation are not covered unless you change the scope |
| ai-lakera-guard | Organizations that want a dedicated prompt-injection and content screening service evaluated against their own test set | Prompt content is sent to an external service. Confirm data handling, hosting location and contract terms with the vendor before routing regulated data through it |
| fail_mode setting | Choosing whether a guard outage blocks traffic (fail closed) or lets it pass (fail open) for each route with different risk | Fail closed protects against unscreened prompts but makes the guard a availability dependency. Fail open keeps service running while leaving prompts unchecked during an outage |
| Skip behavior for non-AI traffic | Mixed gateways where the same plugin set applies to AI and non-AI routes, so AI-only checks should not affect other APIs | Confirm the default in the 3.18.0 changelog and the plugin docs. A permissive default can let traffic through that you expected to be screened |
| Scope: latest user turn only | Low-latency checks where the newest message carries the risk and earlier turns were already screened | Injected content in earlier turns, or context carried in a system or tool message, may not be inspected. Test multi-turn conversations explicitly |
| Scope: full conversation or decoded prompt | Multi-turn chat or agent flows where an attack can be split across messages or hidden in encoded text | Higher payload size and latency per request, and more false positives from quoted or historical content. Measure on representative traffic |
| Model-level controls (provider or self-hosted guardrails) | Checks that need the model's own context, such as output behavior, tool use or refusal logic, which a gateway cannot see | Depends on the model host and adds work per model. Use alongside gateway checks rather than as a replacement for them |
How to set up prompt guards on an APISIX AI route
This sequence takes an APISIX deployment from version and mode checks to a tested AI route that screens prompts before they reach a model provider. Each step ends with a check you can record as evidence that the guard is working.
Confirm APISIX version and deployment mode
The platform team confirms which APISIX release is running and whether it uses traditional, decoupled or standalone mode. Guard behavior changed in 3.18.0, including the access-phase moderation scope for ai-aws-content-moderation and the fail_mode option, so the version determines the configuration you write. Check the running version against the APISIX downloads page and the changelog, and record the deployment mode for the runbook. The step is done when the version, mode and listener ports (9080 for client traffic in traditional mode) are written down and match the environment you will change.
Create the AI route with ai-proxy or ai-proxy-multi
The team creates a route whose upstream is an LLM provider through ai-proxy for a single provider or ai-proxy-multi when traffic spans several. Guard plugins only apply to AI routes that use one of these two plugins, so this route is the foundation for everything that follows. Use a test consumer and a non-production upstream first. The step is done when a request sent to the route with a harmless prompt returns a normal model response, and the access log shows the request reached the route.
Add the guard plugin in the access phase
The engineer attaches the chosen guard plugin, such as ai-prompt-guard for pattern rules or a moderation plugin for a managed service, to the same route. For ai-aws-content-moderation on 3.18.0 and later, the plugin inspects the decoded prompt in the access phase, so the plugin must sit where the request is screened before the upstream call. Confirm the plugin's configuration keys against the plugin documentation for your release. The step is done when the route configuration saved in the Admin API or the YAML file lists the guard plugin and the route still responds to a benign prompt.
Store moderation provider credentials as secrets
The security team places provider keys, such as AWS or Aliyun credentials for moderation, in the secrets store used by the APISIX deployment rather than in route JSON or YAML committed to source control. The engineer references the secret from the plugin configuration and confirms which secret backend and reference syntax your release supports. Restrict who can read the secret and who can change the route. The step is done when the route configuration contains no plaintext credential, the moderation call succeeds with the secret reference, and the change is recorded in your configuration history.
Set fail_mode for AI plugin failures
The team decides how the route behaves when the guard cannot return a verdict, such as a timeout or an unreachable moderation endpoint. The fail_mode option added in 3.18.0 governs AI plugin failures, and its skip default can let non-AI traffic through, so confirm the default for each plugin you use and set the value your risk owner approves. Document the choice with the reason, because an open failure mode and a closed one have different consequences for users. The step is done when the setting is saved in the route configuration and the owner has signed off on it in writing.
Test allowed and blocked prompts
The tester sends a fixed set of prompts through the route: some that should pass, some that the guard rules or moderation categories should block, and some edge cases such as prompts placed in earlier conversation turns. The expected result for each prompt is written before the test runs, so the outcome is compared against a target rather than judged afterward. Blocked prompts should return the rejection status configured for the plugin without reaching the provider. The step is done when every test case matches its expected result and the results are saved with the test date and APISIX version.
Check gateway logs and metrics for each decision
The operations engineer reviews the access logs, error logs and any metrics exposed for the route to confirm each blocked and allowed request was recorded with the expected decision. The check verifies that blocked prompts never produced an upstream call and that the fail_mode behavior appears when a moderation dependency is stopped on purpose in a test environment. Confirm the log fields and retention period with your logging platform owner, because retention is set by your environment rather than by APISIX. The step is done when a sample of decisions can be traced from the request to the log entry and the test evidence is filed with the change record.
Security and operational risks of gateway-level filtering
Gateway-level filtering puts a new dependency and a new data flow in front of every LLM request. The cards below cover the risks that come with that position: credentials, third-party data handling, failure behavior, logging and rule maintenance. Each one should have an owner and a documented decision before the filter goes live.
Moderation provider credentials
Moderation calls to AWS, Aliyun or Lakera need API keys or cloud credentials stored where APISIX can read them at request time. Keep these out of route definitions and plain YAML in version control, and use a secrets store with scoped, least-privilege identities. Confirm how each provider supports key rotation and whether rotation can happen without a gateway restart.
Prompt data sent to third parties
A moderation check sends the prompt, and sometimes the response, to an external endpoint before the model sees it. That makes the provider a processor of user content. Before enabling a provider, ask for its data retention period, whether data is used for training, and which regions can process the request. Record the answers in your data inventory.
Fail-open or fail-closed decisions
When a moderation provider times out or returns an error, the gateway must either pass the request or block it. Choose the behavior per route, not once for the whole gateway. Fail-closed protects sensitive workloads but makes the AI feature unavailable during an outage. Fail-open keeps service running but leaves unscreened traffic reaching the model. Test both paths before production.
Logging blocked content safely
Gateway logs are useful for investigations, but a log entry for a blocked prompt can contain the very data the filter was meant to protect. Log the rule that fired, the route, the consumer and a request identifier rather than the full prompt. Confirm where logs are stored, who can read them, and how long they are kept, since these settings usually sit outside APISIX.
Tuning rules over time
Blocklists and moderation thresholds drift. Legitimate users hit false positives, and attackers rephrase around known patterns. Assign an owner for each rule set, review blocked and passed samples on a fixed schedule, and roll out changes through version control with a staged route before full traffic. Treat rule changes as production changes with a rollback plan.
Why LLM Guard is no longer a new rollout choice
LLM Guard should not be the basis of a new prompt guard rollout. The protectai/llm-guard repository was archived by its owner on 9 July 2026, and its README states that the project and its associated Hugging Face models are no longer under active development or maintained. The last release on PyPI is 0.3.16, uploaded on 19 May 2025. DeployOpen does not recommend new LLM Guard rollouts. If your organization already runs LLM Guard as a scanner behind ai-prompt-guard style checks or as a separate API server, we will support that existing deployment, including pinning versions, reviewing the Python runtime (3.10 or newer but below 3.13), and documenting its scanner settings.
Before you invest further, decide how the scanner will be replaced. Use the LLM Guard documentation to list which of its 15 input scanners and 20 output scanners your current gateway policy depends on, such as prompt injection, secrets, toxicity or sensitive data detection. Then evaluate NeMo Guardrails, an open-source toolkit for programmable guardrails, against that list. Confirm with the NeMo Guardrails project which scanners it covers natively, which need custom actions, and how it integrates with an APISIX request path. Treat that evaluation as a proof of concept with measured false positive and false negative rates on your own prompts, since no comparison figures are supplied here.
A migration from LLM Guard to a maintained guardrail needs its own scope, owner and acceptance criteria. That scope should cover the guard API contract APISIX calls, the fail behavior during cutover, the test corpus, logging of blocked requests, and a rollback path to the existing scanner until the new control is verified. DeployOpen plans and delivers that migration as a separate fixed-scope engagement, agreed in writing before work starts.
Where gateway checks stop and model-level controls begin
A gateway guard sees the text that crosses the APISIX data path and nothing the model does with it. The cards below mark the places where that view ends, so you can decide which controls sit in front of the model, which sit inside the application, and which need to be confirmed with the model provider.
The gateway cannot see model internals
An APISIX guard inspects the request and response bodies that pass through the route. It does not see how the model weighs those tokens, which hidden instructions it follows, or what it retains internally. Treat a passed check as a text filter result, not as proof that the model behaves safely. Confirm with each provider what it logs, retains and trains on, since that data never passes through your gateway.
Injected instructions in documents
Prompt injection often arrives inside retrieved documents, web pages, emails or uploaded files, not in the user's own message. A guard that scans only the latest user turn will miss instructions carried in that content. Test your guard with injected text placed in each content source your application reads, and confirm which fields the plugin scans in your version.
Partial view of multi-turn context
Some guards check only the latest user turn by default, so an attack split across several messages can pass one turn at a time. The APISIX 3.18.0 change to ai-aliyun-content-moderation made the latest user turn the default scope, which is a deliberate limit to review. Decide whether your policy needs full conversation scanning, and test it with a conversation that builds up across turns.
Tool calls and agent actions
When a model calls tools, its arguments and the results it receives can trigger database writes, payments or messages to other systems. A prompt guard on the HTTP body does not check whether a proposed action is allowed. Plan separate controls in the tool layer, including allowlists, parameter validation and approval steps for sensitive actions, and confirm how any MCP path in your gateway handles tool traffic.
Output checks do not replace alignment
Output moderation can block a harmful answer after the model produces it, but it cannot make the model refuse in the first place or shape how it reasons. Keep system prompt design, model selection and safety tuning as separate work items with their own owners. Ask your model provider which alignment and safety evaluations they publish for the model version you run.
Application authorization stays in the application
A guard cannot decide whether the requesting user may see a customer record that the model retrieves. Access decisions belong in the application or data layer, enforced on the user's identity and not on the model's wording. Confirm that each retrieval and tool call checks the end user's permissions, and that the gateway consumer identity matches the user you are authorizing.
Questions teams ask before filtering prompts at the gateway
How much effort does a prompt guard rollout take and what drives it?
The effort depends mostly on scope, and no fixed figure applies to every environment, so ask for the estimate in writing. The main drivers are the number of guards, the number of routes and consumers they cover, how many moderation providers are integrated, the size of the test set, and how much tuning the block policy needs after the first runs. Costs fall into two groups. One-time costs cover design, configuration, integration, testing and documentation. Recurring costs cover provider fees, log storage, rule updates and periodic review of blocked traffic. Ask the partner to separate these in the quote and to state which provider charges, if any, sit outside the engagement.
Can our own engineers configure prompt guards in APISIX themselves?
Yes, a team that already runs APISIX can configure the basic guards itself. The ai-prompt-guard plugin is applied per route, and the other AI moderation plugins follow the same pattern, so the configuration work is largely familiar Admin API or declarative configuration. The harder parts are outside the YAML: choosing which content to block, handling provider credentials and secrets, designing responses for blocked requests, writing a test corpus that includes benign prompts so false positives are visible, and deciding how the guard behaves when a provider is slow or unavailable. Teams without prior experience with AI traffic often underestimate the testing and policy work.
What does a guard rollout need from our security and legal teams?
Security needs to define the blocking policy, the response to a blocked request, the fail mode when a moderation service is unreachable, and what gets logged. Those logs can contain user prompts, so retention and access rules need an owner. Legal and privacy teams need to confirm whether prompts containing personal data may leave your network for a third-party moderation service, what the provider's data processing terms say about storage and region, and whether users must be told that their input is screened. Compliance teams should record these decisions as evidence. DeployOpen prepares the controls and documentation, but the policy decisions and legal interpretation stay with your organization and counsel.
How long does guard testing take with many rules and several providers?
No testing duration is supplied for this work, so confirm it in the written scope rather than relying on a general figure. Testing time grows with the number of rules, the languages your users write in, the number of provider integrations, and the number of routes that must behave the same way. Plan for three test groups: prompts that should pass, prompts that should be blocked, and adversarial prompts such as injection attempts. Each group needs review of false positives and false negatives, and every provider adds its own error and timeout cases. Ask for the test plan, the acceptance criteria and the number of tuning rounds included in the quote.
Do moderation providers see our prompts and responses?
Hosted moderation services receive the text they are asked to check, so prompts sent to them leave your network. This applies to the AWS and Aliyun content moderation plugins and to Lakera Guard, because each one sends the content to the provider's service for classification. Confirm each provider's data handling terms, including where the data is processed, how long it is kept, and whether it is used for model improvement. If prompts must stay inside your network, the options are narrower. LLM Guard can run as an API server, but the project is archived, as described in the next question, so it is not a sound basis for a new rollout. Decide the data flow before choosing a provider.
Does upgrading to APISIX 3.18 or later change existing guard behavior?
Yes, the 3.18.0 release changed several behaviors that affect existing configurations. The ai-aws-content-moderation plugin now moderates the decoded prompt in the access phase and requires ai-proxy or ai-proxy-multi on the route. The ai-aliyun-content-moderation plugin checks the latest user turn by default. A new fail_mode option was added for AI plugins, and its skip default lets non-AI traffic through. Review each route that uses a moderation plugin before upgrading, test blocked and allowed prompts on a staging copy, and confirm the fail mode you want. The APISIX changelog lists the full set of changes for the release.
Should we still use LLM Guard for prompt and output scanning?
For a new rollout, no. The protectai/llm-guard repository was archived by its owner on 9 July 2026, and its README states that the project and its models on Hugging Face are no longer under active development or maintained. The last PyPI release, 0.3.16, was uploaded on 19 May 2025 and requires Python 3.10 or newer but below 3.13. The project is MIT licensed and documents 15 input scanners and 20 output scanners. Existing deployments can keep running while you plan a move to a maintained option such as NeMo Guardrails. Ask the partner to inventory any LLM Guard use, assess the security exposure of keeping it, and include the migration in the scope.
What can a gateway prompt filter miss that model-level controls must cover?
A gateway filter inspects the text that passes through a route, so it catches known patterns, banned topics and content flagged by the provider. It does not see everything the model will do with that text. Indirect prompt injection in documents retrieved by a RAG pipeline, harmful output that appears only in combination with earlier turns, tool calls made by an agent, and responses that break policy without matching a rule can all pass a gateway check. Covering these needs model-level and application-level controls, such as system prompt design, output checks, restricted tool permissions and red team testing. Treat the gateway as one layer in the design, and ask for those other layers to be named in the scope.
How DeployOpen scopes and supports prompt guard work
DeployOpen plans, implements and documents prompt and content guards in Apache APISIX inside your own infrastructure. Engagements start from a written scope that names which guard plugins, routes and consumers are covered, and each guard is tested against your own traffic before it is trusted in production.
How we scope
- Inventory which LLM routes, consumers and providers need prompt or response filtering, and where the guard sits in the request path
- Decide between ai-prompt-guard, ai-aws-content-moderation, ai-aliyun-content-moderation and ai-lakera-guard based on your providers and data rules
- Agree in writing what happens on failure, including fail mode, and which owner accepts each residual risk
How we build
- Configure guard plugins on APISIX routes with ai-proxy or ai-proxy-multi, matching the version-specific behavior of your target release
- Build a test set of benign, borderline and hostile prompts and responses, and record how each guard decision is logged
- Confirm that non-AI traffic is handled as intended under the skip behavior and fail mode you chose
How we hand over
- Deliver the APISIX configuration as code, with guard settings, route bindings and version pins documented
- Hand over test results, known gaps and the list of controls that remain at the model or application layer
- Train your operators on how to read guard logs and how to change a rule safely in your environment
How we support
- Engagements are fixed-scope, and scope, deliverables and timelines are agreed in writing before work starts
- Ongoing support, upgrades and continuous testing of guard rules are available as a retainer
- Re-test guards after APISIX upgrades or provider model changes, and review any replacement of archived guardrail components
