When an Apache APISIX AI gateway fits your LLM traffic
When several teams call LLM providers directly, each team usually holds its own API keys, retry logic and logging, and nobody can see total spend or enforce a common policy. A self-hosted Apache APISIX AI gateway puts one controlled layer in front of those providers. The APISIX AI plugins, including ai-proxy and ai-proxy-multi for provider routing, token-based rate limits, prompt guards and LLM metrics, let you apply the same rules to every application that sends a prompt, without changing each application first.
A self-hosted gateway is the right layer when the organization needs one place to set provider access, budgets, content rules and audit logs, and when prompts or responses must stay inside infrastructure you control. It fits best when at least two teams or applications share providers, or when a regulator or internal policy requires central logging of model traffic. Confirm the plugin names, their configuration keys and the features each one supports against the APISIX documentation for the version you plan to run, because AI plugin behavior changes between releases.
The gateway does not replace the model itself. If you run your own models, serving still belongs to a serving engine, and the gateway only routes requests to it. It also does not replace application logic such as retrieval design, prompt construction, tool calls or evaluation, which stay in the application or in a separate workflow platform. Token counting and budgets are only as accurate as the usage data each provider returns, so check that before you rely on them for chargeback. Before you commit, run an inventory of which applications call which providers, which data classes they send, and which teams own the budgets. That inventory decides the route design, the consumer model for authentication and the scope of the logs you must retain. If the inventory shows only one application and one provider, a direct integration with a provider SDK may be the simpler choice.
Which provider routing option fits each team
The ai-proxy plugin fits a team that sends all LLM traffic to one provider account and wants the simplest route to govern. A single route forwards requests to one upstream model API, so the gateway adds authentication, rate limits, logging and guards without a routing decision to make. This suits a pilot, an internal assistant with one approved vendor, or a team whose contract allows only one model source.
- Configuration scope: store the provider API key in the gateway's secret handling rather than in client applications, and confirm the exact provider fields in the plugin documentation for your APISIX version.
- Failover behavior: there is no second backend, so an outage or throttling at the provider surfaces directly to callers unless the application retries.
- Cost visibility: all spend lands on one account, which makes attribution simple, but per-team breakdowns still need consumer or header-based labels in your metrics.
- Change cost: moving to another provider later means changing the route and client expectations, so keep the request format consistent across applications.
- Ownership: one team owns the provider contract, key rotation and quota limits.
Before choosing this option, confirm with the provider's terms and your data policy that a single vendor meets your resilience and data residency needs.
The ai-proxy-multi plugin fits teams that use more than one hosted model, want traffic spread across them, or need a fallback when one provider degrades. It sends requests to a set of configured instances and applies balancing and failover rules that you define. This is the option for platform teams serving several application owners who need reliability without changing client code.
- Configuration scope: define each provider instance with its own key, model and weight, and test each one against the same prompts so output differences are known.
- Failover behavior: failover depends on which errors and timeouts trigger a switch. Check how retries interact with streaming responses and with requests that have already consumed tokens.
- Cost visibility: spend is split across accounts, so tag requests by consumer or route and reconcile provider invoices against gateway metrics each billing period.
- Model consistency: different providers may vary in output quality, context length and safety behavior, so a failover can change answers users see.
- Ownership: the platform team usually owns routing rules, and each provider contract has its own owner.
Confirm the balancing and failover options available in your APISIX version and test them with real provider error responses before relying on them.
Routing to self-hosted models fits teams with data that cannot leave their network, teams running their own inference servers, or teams that need predictable capacity. The gateway then fronts model servers inside your infrastructure, so the same authentication, token limits and logging apply to internal and external models. Serving engines and model choice are separate decisions that belong to the model serving tooling, and this option only covers the gateway side.
- Configuration scope: route to internal model endpoints through upstream definitions, apply the same consumer and token policies, and confirm which AI plugin fits the serving API your servers expose.
- Failover behavior: a self-hosted pool has fixed capacity, so failover to a hosted provider may be the only overflow path and needs an explicit policy on data handling.
- Cost visibility: hardware, power and staff are fixed costs outside the gateway, while the gateway can report request and token volume per consumer to size them.
- Network controls: keep model endpoints on private networks and restrict which services can reach them.
- Ownership: the infrastructure team owns capacity and model hosting, and the platform team owns the gateway policy in front of it.
Confirm with your model server vendor which API formats and token counts it reports, because these determine what the gateway can meter.
Which AI plugin handles each gateway job
This table maps each gateway job in an LLM setup to the APISIX AI plugin that usually handles it, the configuration decision that shapes the design, and what to check in the current documentation before you commit. Plugin names and option details change between releases, so use the table as a starting point for your own review. The main takeaway is that routing, budgets, guards and metrics sit in separate plugins, so each needs an owner and its own tests.
| Gateway job | Plugin | Key configuration decision | What to confirm in current docs |
|---|---|---|---|
| Provider routing to one LLM provider | ai-proxy | Provider, model, and where the provider API key is stored | Supported providers, auth fields, and how keys are referenced |
| Provider routing across several providers | ai-proxy-multi | Balancing or failover weights, and fallback order between providers | Balancing algorithms, failover conditions, and priority options |
| Token-based budgets and limits | ai-rate-limiting | Limit keys (per consumer or per route) and the token counting basis | Which token counts are enforced and the reset window options |
| Prompt guards and templates | ai-prompt-guard, ai-prompt-template | Allowed and denied patterns, and whether templates run before or after guards | Plugin execution order and how blocked requests are reported |
| Content moderation | Moderation plugins (for example, ai-aws-content-moderation) | Which moderation service applies and whether unsafe content is blocked or logged | Which moderation providers are supported and their region and data handling terms |
| Retrieval-augmented generation (RAG) | ai-rag | Where retrieval context is sourced and how it is injected into prompts | Supported vector stores, embedding providers, and injection options |
| LLM metrics for cost tracking | prometheus and logging plugins | Which token and model fields are exported, and where logs are shipped | Exact LLM metric names, labels, and whether usage fields are populated per provider |
How to set up the AI gateway in phases
This sequence takes APISIX from a chosen deployment mode to a gateway that routes across approved LLM providers, enforces per-consumer token limits and guards prompts, and reports usage for cost tracking. Each step closes with a check so failures surface before the next layer is added.
Confirm the deployment mode and ports
Your platform or infrastructure lead chooses between traditional, decoupled and standalone modes, based on whether you want etcd-backed dynamic configuration or a file-driven setup under version control. In traditional mode, client traffic arrives on port 9080 and the Admin API listens on port 9180, so confirm those ports against your network policy and load balancer design. Done means a running APISIX instance answers a test request on the data port and the Admin API responds to an authenticated call, with the chosen mode written down in your architecture record.
Define approved providers and store their keys
Your security and AI platform owners list which LLM providers and models are approved, and who may use each one. Provider API keys go into a secrets store you control, not into route definitions or plain configuration files, and APISIX references them from there. Done means each approved provider has a named credential, a documented owner and rotation responsibility, and a test call through the gateway returns a response without the key appearing in any exported configuration.
Configure ai-proxy-multi for provider routing
The gateway team creates a route that uses the ai-proxy-multi plugin, listing the approved providers with their weights and fallback order. Confirm the exact provider options and configuration keys against the plugin documentation for your APISIX version before writing them. Done means a request to the route reaches the primary provider, and a logged or traced response shows which upstream served it.
Add consumer-based token limits
Platform engineers define a consumer for each team or application, authenticate those consumers with the authentication method your policy requires, and attach token-based rate limits to each one. Limits should match the budgets your finance and AI governance owners approved, and the numbers come from those owners, not from defaults. Done means a consumer that exceeds its allowance receives the rejection you designed, and other consumers continue to get responses.
Add prompt guards and content checks
Security engineers add prompt guard and moderation controls to the route, covering the content categories and blocked patterns your acceptable use policy defines. Start with logging or a non-blocking mode where the plugin supports it, review what it flags on representative traffic, then enforce. Done means test prompts that violate policy are blocked or flagged as designed, and legitimate test prompts pass without modification.
Enable LLM metrics and usage logging
Observability engineers enable the LLM metrics plugin and route its output to the metrics and log systems your organization already runs. Each record should identify the consumer, provider, model and token counts, without storing prompt or response content unless your data policy allows it. Done means a test request produces a metric and log entry with those fields, visible in your dashboard.
Test failover and cost reporting end to end
The team disables or degrades the primary provider in a test environment and confirms that ai-proxy-multi fails over to the next provider, while consumer limits and guards still apply. Then they run a reporting query over the collected metrics and compare token totals against a known set of test requests. Done means failover behavior matches the design, the cost report reconciles with the test traffic, and the results are recorded for the go-live review.
Security controls for an LLM gateway
A gateway in front of LLM providers concentrates API keys, user identity, prompt content and spend in one place. The cards below cover the controls that tie those elements together, and each one names a decision to make before the gateway carries production traffic.
Provider key handling
Each provider API key grants direct spend and data access, so the gateway should be the only system that holds it. Decide where keys are stored, who can read them through the Admin API, and how they rotate without a route change. Ask whether your secrets store can feed APISIX configuration and whether keys can be scoped per provider and per environment. Test that a key never appears in logs or error responses.
Admin API exposure
In traditional mode APISIX serves the Admin API on port 9180, separate from client traffic on port 9080. That API can change every route, plugin and upstream, so it must never sit on a network that application users or the internet can reach. Restrict it to automation and administrators, require strong credentials, and confirm that the Admin API path is absent from the public ingress before go-live.
Prompt and output content controls
Prompt guard and moderation plugins can reject or screen requests before they reach a provider, and they can screen responses on the way back. Define which categories of content are blocked, which are logged for review, and what the client receives when a request is refused. Test these rules with your own sample prompts, since a rule that passes benign text is not evidence it catches policy violations.
Logging prompts that hold personal data
Prompts often contain names, account numbers or patient details, and the gateway's access logs can copy them. Decide which request fields are logged, whether bodies are captured at all, and how long logs are kept. Mask or omit personal data at the gateway where possible, and confirm who can read the log store. Treat the log pipeline as a data store in its own right.
Consumer identity for token limits
Token-based rate limits only mean something when each request is tied to a known consumer, such as a team, application or service account. Without that identity, one caller can exhaust a shared budget for everyone. Map each consumer to an authentication method, set limits per consumer, and check that the identity is verified by the gateway rather than trusted from a client-supplied header.
Trust boundary with providers
Everything sent past the gateway leaves your control, including prompts, retrieved documents and user identifiers. Document which providers and regions are approved, which data classes may be sent to each, and whether a private or self-hosted model handles the sensitive routes. Confirm contractual terms on data use and retention with each provider, since the gateway cannot enforce those terms on its own.
What teams ask before running an AI gateway on APISIX
What drives the cost of an APISIX AI gateway implementation?
Scope drives most of the cost: how many LLM providers you connect, how many teams and applications send traffic through the gateway, and how much policy you need (token limits, prompt guards, content moderation, logging). Each provider adds account setup, credential handling and routing tests. Recurring costs include infrastructure you run for the gateway, etcd, logging and metrics storage, and upgrades. APISIX itself is licensed under Apache-2.0, so the software carries no license fee. Ask any partner for a quote that lists providers, environments, plugins, integrations and handover separately, and states which items are one-time and which recur.
Can our internal platform team build an APISIX AI gateway without a partner?
Yes, if the team already runs APISIX or NGINX-based infrastructure and has people who can write and test gateway configuration. The work is familiar: routes, consumers, authentication and plugin configuration. The new parts are provider-specific behavior, token accounting and LLM metrics, which need testing against real provider responses. Teams without APISIX experience usually spend the first phase learning the Admin API, plugin precedence and deployment mode choices. Decide early whether you want an internal build, a partner-led build, or a partner for design and review only.
What does our team need to provide for the gateway project?
Plan to provide the following before build work starts:
- Accounts and API keys for each approved LLM provider, held in your secrets store
- An identity provider and group model for who may call which models, such as Keycloak, Entra ID or Okta
- Infrastructure for the gateway in your cloud account, data center or restricted network, with sizing based on expected request volume and concurrency
- Named owners for model access policy, token budgets and log retention
Sizing depends on your traffic profile, which should be measured during design rather than assumed. The APISIX deployment modes documentation covers the topology options you will need to choose between.
How long does a phased APISIX AI gateway rollout take?
Duration depends on scope, not on the gateway software. A first phase that routes one or two providers through ai-proxy, applies key-based access and emits basic logs is the smallest useful step. Later phases add ai-proxy-multi for provider failover, token-based rate limits, prompt guards, RAG or moderation plugins and cost dashboards, each with its own testing. Provider account approval and identity integration often set the pace more than configuration does. Ask for a phase-by-phase schedule in the written scope, with a go-live criterion for each phase.
Does APISIX replace LiteLLM-style AI proxies?
APISIX can cover some of the same jobs, such as routing to multiple providers, authentication, rate limiting and logging, because it provides AI gateway plugins alongside its general API gateway features. LiteLLM is a purpose-built LLM proxy with its own virtual keys and budget model. The right choice depends on whether you want one gateway for both APIs and LLM traffic, and on which budget and key model your teams need. Compare the features your teams require against the documentation for each product, and confirm specific capabilities with the vendor or project before you commit.
How do we validate that token cost tracking is accurate?
Validate it by reconciling gateway counts against provider invoices and usage reports for the same period and models. Send a known set of test requests, record the token counts the gateway logs, and compare them with what each provider bills. Check that every consumer, team and model appears in the metrics, and that failed or retried requests are counted the way your finance rules expect. Repeat the check after each provider or plugin change. Agree the acceptance tolerance in the scope before go-live, since the right threshold depends on your billing model.
What security controls should the AI gateway enforce?
Start with the controls that protect provider credentials and the gateway itself: keep provider keys in a secrets store rather than route files, restrict the Admin API to a management network, and use TLS for client and upstream traffic. Next, decide which consumers may call which models, and apply key-based or OpenID Connect authentication per consumer. Prompt guards and moderation plugins reduce misuse but do not replace review of what your applications send. Record who may change routes and plugins, and keep configuration under version control so changes can be traced.
Where should logs and prompts be stored when running the gateway?
Storage location is a data governance decision. Prompts and responses can contain personal or confidential data, so decide whether to log full payloads, redacted fields or only metadata such as model, token count and consumer. Log destinations are set in your gateway configuration, so confirm the logging plugins and retention controls you need in the APISIX documentation before design. Route logs to a system you control, set a retention period that matches your policy, and confirm with your compliance owner which regulations apply to the data flowing through the gateway.
How DeployOpen scopes and delivers an APISIX AI gateway
DeployOpen plans, implements, hardens and documents Apache APISIX as a self-hosted AI gateway inside infrastructure you control. Each engagement starts with a written scope that names the LLM providers, routes, token policies and telemetry in play, so the gateway can be tested and handed over against clear deliverables.
How we scope
- Inventory the LLM providers, self-hosted models and client applications that will send traffic through the gateway.
- Define which ai-proxy or ai-proxy-multi routes, fallback rules and token limits are in the written scope.
- Agree which prompt guards, moderation steps and LLM metrics your teams need before build work begins.
How we build
- Deploy APISIX in the traditional, decoupled or standalone mode your architecture and operations team choose.
- Configure provider credentials, consumer identities and key storage so no provider secret sits in route definitions.
- Test routing, failover, token-based rate limits and guard behavior with recorded requests before enabling production traffic.
How we hand over
- Deliver route and plugin configuration as versioned files, with a written description of each AI policy and its owner.
- Document the metrics pipeline used for LLM cost tracking, including what is collected, where it is stored and who reviews it.
- Provide a runbook covering provider outages, key rotation, configuration rollback and how to add a new model route.
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- Ongoing support, APISIX version upgrades and continuous testing are available as a retainer.
- Ask any partner to confirm how upgrades to the APISIX release line and its AI plugins will be tested before they reach production.
