Routing to self-hosted LLMs with Apache APISIX: when a gateway in front of your models pays off
Apache APISIX gives teams that run models with vLLM or Ollama one controlled entry point. Its ai-proxy plugin translates client requests into the format each upstream expects, while authentication plugins and per-consumer controls decide who may call which model and how much they may use. Internally, APISIX routes requests to the right model server, so applications never need the address or credentials of each backend. A gateway earns its place when several applications or teams share the same models, when you need to tell callers apart, or when some traffic must leave your network for a cloud provider while the rest stays inside it.
vLLM and Ollama each expose their own API surface. A vllm serve process offers OpenAI-compatible endpoints such as /v1/chat/completions and /v1/embeddings, and its --api-key option does not cover every path, including /metrics and /health. The vLLM security documentation recommends a reverse proxy that allowlists only the endpoints meant for users. Ollama listens on 127.0.0.1 port 11434 by default and its local API requires no authentication. Neither runtime provides per-consumer identities, quotas or budgets on its own, so any control that separates one team from another has to sit in front of them.
Before adding the gateway, confirm three things: which endpoints each backend must expose to callers, which identity source your consumers use, and whether any upstream is a cloud provider that will receive prompts. Ask the team that owns the models whether they have already bound vLLM or Ollama to a private interface, since that decides where APISIX must sit. Treat the gateway as one more production component with its own upgrade, logging and failure plan, and check the current APISIX and plugin behavior in the APISIX ai-proxy plugin documentation before you commit to a design.
Which upstream and deployment options fit your models
This table compares the upstream options for models you run yourself or buy from a cloud provider, and the APISIX deployment modes that carry them. Read each row against your user count, concurrency needs and where your prompts are allowed to travel. The main takeaway is that the serving engine, the plugin that routes to it and the deployment mode should be chosen separately.
| Option | Best fit | Trade-off for multi-user production | Confirm before choosing |
|---|---|---|---|
| vLLM behind ai-proxy | Production serving of models on GPUs you control, with an OpenAI-compatible API | Serves many concurrent users well when accelerators are sized for it, but the operator owns capacity, upgrades and scaling | Accelerator vendor support and model compatibility on your hardware |
| Ollama behind ai-proxy | Smaller teams, development and evaluation, or single-host deployments | Default concurrency is low (1 parallel request per model), so multi-user traffic needs OLLAMA_NUM_PARALLEL and capacity tuning | Your user load against OLLAMA_NUM_PARALLEL, OLLAMA_MAX_LOADED_MODELS and OLLAMA_MAX_QUEUE settings |
| Cloud OpenAI-compatible provider via ai-proxy | Workloads where a hosted model is approved for the data involved | Prompts leave your network, so data classification and provider terms govern use | Provider data handling terms and the regions where requests are processed |
| ai-proxy (single upstream) | One model endpoint with one set of credentials and no failover requirement | Simplest configuration, but a failed upstream means failed requests | Whether a single endpoint meets your availability target |
| ai-proxy-multi (balancing or failover) | Mixing self-hosted and cloud instances, or keeping traffic flowing when one fails | Adds weights, priorities and retry rules to test, and retries on 429 or 5xx can add latency | Unique instance names and retry limits for your APISIX version |
| Traditional deployment mode | A single site where the gateway and its configuration store run together | Simplest to operate, but control and data planes share failure domains | Ports 9080 for traffic and 9180 for the Admin API are exposed only where needed |
| Decoupled deployment mode | Several gateway data planes that must share one configuration, typically across sites or zones | Separate planes and a shared etcd cluster add components to secure and back up | Network paths from each data plane to the shared etcd cluster |
| Standalone deployment mode | Environments that avoid etcd and manage configuration as files or through the API | Config changes come from file or API, so version control and review must be designed in | Whether your change process can manage conf/apisix.yaml or the API safely |
How to set up APISIX in front of vLLM or Ollama
This sequence places APISIX between your applications and a model server you run, so every prompt passes through one gateway with authentication and usage logging before it reaches the model. Each step ends with a check, and the cloud fallback is added only after the internal path has been verified.
Confirm the model server is private
Your platform team starts by checking where the model server listens. For Ollama, the default bind address is 127.0.0.1 on port 11434, so a server on another host needs OLLAMA_HOST set deliberately, and that address should sit on an internal network segment. For vLLM, the --api-key option does not cover every endpoint, and the vLLM security documentation recommends a reverse proxy that allowlists only the endpoints meant for users. Confirm that the model host accepts connections from the APISIX data plane and from nothing else. Done means a connection test from a host outside that path fails, and a connection test from the APISIX host succeeds. Confirm the exact network rules with your own security team, since they depend on your environment.
Define the upstream and route
Next, the APISIX administrator creates a route that uses the ai-proxy plugin and points it at the internal model endpoint. The plugin translates the request into the format of the chosen provider, and for vLLM or Ollama this is an OpenAI-compatible upstream with the model name set in the configuration. Keep the route narrow, exposing only the chat completion path your applications need instead of the whole server. Confirm the configuration fields against the current ai-proxy documentation for your APISIX version before you apply them. Done means the route appears in the Admin API or your configuration file, and a request to the gateway's client port returns a response from the model, not an error from the proxy.
Add consumer authentication
Now the gateway needs to know who is calling. Create one consumer per application or team and attach an authentication plugin such as key-auth to the route, so requests without a valid credential are rejected before they reach the model. Store each credential in your secrets system rather than in plain configuration files. Confirm the client identity model with your security and compliance leads, including how consumers are named and who may create them. Done means a request without a key returns an authentication error, and a request with a valid key for one consumer succeeds. Keep a record of which consumer made each call, because later steps depend on it.
Test a chat completion through the gateway
With the route and authentication in place, send a chat completion request to the gateway using the OpenAI-compatible request format that your applications already use. Send the request from the application host, not from the model host, so the test follows the path that production traffic will take. Check that the response text is correct, that the response comes back through APISIX, and that the model server logs show a request from the APISIX host only. Done means the test completes end to end with a valid consumer key and the model server shows no direct client connections. Record the request and response for your change file.
Check token usage in logs
Confirm that the gateway records what the plugin is designed to log, including token usage, the model name and time to first response, as described in the ai-proxy documentation. Look at the APISIX access or plugin logs for the test request and match the consumer identity to the usage record. Also check the vLLM metrics endpoint if your monitoring team needs server-side figures, and remember that the /metrics endpoint should not be exposed to users. Confirm log destinations and retention with your compliance owner, since logs can contain sensitive metadata. Done means one test call produces a log entry with the consumer, model and token counts that you can find without querying the model server directly.
Add the cloud fallback after verification
Only after steps one through five pass should the team add a cloud provider. The ai-proxy-multi plugin can balance traffic and retry on rate limits, 429 or 5xx responses, across instances, but it sends prompts to a third party when the cloud path is used. Decide with your data owners which request types may leave your network, and document that decision before you change routing. Add the cloud instance with a lower priority, confirm the provider's data handling terms, and run a failover test by stopping the internal model in a controlled window. Done means the internal path still serves requests when it is healthy, and the failover test shows traffic moving to the cloud instance and logging under the same consumer.
Mixing self-hosted and cloud providers without moving prompts
The first decision is which requests may leave your network at all. Before APISIX sends anything to a cloud model, the organization should classify prompts by data sensitivity and record which classes may reach an external provider. A cloud fallback is only safe once that classification and the data handling rules behind it have been signed off by the people who own the data, security and compliance. Without that sign-off, a failover rule that looks like resilience becomes an unreviewed transfer of regulated data to a third party.
In APISIX, the separation is best enforced by route design rather than by hoping clients choose correctly. Give internal traffic its own route that points only at the vLLM or Ollama upstream, and give any external provider its own route with a separate consumer group and authentication. Keep the ai-proxy-multi fallback list on the internal route free of cloud instances unless the classification review approved that path. Confirm in your own environment that the consumer and route rules block a cloud route from being reached by credentials issued for internal use.
Ollama's documentation states that when it runs locally it does not see your prompts or data, and setting OLLAMA_NO_CLOUD=1 or the equivalent server configuration turns off its cloud features, which is a useful control to set on every self-hosted node. Direct cloud inference at ollama.com requires an API key sent as a Bearer token, so the key should be treated as a credential that only the external route holds, never one that clients or internal services carry. Ask your team to verify the setting on each host and to test that a request on the internal route cannot reach a cloud endpoint.
Keeping prompts and model endpoints inside your network
Self-hosted models keep prompts on your hardware only if the endpoints behind APISIX are reachable solely through the paths you intend. The cards below cover where vLLM, Ollama and cloud upstream credentials leak exposure by default, and what to check at the gateway and network layers before routing live traffic.
vLLM API key does not cover everything
The --api-key option protects the /v1, /v2, /inference and /cohere paths, but not /invocations, /pause, /resume, /abort_requests, /metrics or /health. The vLLM docs warn against relying on the key alone and recommend a reverse proxy that allowlists only user-facing endpoints. At the gateway, expose only the OpenAI-compatible routes your applications call, and test that every other path returns a denial from outside the network.
Multi-node vLLM traffic is unauthenticated
In a multi-node deployment, communication between nodes is insecure by default, with no authorization or encryption. The vLLM security docs say this traffic must be protected by placing the nodes on an isolated network. Confirm that the node-to-node network is separate from the network where APISIX and client traffic run, and that no inference node is reachable from user subnets.
Ollama binds to localhost by default
Ollama listens on 127.0.0.1 port 11434 by default, and its local API requires no authentication. Exposing it is a deliberate step: changing the bind address with OLLAMA_HOST and placing a proxy in front. When APISIX is the proxy, add the authentication and consumer checks at the gateway, and confirm that no host firewall rule opens 11434 to other networks.
Cloud upstream API keys
Mixed routes to cloud providers mean the gateway holds provider API keys that can be billed and misused if exposed. Store them in a secrets system you control, not in route definitions committed to source control, and limit which consumers can trigger each upstream. Ask who rotates keys, how often, and how a leaked key is revoked without a gateway restart. The Ollama cloud API also uses Bearer keys, so apply the same handling if you route to it.
Metrics endpoints leak operational data
vLLM serves Prometheus-format metrics at /metrics, covering request counts, token throughput, latency and KV cache use. Those figures reveal load patterns and model usage to anyone who can reach the port. Keep /metrics off the gateway's public routes, scrape it from your monitoring network, and decide who may read dashboards built from it.
Questions teams ask before routing self-hosted LLMs through APISIX
What drives the cost of routing self-hosted LLMs through APISIX?
Cost comes from four separate places: the gateway setup, the model server operations, the GPU or other accelerator capacity, and the people who run it. APISIX and the ai-proxy plugins are open source under Apache-2.0, and vLLM has no paid edition, so license fees are not the main line item. The larger costs are accelerator capacity, which is usually recurring, and staff time, which covers initial build and ongoing operations. Ollama is free to run on your own hardware, but its paid cloud plans (Pro, Max, Team and Enterprise) are a separate purchase for cloud-hosted models and do not change the cost of self-hosting. Ask any quote to separate one-time build work from recurring support, and to state which hardware and cloud accounts it assumes.
How long does a self-hosted LLM routing project take?
Duration depends on how many model servers you run, how many providers you mix with them, and how many consumers need authentication and quotas. A single vLLM instance behind one APISIX route is a much smaller job than several model servers across GPU nodes with failover to cloud providers, consumer-level keys and token limits, and metrics feeding your monitoring. Other drivers include accelerator procurement and lead times, whether the GPU nodes are already provisioned, how well your network team can isolate the model servers, and how many approval steps your security review requires. Your partner should give a duration only from the written scope, and you should ask for the milestones and the client inputs each one depends on.
What does the work need from our team?
Expect to provide access and decisions in several areas. You need owners for the GPU or accelerator hosts, the network segment where model servers will sit, and the identity source used for consumer authentication. Someone must decide which models and providers are approved, which teams get which quotas, and how long prompt logs may be kept. Your security team should review the rule that exposes only user-facing endpoints from vLLM, because the vLLM security documentation warns that its API key does not cover every endpoint. Platform engineers will also need to run the APISIX and model server environments after handover, so plan for staff who will own upgrades and on-call response.
Can our own team build this without a partner?
Yes, if your team already runs Kubernetes or Linux services, has GPU experience and can review APISIX configuration. The components are open source, and the APISIX deployment modes documentation covers traditional, decoupled and standalone setups. The hard parts are less about installing software and more about correct isolation of vLLM, which the project says must sit on an isolated network for multi-node communication, safe exposure of only the endpoints users need, consistent consumer keys and quota design, and verification that prompts stay inside your network. A partner can take that scope under a written agreement, while your team keeps ownership of the outcome. Decide early which parts you want to keep in-house, such as model selection, and which you want to hand over.
Does APISIX replace our existing gateway or run beside it?
Either is possible, and the choice depends on your current gateway and risk tolerance. Many teams start by running APISIX beside the existing gateway for only the LLM routes, which limits the blast radius and lets them compare authentication and logging behavior before moving other APIs. A full replacement requires moving routes, authentication plugins and traffic policies, and it needs a parallel run and a rollback path. Because APISIX can be deployed in traditional, decoupled or standalone mode, you can match your existing operating model. Confirm the coexistence design, DNS and TLS cutover steps and the rollback procedure before the build begins.
What is the difference between ai-proxy and ai-proxy-multi for self-hosted models?
The ai-proxy plugin sends requests to a single configured LLM or embedding provider, transforming the request into that provider's format and logging token usage, model and time to first response. The ai-proxy-multi plugin balances traffic across several instances using weighted round robin, consistent hashing or semantic selection, and it can fall back on 429 or 5xx responses with a configured retry limit. For a single vLLM server exposing OpenAI-compatible endpoints, ai-proxy may be enough. Once you combine self-hosted models with cloud providers, or run several model replicas, ai-proxy-multi is the usual choice. Confirm the exact provider options and retry behavior against the plugin documentation for the APISIX version you deploy.
Do prompts stay inside our network when we use Ollama or vLLM?
They can, but only if the design keeps every hop inside your boundary. Ollama states that when it runs locally it does not see your prompts or data, and cloud features can be turned off with OLLAMA_NO_CLOUD=1. Its local API on port 11434 has no authentication by default and binds to 127.0.0.1, so exposing it requires a proxy such as the APISIX gateway in front of it. Direct cloud inference at ollama.com requires an API key, so any route to a cloud provider sends prompts outside your network. For vLLM, the --api-key option does not protect every endpoint, so rely on the reverse proxy allowlist the vLLM security documentation recommends. Ask your partner to document each route and to test that cloud-bound traffic is blocked where policy requires it.
Running the gateway and model servers after go-live
Day-two work for this design has three parts: keeping the model servers current, proving that each upgrade leaves the APISIX routes working, and watching capacity. vLLM ships new minor releases about every two weeks, and Ollama ships frequent releases as well, so an unplanned upgrade is a routine risk, not an occasional one. Assign one owner to read the vLLM GitHub releases and the Ollama GitHub releases, and decide in advance which versions go to a staging copy first. Because the gateway sits in front of both servers, an upgrade can change request handling, endpoint paths or response fields that your ai-proxy and ai-proxy-multi routes depend on, so treat each model server upgrade as a change to the gateway too.
Before a model server goes to production, replay a fixed set of requests through a staging APISIX instance that mirrors the production route file. Include a chat call, a completion call, an embedding call if you use one, a request from each consumer key, and a deliberate failure to confirm that fallback to your cloud provider still triggers. Keep the route configuration in version control so you can diff it against the last known good state. Confirm with your vendor or your own tests which endpoints your server version exposes, since the vLLM security docs recommend allowlisting only the paths meant for users and that list can change between releases.
Monitor both layers. vLLM exposes Prometheus-format metrics at /metrics, covering request counts, token throughput, latency histograms and KV cache use, and that endpoint should be scraped only from your monitoring network, not routed through the gateway to consumers. Compare those numbers with the usage and latency fields APISIX logs for each request. For Ollama, capacity is governed by OLLAMA_NUM_PARALLEL, which defaults to 1 parallel request per model, and OLLAMA_MAX_QUEUE, which defaults to 512 queued requests. Tune these against measured latency and queue depth rather than defaults, and set the queue limit so that a backlog produces a clear error at the gateway instead of silent timeouts for users.
What DeployOpen delivers and what you provide
The work is a fixed-scope engagement with a written scope before anything starts. DeployOpen delivers the APISIX configuration, controls and evidence, while your team supplies the model hosts, network access, identity source and accelerator capacity the design depends on. Retainers cover what happens after handover.
Written scope and fixed deliverables
The scope names which model hosts, APISIX routes, consumers and providers are in, and which are out. Agree the list of upstream models and providers in writing first, because every route, weight and quota follows from it. Ask for each deliverable to be listed with its acceptance test so you can tell when the work is done.
APISIX routes and upstream configuration
DeployOpen configures the ai-proxy routes that point to your vLLM or Ollama servers as OpenAI-compatible upstreams, plus any cloud providers you keep. Expect the configuration to be stored as code in your repository. Confirm which deployment mode (traditional, decoupled or standalone) your platform team wants before the routes are written.
Consumer authentication and quotas
Each application or team receives its own consumer identity, with API keys or JWT validated at the gateway. Quotas and per-consumer limits are configured so that one client cannot exhaust shared GPU capacity. Ask which identity source issues tokens and who owns consumer onboarding after handover.
Network controls and endpoint allowlisting
DeployOpen restricts the gateway so that only user-facing inference endpoints are reachable, and keeps vLLM management and metrics paths off the public route. Your team provides the network segments, firewall rules and certificates. Confirm that model hosts accept traffic only from the gateway, because the vLLM security guidance expects an isolated network.
Verification tests and documentation
Verification covers routing, failover, authentication rejection, quota enforcement and confirmation that prompts stay on your network when a self-hosted route is used. You receive test scripts, results and runbooks. Ask for the test cases to be shared before sign-off so your security team can review them.
Client inputs and ongoing retainers
Your team provides the model hosts, accelerator capacity, network access and an identity source to integrate with. Ask the quote to state which of these come from you and which DeployOpen will procure or configure. Ongoing support, upgrades and continuous testing are available as retainers once the fixed-scope work is handed over.
How DeployOpen works on this
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, so model traffic and prompts stay in networks you control. For self-hosted LLM routing, we agree the written scope with your platform, security and model owners before any gateway configuration changes.
How we scope
- Inventory the vLLM or Ollama endpoints, their models, and which consumers and applications will call them through APISIX.
- Agree which OpenAI-compatible routes are exposed, which are blocked, and how Admin and metrics endpoints stay off the public path.
- Define consumer groups, per-consumer authentication and quota requirements, and any cloud providers that must sit beside self-hosted models.
How we build
- Configure ai-proxy routes to each self-hosted upstream, then test request format, token logging and time to first response against real clients.
- Set up ai-proxy-multi priorities and fallback rules between self-hosted and approved cloud providers, with failure tests for timeouts, 429 and 5xx responses.
- Verify network isolation so vLLM nodes and Ollama listeners accept traffic only from the gateway, and that prompts are not sent to any external provider unless approved.
How we hand over
- Deliver the route and plugin configuration as version-controlled files, with a record of each upstream, consumer and fallback rule.
- Document the test results, the allowlist of exposed endpoints, the network boundaries, and the steps to add a model or retire one.
- Walk your platform and security teams through failover, quota reset and log retention so they can operate the gateway after handover.
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts and a quote following from that scope.
- Ongoing support, upgrades of APISIX and related components, and continuous testing are available as a retainer.
- Ask any partner, including DeployOpen, to confirm the vLLM and Ollama versions, APISIX version and plugin behavior in your environment before go-live.
