Why a self-hosted AI gateway is the control point for model access
When several teams call multiple model providers with separate API keys, nobody can answer basic questions about spend, data flow or who sent what. A self-hosted AI gateway centralizes routing, keys, budgets, guardrails and logging in one layer between your applications and the model providers, so each control is set once and enforced for every caller. The point of the gateway is not a single product feature but a single place where policy is applied and evidence is produced.
LiteLLM is the option to evaluate first when the main problem is model access itself. It is MIT licensed, except for its enterprise directory, and it offers an OpenAI-format interface to more than 100 LLM APIs, with cost tracking, guardrails, load balancing and logging. It fits teams that want application developers to change one endpoint and let the gateway handle provider keys, virtual keys per team and spend limits. Before adoption, confirm which features you need sit outside the enterprise directory, since those carry a separate license, and check the current release (v1.105.1 was published on October 14, 2026) against your own test plan.
Apache APISIX fits better when an API gateway already governs your estate and AI traffic should follow the same authentication, rate limiting and audit path as other APIs. It is Apache-2.0 licensed and is described as both a cloud-native API gateway and an AI gateway, with 3.19.0 published on September 28, 2026. Confirm with the project which AI-specific plugins and provider integrations meet your requirements, and whether your team has the operational skills to run it. Many organizations will need to decide which of the two becomes the authoritative control point, because running both for the same traffic creates two sets of keys, budgets and logs.
Which requirements to agree before choosing a gateway
A gateway sits between every application and every model provider, so its design follows from a few upstream decisions. Agree these six points before comparing LiteLLM, Apache APISIX or any other option, because each one changes which features are required, where the gateway runs and what evidence it must produce.
Providers and models in scope
List every model provider, hosted and self-hosted, that applications must reach, along with the API formats your teams already use. A gateway that translates many provider APIs into one OpenAI-style format reduces application changes, but the list of supported providers and models changes often. Ask each candidate to demonstrate your specific providers, including streaming responses and tool calls, and confirm which model endpoints must stay on private networks.
Ownership of keys and budgets
Decide who issues virtual keys, who sets spending limits and who approves increases, whether that is a platform team, each business unit or each application. Central key management keeps provider credentials out of application code, but it also makes the gateway a single point of failure for every team. Check whether budgets apply per key, per team and per project, and how the platform reports spend back to the owners.
Guardrails and prompt data rules
Define which prompts may leave your network, which data classes must be masked or blocked, and which content checks run before and after a model call. Gateways offer guardrail hooks, but the rules themselves belong to your security and privacy teams. Confirm whether guardrails run inline and add latency, whether they can call your own classifiers, and whether blocked requests are logged in a way that does not store the sensitive data itself.
Log content and retention
Specify which fields each request must record, such as caller identity, model, provider, token counts, cost, policy decisions and timestamps, and which fields must never be stored, such as raw prompts for certain data classes. Retention periods must match your records policy and any regulation that applies. For high-risk AI systems under the EU AI Act, Article 19(1) requires providers to keep automatically generated logs for at least six months unless other law provides otherwise, so confirm with counsel which period applies to you.
Identity for applications and staff
Map how applications authenticate to the gateway with service credentials, and how staff reach any admin console or self-service key portal through your identity provider. Decide whether group membership from the identity provider drives budgets and model access. Verify which SSO protocols each candidate supports in its open-source edition, because some administrative features, such as single sign-on or audit views, may sit behind an enterprise license.
Availability and failure behavior
Set the expected uptime for the gateway and decide what happens when a provider fails, when the gateway itself is down, or when a budget is exhausted. Options include fallback routing to a second provider, queuing, or failing closed. Each choice affects user experience and data exposure differently. Ask whether the gateway can run as several replicas behind a load balancer and where its configuration and key store are kept.
Open-source gateway options and when each fits
This table compares LiteLLM, Apache APISIX and Kong Gateway as the layer between applications and model providers, covering best fit, licensing, AI features relevant to routing, keys, budgets, guardrails and logging, and operating effort. Read the licensing and feature cells as a starting point, since enterprise features and some guardrail or SSO capabilities may sit behind commercial licenses and must be confirmed for each release before you commit.
| Option | Best fit | Licensing and release | AI features relevant to this function | Operating effort |
|---|---|---|---|---|
| LiteLLM | Teams that need an LLM-specific gateway in OpenAI-compatible format across many model providers, with per-key budgets and cost tracking | MIT except the enterprise directory; v1.105.1 published October 14, 2026 | Calls more than 100 LLM APIs in OpenAI format, cost tracking, guardrails, load balancing and logging; confirm which guardrail and SSO options sit in the enterprise directory | Lower to moderate: a focused service, but you own its database, key management and upgrade path |
| Apache APISIX | Organizations that already run an API gateway estate and want AI routing, plugins and rate limiting in one platform | Apache-2.0; 3.19.0 published September 28, 2026 | Cloud-native API and AI gateway with plugin-based routing, authentication and rate limiting; confirm which AI proxy and guardrail plugins your release includes | Moderate: plugin configuration, etcd or equivalent state store, and gateway operations skills |
| Kong Gateway | Organizations with an existing Kong footprint or a need for a mature general-purpose API gateway with AI extensions | Apache-2.0; 3.9.4 published October 12, 2026 | General API gateway with AI-related plugins for model routing and policy; confirm which AI plugins and admin features are open source versus enterprise for your version | Moderate: database or declarative configuration, plugin governance and gateway operations skills |
Reference architecture components and how they connect
A self-hosted AI gateway is a small set of cooperating services rather than a single product. Each component below holds a distinct piece of state or policy, and the connections between them determine what fails when one part is down. Review each component against your own availability targets and data handling rules before fixing the design.
Gateway service and request path
The gateway receives every call from applications, applies authentication and routing, then forwards the request to a model provider. LiteLLM and Apache APISIX (with its AI gateway features) are the two options most often evaluated here. Confirm which one exposes the OpenAI-compatible interface your applications already use, and whether it can stream responses without buffering them. Because every model call passes through this service, it sits on the critical path and needs at least two instances behind a load balancer.
Database for keys and spend
A relational database stores virtual API keys, team and user budgets, and per-request spend records. Losing it means losing the spend history that finance and audit teams rely on, so it needs managed backups, point-in-time recovery and a tested restore procedure. Ask your implementation partner how the gateway behaves when this database is unreachable: some configurations fail closed, others serve cached keys for a limited period. Choose that behavior deliberately.
Cache and rate-limit store
A shared in-memory store holds rate-limit counters, short-lived key lookups and response caches. Keeping these counters in a shared store is what makes limits hold across gateway replicas; per-instance counters let a team exceed its budget by spreading requests. Treat cached responses as data: if prompts may contain personal information, decide whether caching is permitted, how long entries live and whether they are encrypted at rest.
Guardrail layer
Guardrails inspect prompts before they leave your network and responses before they reach users. They may block secrets, redact personal data, or flag disallowed topics. Guardrails can run inside the gateway or as an external service the gateway calls. Each check adds latency and a new failure point, so confirm the timeout behavior, whether a failed check blocks the request or lets it pass, and which policy version was applied to each call.
Model routing configuration
Routing rules map a logical model name to one or more provider deployments, define fallbacks, and set load-balancing weights. This configuration is typically kept in version-controlled files and applied through a deployment pipeline. Every change to a route alters where data is sent, so require review and record who approved each change. Confirm that credentials referenced by routes are never written into these files in plain text.
Log and metrics pipeline
Request logs, token counts, latency and guardrail decisions flow from the gateway into a log store and a metrics backend. Observability tools such as Langfuse or Phoenix can receive traces from the gateway, though each has its own license terms. Decide which fields are logged, whether prompt bodies are stored or only metadata, and how long records are kept. Article 19(1) of the EU AI Act sets a minimum retention period of six months for logs from high-risk systems, where other law does not provide otherwise.
Secrets storage for provider credentials
Provider API keys for model vendors must sit in a dedicated secrets manager, not in environment files or the gateway database in readable form. The gateway should fetch credentials at runtime and support rotation without a redeploy. Confirm who can read these secrets, whether access is audited, and how a leaked provider key is revoked across all replicas. This is the component auditors most often ask about first.
Build in-house, use a partner or buy SaaS for AI gateway control
This table compares running an open-source AI gateway such as LiteLLM or Apache APISIX in-house, with an implementation partner, or with a commercial SaaS gateway. Read each row as a question to answer for your own environment, since the right model depends on where your keys, prompts and logs may legally sit. The main takeaway is that open source shifts cost from license fees to setup, operation and upgrade work, and that you keep the most control when you own the deployment.
| Dimension | In-house on open source | Open source with a partner | Commercial SaaS gateway |
|---|---|---|---|
| Control over data path and keys | Full control. Prompts, responses, provider keys and logs stay in your cloud account or data center if you deploy them there. | Same deployment control as in-house, with the partner working inside your infrastructure. Confirm who holds administrative credentials and where backups are stored. | Traffic and logs pass through the vendor's environment. Confirm hosting region, key storage, subprocessors and whether you can bring your own keys. |
| Operating effort | Your platform, security and on-call teams run the gateway, provider routing, key rotation, monitoring and incident response. | Partner handles planning, hardening, backup, documentation and support under a written scope. Your team keeps ownership of day-to-day decisions. | Vendor operates the service. Your team configures policies, budgets and integrations, and reviews vendor change notices. |
| Cost structure | Recurring infrastructure, staff time and monitoring. Software license cost is typically nil for the gateway core, but enterprise features may be licensed separately. | One-time scoped implementation, then optional retainer for upgrades and support, plus the same recurring infrastructure as in-house. Ask for the quote to split these lines. | Recurring subscription, often priced by usage or tier, plus internal configuration time. Ask how overage, log retention and premium guardrails are billed. |
| Upgrade ownership | Your team tracks releases, tests provider compatibility and applies patches. Check each release note before rollout. | Partner plans upgrades, tests them in a staging environment and documents changes, if the retainer covers this. Confirm the cadence in writing. | Vendor deploys upgrades on its schedule. You may have limited control over timing and must validate changes that affect your policies. |
| Exit options | High. The gateway runs on your infrastructure, and configuration, keys and logs remain in formats you control. Plan for migration of routing rules and budgets. | High if the partner documents the configuration and hands over the repository, runbooks and infrastructure code. Confirm handover terms before work begins. | Lower. Exporting configuration, logs and usage history depends on vendor formats and contract terms. Confirm export rights before signing. |
| Evidence for assessors | You produce all evidence: configuration records, access reviews, log retention settings and change history. | Partner can prepare controls, runbooks and evidence artifacts for your assessment. Audit reports and certifications still come from independent bodies. | Vendor may supply its own security reports. Check whether they cover the gateway features you use and whether logs are available to you on request. |
How a gateway rollout runs from pilot to production
The sequence moves a gateway from an inventory of model providers and applications to a governed production service, with each phase producing a artifact the next phase depends on. Done in this order, the team avoids configuring keys, budgets and logs for traffic nobody has mapped.
Inventory providers, applications and requirements
The team lists every application that calls a model today, every provider and endpoint it uses, and the data each call may carry. Security, compliance, platform engineering and the application owners agree on the requirements: which models are approved, which users may call them, what must be logged and who reviews spend. The output is a signed requirements document and a provider inventory that names the data classification allowed for each provider. The step is done when no application calling a model remains unlisted and each owner has confirmed the list.
Run a pilot with one application and one provider
DeployOpen deploys the chosen gateway, such as LiteLLM or Apache APISIX, in the client's own infrastructure and routes a single application through it to one approved provider. The pilot tests request translation, failover behavior, timeouts and how the application handles errors returned by the gateway. The output is a pilot report that records the configuration used, the test cases run and the defects found. The step is done when the application runs in production-like conditions without changes to its business logic.
Define the key and budget model
The team decides how virtual keys map to applications, teams and users, how credentials for each upstream provider are stored and rotated, and what spending limits apply at each level. Platform engineering configures these in the gateway and tests that a key cannot reach a model outside its approved list. The output is a written key hierarchy, a budget table and a tested revocation procedure. The step is done when a test key exceeding its budget is blocked and the block appears in the gateway's records.
Configure guardrails and logging
Security engineers configure the input and output checks the requirements call for, such as content filtering, data loss prevention patterns or prompt injection screening, and decide which of these run before a request leaves the network. Logging is set to capture the fields the requirements name, with sensitive content handled according to the data classification. The output is a guardrail policy with test results showing known bad inputs are caught and a logging specification. The step is done when test traffic produces the expected log entries and the guardrail blocks are visible to the security team.
Set up evidence and retention
The team connects gateway logs and cost data to the storage and retention arrangements the compliance lead specifies, including access controls on the log store and a documented process for exporting records. Where the use falls under the record-keeping duties of the EU AI Act, the team confirms with counsel whether the retention period in Article 19(1) applies and how it interacts with other law. The output is a retention schedule, an evidence map linking each control to the log or report that proves it, and a tested export. The step is done when a sample audit request can be answered from stored records within the documented process. DeployOpen prepares these artifacts but does not issue audit opinions or certifications.
Cut over production traffic in controlled waves
Additional applications move to the gateway in groups, each with a rollback path that restores direct provider access if the gateway fails. Monitoring watches latency, error rates and spend against the budgets set earlier. The output is a cutover log for each wave and a confirmed rollback test. The step is done when all approved applications route through the gateway, direct provider access from application networks is blocked, and the monitoring dashboards are reviewed by the operations team.
Hand over runbooks and ownership
DeployOpen documents the architecture, configuration, key rotation, budget changes, guardrail updates and upgrade steps in runbooks the client's team can operate. Ownership of each area, including who approves new providers and who reviews spend and blocked requests, is written down and accepted. The output is the handover package and a training session record. The step is done when the client's operators perform a key rotation and a budget change from the runbook without assistance. Ongoing support, upgrades and testing can be covered by a separate retainer if the client wants them.
Systems the gateway must connect to
A self-hosted gateway is only as useful as its connections. Identity decides who may call which models, secrets management protects provider keys, observability and SIEM record what happened, and the applications and workflow tools are the traffic that makes the gateway worth running. Each connection needs an owner and a documented check before go-live.
Identity providers for staff and services
Staff sign in through your existing identity provider, and service accounts authenticate with their own credentials, so each caller maps to a team, a project and a budget. Decide early whether gateway keys are issued from identity groups or created by administrators by hand. Confirm with the gateway vendor which SSO protocols and group claims its admin and user interfaces support, and whether virtual keys can be scoped per team and per model. Test revocation: removing a user from a group should stop their gateway access without a manual key cleanup.
Secrets management for provider keys
Upstream model provider keys are the most valuable credentials in the stack. Store them in your secrets manager or vault, inject them at runtime, and rotate them on a schedule that your security team sets. Keep them out of configuration files, container images and gateway databases in plain text. Confirm how the gateway reads secrets, whether it can refresh them without a restart, and how a leaked key is revoked across all environments.
Observability stack for traces and evaluations
Tracing and evaluation tools record prompts, responses, latency and token use so teams can debug and measure quality. Langfuse is MIT licensed except its ee directories, so confirm which features your deployment needs sit outside those directories. Arize Phoenix is licensed under the Elastic License 2.0, which prohibits providing the software to third parties as a hosted or managed service, so a provider-run arrangement needs a licensing review. Decide what the gateway forwards to each tool and how long traces are kept.
SIEM and security alerting
Gateway logs for authentication failures, key creation, budget breaches, blocked prompts and unusual model usage belong in your security information and event management platform. Define which events are security-relevant, map them to detection rules, and set who receives alerts. Check that the gateway can send structured logs to your collector, that timestamps and caller identifiers are consistent with other sources, and that log forwarding failures are themselves alerted on. Logs that never reach the SIEM do not support incident response.
Workflow automation tools that call the gateway
Tools such as n8n and Node-RED often hold their own credentials and call the gateway on behalf of business processes. n8n is licensed under the Sustainable Use License, with files marked .ee requiring an n8n Enterprise License, while Node-RED is licensed under Apache-2.0, so the licensing review differs between them. Route these callers through gateway keys with their own budgets, and confirm which workflows can send personal data to a model, and who approves new flows that do.
Application callers and service configuration
Internal applications, chat front ends and retrieval assistants call the gateway through an OpenAI-format endpoint. Inventory each caller, its model list, its rate limits and its fallback behavior before cutover, and avoid letting applications hold provider keys directly. Confirm how the gateway handles timeouts, retries and failover between providers, and test that a caller cannot reach models outside its allowed list.
What buyers ask before hiring help for an AI gateway
How much does a self-hosted AI gateway implementation cost?
The cost depends on scope, not on a fixed package. The main components are the planning and design work, the deployment into your cloud account or data center, the configuration of routing, keys, budgets and guardrails, the logging integration, and the handover documentation. Infrastructure is a separate line that you pay your cloud or hardware provider for, and it recurs. Licensing for the core gateway is not the main variable, because LiteLLM is MIT licensed outside its enterprise directory and Apache APISIX is Apache-2.0, but you should confirm which features your chosen edition needs. Model provider usage is billed by the provider and usually dominates running costs. DeployOpen does not publish a price list. Ask for a quote built from a written scope that lists each component, what is one-time, what recurs, and who pays for each line.
How long does a gateway rollout take and what sets the pace?
The pace is set by the number of model providers, the identity and network integration, and how many teams must be onboarded and approved. A single-team pilot with one provider moves much faster than a multi-provider rollout that needs SSO, per-team budgets, guardrail policies and log retention agreed with security and legal. Approval cycles for cloud accounts, DNS, certificates and outbound network access often take longer than the technical work. Ask for a phased plan with a written scope and milestones, and ask which dependencies sit with your team and which sit with DeployOpen. No duration is promised here because it depends on those inputs.
What does the client team need to provide for a gateway project?
The work needs a few named people from your side, and their availability often determines the schedule. Typically this includes an owner for the cloud or data center environment, someone who manages identity and single sign-on, a security reviewer for key handling and logging, and a finance contact who sets budget rules per team. You will also need the list of model providers and their contract terms, the applications that will call the gateway, and a decision on where logs are stored and for how long. Confirm these roles in the written scope before work starts.
Can an internal team run a self-hosted AI gateway alone?
Yes, many internal teams can operate a gateway once it is deployed, but they need Kubernetes or container, database, secrets and observability skills, plus a process for upgrades. Gateway projects move quickly, so release tracking matters. LiteLLM published v1.105.1 on October 14, 2026, and Apache APISIX published 3.19.0 on September 28, 2026, so an operator should plan a regular review cycle. A common split is for the internal team to own day-to-day operations and budget changes while an implementation partner handles the initial build, hardening, documentation and upgrades under a retainer. Ask which runbooks, tests and upgrade steps are included at handover.
How are gateway keys and logs protected?
Protection rests on three controls: how provider API keys are stored, how application keys are issued and revoked, and who can read logs. Provider keys should sit in a managed secrets store inside your environment and never in application code. Application keys should be scoped to a team or workload, carry budgets, and be rotated on a schedule. Logs can contain prompts and personal data, so access should be role-based and the content should be masked where policy requires it. Confirm each control in the design review, and test it in the security assessment before go-live.
What happens to gateway logs when a model provider changes?
Logs should stay in your storage, independent of the provider, so a provider change does not erase your record. The gateway normalizes requests to an OpenAI-style format, which helps keep logging consistent across providers, but response fields, token counts and model names can still differ. Your log schema should record the provider, model version and routing decision for each call. Retention matters for compliance: Article 19(1) of the EU AI Act requires providers of high-risk AI systems to keep automatically generated logs for at least six months unless other law provides otherwise, and Article 12(1) requires the system to technically allow automatic event recording over its lifetime. Confirm with counsel whether your use case is in scope and which retention period applies.
Does a gateway by itself satisfy compliance requirements?
No. A gateway supplies controls and evidence, such as access records, budget enforcement and logged calls, but it does not make a system compliant. Auditors and certification bodies assess the whole system and its governance, and DeployOpen prepares systems, controls and evidence for those assessments without issuing audit reports or certifications. Ask your compliance lead which requirements apply, map each one to a gateway control or an operational process, and agree who owns the evidence. The implementation partner should document the mapping as part of the scope.
What to ask an implementation partner before signing
Start with the written scope, because it decides what the gateway is responsible for. Ask the partner to list every model provider, endpoint and model alias the gateway will route to, and who requests a new one. Ask how virtual keys are issued, rotated and revoked, which team owns each provider credential, and whether provider secrets stay in a secrets manager instead of configuration files. Then ask which guardrails are in scope, such as prompt filtering, PII masking or output checks, and whether each one is configured in the gateway or delegated to an external service. A scope that says only "AI gateway with guardrails" leaves those decisions open, and they will surface as change requests after go-live.
Ask who patches and upgrades the gateway, and at what cadence. The LiteLLM project publishes frequent releases (v1.105.1 on October 14, 2026 is the version supplied for this page), and Apache APISIX is on its own release track (3.19.0 on September 28, 2026). Confirm whether the partner tracks upstream security advisories, how long a critical fix takes to reach your environment, who tests upgrades against your routing and budget rules, and whether rollback is in the price of the support retainer or a separate item. Also ask how license terms are handled. LiteLLM is MIT licensed except its enterprise directory, so ask which features you would need from that directory, whether a commercial license is required for them, and who verifies that the deployed build contains only what the license allows.
Ask how logs are retained and exported before anything goes into production. The gateway's request logs may contain prompts, responses and user identifiers, so the partner should state which fields are stored, where, for how long and who can read them. Confirm that the export path to your SIEM or to an observability tool is part of the scope, and that deletion requests and retention limits can be enforced. For high-risk systems, Article 19(1) of the EU AI Act requires providers to keep automatically generated logs for at least six months unless other law says otherwise, so ask the partner to show how that period is configured. Finally, ask how guardrails and AI security are tested. Request a written test plan covering bypass attempts on routing rules, budget enforcement, key leakage and guardrail evasion, and confirm whether that testing is a fixed-scope engagement or part of a retainer.
How DeployOpen scopes, builds, hands over and supports this function
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For an AI gateway, that means agreeing in writing which model providers, keys, budgets, guardrails and log destinations are in scope before any configuration begins.
How we scope
- Inventory every application and model provider that will call the gateway, with the traffic and data classes each one handles.
- Agree in writing which gateway (for example LiteLLM or Apache APISIX), deployment topology and identity integration are included.
- Define key ownership, budget units, guardrail policies and log retention targets as explicit acceptance criteria for the fixed scope.
How we build
- Deploy the gateway in your environment with provider credentials held in your secret store, not in application code.
- Configure virtual keys, team budgets, routing and fallback rules, and guardrail hooks according to the agreed policy set.
- Test routing, spend limits, guardrail blocking and log completeness against the written acceptance criteria before handover.
How we hand over
- Document configuration, runbooks, key rotation steps, budget change procedures and the cost tracking reports your finance team will receive.
- Deliver the governance evidence set, including log schemas, retention settings and access records, mapped to the controls you need to show.
- Train the operators who will own the gateway day to day, and record which parts remain with your internal team.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer, covering gateway version updates and policy changes.
- Re-run routing, guardrail and logging tests after each upgrade or new provider is added to the gateway.
- Confirm with your team, before engagement, which support levels, response expectations and upgrade cadence the retainer will cover.
