AI gateway for routing between self-hosted and hosted models

A self-hosted gateway lets regulated teams keep sensitive prompts on in-house models and use hosted APIs only where policy allows, with a log of which model saw which data. This page covers the requirements, platforms and rollout.

Book a callTalk to us
On this page

How one gateway can split traffic between self-hosted and hosted models

Teams in your organization probably already call hosted LLM APIs directly, and some of those prompts now contain data that regulators, contracts or internal policy restrict. The usual pattern is to place one gateway, such as LiteLLM or Apache APISIX, in front of self-hosted vLLM servers and a short list of approved hosted providers, so applications call a single OpenAI-compatible endpoint and the gateway decides where each request goes. A generic API gateway can forward traffic, but this pattern needs three things it does not handle by default: classification of each request by data sensitivity, routing rules that keep regulated prompts on local models, and a per-call record of which model and which data path handled the request.

The sensitivity decision is the part most likely to fail in practice, so define it before choosing rules. Agree on a small set of labels (for example, public, internal and regulated) and decide who assigns them: the calling application through a header or virtual key scope, or a classifier or guardrail at the gateway that inspects content. Each label then maps to an allowed set of models. Requests labeled regulated should only match self-hosted routes, and the gateway should reject, rather than silently fall back to, a hosted provider. Failover rules need the same care, because a retry that moves a regulated prompt to an external API is a data transfer your records must show.

On the platform choice, Apache APISIX is a general API gateway with AI features, while LiteLLM is purpose-built for model calls, covering provider translation, virtual keys, budgets and model-level logging. Which one fits better depends on what your organization already runs and how much custom policy you need, and this should be confirmed in a proof of concept using your own labels, your own routes and a test vLLM endpoint. Ask the implementation partner to demonstrate that a regulated request cannot reach a hosted model under a forced failure, and to show the log entry that proves where it went.

What the regulations and controls require from the routing layer

This table maps each regulatory or security control to what it means for the gateway build, the evidence an assessor will ask for, and where the setup usually fails. Read it as a build checklist: each row names a configuration decision your team or partner must make and record before go-live.

RequirementWhat it means for the buildEvidence to keep
EU AI Act Article 12(1), automatic event recordingConfigure the gateway to log every model call, including model, route, key, team and timestamp, so high-risk system events are recorded automatically rather than sampledLogging configuration exports, sample log entries per model and a test showing a call to each route is recorded
EU AI Act Article 19(1), six-month log retentionSet log retention in the gateway database and downstream storage to at least six months unless other law requires longer, and confirm backups follow the same ruleDocumented retention settings, backup policy and a restore test showing older logs remain available
Ownership of gateway logsDecide who controls the log store, who can read prompts, and how deployers keep logs under their own control rather than relying on a model provider's retentionRACI for log ownership, access list for the PostgreSQL log store and provider contracts that state retention terms
Sensitivity-based routingTag each request or application with a data classification and map each class to an allowed route list, so regulated data never reaches an unapproved endpointClassification-to-route mapping table, test cases showing blocked and allowed routes, and change approval records
Regulated prompts on self-hosted vLLMPin routes for regulated classes to the in-house vLLM endpoint with no hosted fallback, and disable failover to external providers for those keysRoute configuration showing no external fallback for regulated keys, and a failover test that confirms the request fails closed
Blocking unapproved endpointsAllow outbound model calls only to approved provider hostnames at the network layer, and restrict which models each virtual key can callEgress firewall rules, model allowlist per key and a test attempt to reach an unlisted provider that fails
Secrets in a secret managerStore LITELLM_MASTER_KEY (which must start with sk-) and provider API keys in an approved secret manager, not in configuration files or container imagesSecret manager policy, rotation records and a scan of config files and images showing no plaintext keys
Ollama network exposureKeep Ollama bound to 127.0.0.1 on port 11434 unless a reverse proxy in front enforces authentication, since the local API does not require authenticationBind address configuration, firewall rule showing port 11434 is closed to other hosts, and proxy authentication settings
vLLM network exposureUse the --api-key option only as one layer, place a reverse proxy in front that allowlists user-facing endpoints, and isolate nodes in multi-node runsProxy allowlist config, network segmentation diagram and a check that /metrics and /health are not reachable from user networks

Which prompts, keys and logs cross the gateway and where they land

Before any routing rule is written, the team needs a complete inventory of what passes through the gateway and where each category is stored. For a hybrid setup, that inventory covers prompt and response bodies, file attachments sent with a request, user and team identities taken from the calling application or SSO session, virtual keys, spend records, and routing decisions showing which provider or self-hosted vLLM endpoint served each call. Routing decisions matter most here, because they are the record that answers which model saw which data. Map each category to a storage location and an owner, then check it against the data classes your sensitivity policy defines.

In a LiteLLM deployment, PostgreSQL is the primary supported database and holds the virtual keys and spend tracking that budgets depend on. Redis becomes necessary once more than one proxy instance runs, because shared rate limits, router state and cache must be consistent across instances. If you adopt Langfuse or Phoenix for traces, those tools hold prompt and output copies in their own storage, so they widen the data footprint. Langfuse is MIT licensed outside its ee directories. Phoenix uses the Elastic License 2.0, which prohibits providing the software to third parties as a hosted or managed service, so confirm with counsel that your intended use, including any service you offer to business units or customers, falls within its terms.

Before go-live, confirm in writing which fields you can disable or redact before anything is written to disk. Ask the implementation partner to test, in a non-production environment, whether request and response bodies, file content, and user identifiers can each be turned off or masked at the gateway, and whether tracing callbacks can be configured to skip sensitive routes entirely. Verify that regulated prompts routed to self-hosted vLLM are logged at the same fidelity as those routed to hosted APIs, and that the vLLM /metrics endpoint, which exposes operational counters, is not exposed to users. Document the result as your data flow record, since auditors and EU AI Act record-keeping reviews will ask for it.

Which gateway and model platforms fit a hybrid setup

LiteLLM is the option most often evaluated first for this pattern because it runs both as a Python SDK and as a proxy server that exposes an OpenAI-format endpoint to applications. Its core is MIT licensed, with the exception of the enterprise/ directory, which carries its own license. Routing, virtual keys and budgets sit in the open source tier, while several governance features sit behind an Enterprise license.

  • Routing and failover: The router supports simple-shuffle, usage-based, latency-based, least-busy and cost-based strategies, with retries, cooldowns and priority ordering for failover.
  • Virtual keys and budgets: Keys can be scoped per user, team or key, with budgets, TPM and RPM limits, and restriction to specific models. These require a PostgreSQL database.
  • Guardrail hooks: Guardrails can run as part of the request path, but team-level guardrails are part of the Enterprise scope, so confirm the tier you need.
  • SSO and SCIM: Admin UI SSO is free for up to 5 users from v1.76.0. Beyond that, SSO, SCIM, audit logs and organization and team RBAC require an Enterprise license.
  • Operations: Multiple proxy instances need Redis 7.0 or newer for shared rate limits and router state.

Confirm current license terms, the included feature set for your user count and which guardrail integrations work with your providers in a proof of concept before committing.

How the hybrid routing rollout runs in phases

The sequence moves from knowing which model calls and data classes exist, to signed-off routing rules, to a tested gateway that sends each request only where policy allows, with a fallback proven before cutover.

  1. Inventory current model calls and data classes

    Your team and DeployOpen list every application, script and vendor tool that calls a model, the provider and endpoint each uses, the teams that own them, and the data classes each flow may carry, such as public, internal, confidential and regulated. Owners confirm the list line by line. The output is a model call inventory with a data class assigned to each flow, and the step is done when no known caller remains unlisted.

  2. Sign off routing and data policy

    Security, privacy and legal review the inventory and agree written rules: which data classes may reach self-hosted models only, which hosted providers are approved and for what data, what must be blocked, and what happens when a provider fails. DeployOpen translates the agreed rules into a routing and classification matrix. The output is a signed policy document and that matrix, and the step closes when each approver has recorded their sign-off.

  3. Deploy the gateway, PostgreSQL and Redis

    DeployOpen installs the chosen gateway, such as LiteLLM, with PostgreSQL as the primary database and Redis for shared rate limits and router state once more than one proxy instance runs. Master keys go into your secret manager, admin SSO is configured, and the gateway runs in your cloud account or data center. The output is a working staging gateway with documented configuration, and the step is done when a test request completes through the single OpenAI-compatible endpoint.

  4. Onboard vLLM and approved hosted providers

    Self-hosted vLLM servers are registered behind the gateway and placed behind a reverse proxy that exposes only the endpoints users need, because the vLLM API key does not cover every endpoint. Each approved hosted provider is added with its own credentials and an allowlist of models per data class. Egress rules block every other destination. The output is a model registry and network policy, and the step closes when a request to a non-approved provider is refused in testing.

  5. Test routing, failover and budgets

    DeployOpen runs scripted tests against the staging gateway: classified requests land only on permitted models, failover moves traffic to the priority backup when a provider returns errors, cooldowns apply after repeated failures, and virtual keys stop spend or rate limits at the configured thresholds. Results are recorded against each policy rule. The output is a test report with pass or fail per rule, and fixes are retested until every rule passes.

  6. Confirm log retention and access review

    Logging is set up so each model call records the calling key, team, model, provider and data class, with prompts stored or redacted according to the signed policy. Retention periods are set with your compliance owner, log storage is placed in your infrastructure, and access to logs is limited to named roles and reviewed. Confirm any retention duty that applies to your sector and jurisdiction with counsel. The output is a retention schedule and access list, and the step closes when a sample request can be traced end to end in the logs.

  7. Cut over with a tested fallback

    Applications are moved to the gateway endpoint in stages, starting with lower-sensitivity traffic, while monitoring tracks errors, latency and spend against the test baseline. The previous direct connections are kept available as a documented fallback until the team has run the fallback path successfully. The output is a cutover record and a runbook for reverting, and the step is done when production traffic has run through the gateway with no unapproved calls in the logs.

Deployment patterns for the self-hosted and hosted tiers

A hybrid gateway is only as sound as the placement, isolation and shared state beneath it. The six patterns below cover where the gateway sits, how the self-hosted model tier is protected, what holds shared state, how requests are classified, where logs go, and how administrators sign in. Each one is a decision to make and confirm before the routing rules go live.

Gateway placement in the network

The gateway should sit in the organization's own network as the only path from applications to model endpoints, both self-hosted and hosted. Applications should not hold provider keys, and outbound traffic to hosted APIs should leave through a controlled egress point. Confirm which subnets can reach the gateway, which can reach the internet, and whether the gateway can be bypassed by a direct call to a provider from an application server.

Isolated vLLM tier with a reverse proxy

The vLLM tier must not be exposed directly to users. The vLLM --api-key option does not cover every endpoint, including /metrics, /health and /invocations, and the vLLM security docs recommend a reverse proxy that allowlists only the endpoints meant for users. Place the model servers on an isolated network reachable only from the gateway, and confirm the allowlist with a test that calls each unlisted path.

Multi-node network isolation

When one model spans several nodes with pipeline parallelism, traffic between those nodes is insecure by default, with no authorization or encryption. The vLLM security docs require that these nodes sit on an isolated network. Ask the platform team to document which nodes communicate, how that network is segmented from the user-facing tier, and how node-to-node traffic is protected in transit.

Redis and PostgreSQL for shared state

Once more than one gateway instance runs, rate limits, router state and cache must be shared. LiteLLM names PostgreSQL as the primary supported database and requires Redis 7.0 or newer for multiple proxy instances. Virtual keys, budgets and spend records also depend on PostgreSQL. Ask who backs up these databases, how failover works, and whether a Redis outage leaves the gateway enforcing limits or failing open.

Sensitivity classifier and policy source

The routing decision depends on a classification of each request by data sensitivity, and that classification is only as good as its policy source. Define who owns the sensitivity labels, where the rules live, and how a request with no label is handled. A common safe default is to route unlabeled traffic to the self-hosted tier. Test the classifier against sample prompts that contain regulated data before enabling hosted routing.

Logging sinks and retention

The gateway records which model received which request, so logs become regulated records in their own right. Decide where prompt and response logs are written, who can read them, and how long they are kept, aligned with the retention rules that apply to the data. Confirm that logs from self-hosted and hosted calls land in the same store, and that log access is itself audited.

Admin UI SSO integration

Administrators change routing, keys and budgets, so the admin UI needs strong sign-in and clear role separation. LiteLLM supports SSO with Google, Microsoft Entra ID, Okta and generic OIDC providers, but SSO beyond the free limit of 5 users requires an Enterprise license. Confirm the license scope with the vendor before the design is fixed, and map identity provider groups to administrator roles that match the team structure.

Evidence auditors and regulators ask for from the routing layer

This table lists each evidence item an assessor is likely to request from a hybrid routing layer, the component that produces it and what to confirm in your own environment. Read it as a checklist to agree with your implementation partner before go-live. DeployOpen prepares evidence and documentation, but independent auditors and certification bodies issue any opinion or attestation.

Evidence itemProduced byWhat the assessor expects
Model and provider inventoryGateway model list in the LiteLLM config, exported to your asset registerA current list of every self-hosted and hosted model, its provider, data classification it may receive and an owner
Routing and allowlist configuration historyVersion control for gateway config and router settings (priority ordering, failover, model restrictions per key)Dated change records showing who approved each routing rule and which models a key could reach at each point in time
Per-call logs with model and provider identifiersLiteLLM request logging, with entries stored in your PostgreSQL database or forwarded to your log platformRecords showing which model and provider handled each call, the calling key or team, timestamp and sensitivity class
Log retention settingsGateway logging configuration and database or log store retention policyRetention periods documented against the six-month minimum in Article 19(1) of the EU AI Act, or longer where other law applies
Admin and virtual key access logsAdmin UI audit trail and key management events (creation, budget change, model restriction changes)Evidence that privileged changes were authorized, attributed to a named person and reviewed on a defined cycle
Failover test recordsTest scripts and results for retries, cooldowns and priority failover between self-hosted vLLM and hosted endpointsDated tests showing which route traffic took when the primary failed and whether sensitive data stayed on approved routes
Guardrail and classification evidencePre-call classification rules and guardrail logs at the gatewayRule set versions, sample decisions and records of blocked or redirected requests
Confirm before go-liveYour environment and the vendor documentation for the features you useWhich LiteLLM features need an Enterprise license in your deployment, and where logs are retained and who can read them

Questions buyers ask before hiring help with hybrid model routing

What drives the cost of a self-hosted AI gateway with hybrid routing?

Cost comes from four places: the software, the infrastructure, the people who build and operate it, and the testing that proves it works. LiteLLM's core is MIT licensed outside its enterprise directory, but SSO beyond five users and audit logs require an Enterprise license, so confirm that line item early. Self-hosted vLLM has no paid edition, so its cost is GPU or accelerator capacity plus operations. Recurring costs include hardware, PostgreSQL, Redis and model maintenance. Ask any partner for a quote that separates one-time implementation, classifier tuning, testing and ongoing support, and that names each license it assumes.

How long does a hybrid model routing rollout take?

The timeline depends on scope, and no fixed duration applies to every organization. The main variables are how many models and providers are in scope, how many data sensitivity classes the policy defines, whether the identity provider integration is standard, and whether self-hosted GPU capacity already exists. Most engagements are fixed-scope, so the written quote should list the phases, deliverables and dates before work starts. Ask for the phase plan that places failover testing and logging validation before any regulated traffic is routed.

What does our team need to supply before implementation starts?

Three inputs unlock most of the design work. First, a model inventory listing every self-hosted and hosted model, its endpoint, its data location and its contract owner. Second, named policy owners from security, privacy and the business who can classify data sensitivity and approve which providers may receive each class. Third, access to your identity provider, such as Microsoft Entra ID, Okta or another OIDC provider, along with the groups that map to teams and budgets. Your team should also confirm where logs may be stored and for how long before configuration begins.

Can we build hybrid routing ourselves instead of hiring a partner?

Yes, if your team has the skills to operate the components. The open-source pieces are available: LiteLLM as the gateway, vLLM or Ollama for self-hosted models, and PostgreSQL and Redis for state. The harder parts are usually not the initial configuration but the sustained work: writing and tuning the sensitivity classifier, securing the vLLM endpoints behind a reverse proxy, running multi-node networking safely, validating failover, and producing evidence for auditors. Compare the internal effort against a partner scope in writing, and decide which parts you want to keep in-house.

How do we test failover between self-hosted and hosted models?

Test failover as a scripted set of failure scenarios, not a single manual check. A typical plan covers a self-hosted endpoint that stops responding, a hosted provider returning errors or rate limits, and a model that becomes slow under load. For each case, verify that LiteLLM retries, enters cooldown and moves to the next priority in the failover order, and that the logs record which model answered. Also confirm that failover never sends a request classified as regulated to a provider outside the approved list. Keep the test results as evidence and repeat them after each upgrade.

Can prompts be routed without being logged?

Partly, and the answer depends on the rules you set and the compliance regime you face. The gateway can be configured to limit what is stored, for example by logging metadata such as model, team, token counts and timestamps while excluding prompt and response bodies. However, some regimes expect records of system events. The EU AI Act, for example, requires high-risk systems to allow automatic event logging and requires providers to keep logs for at least six months unless other law provides otherwise. Confirm with your legal and compliance owners which fields must be kept, and ask the partner to show the exact logging configuration.

How do we keep regulated prompts on self-hosted vLLM securely?

Treat the vLLM server as an internal service and place a reverse proxy in front of it. The vLLM --api-key option protects only some paths, and the security documentation warns against relying on it alone, so the proxy should allowlist just the endpoints meant for users, such as /v1/chat/completions. Keep administrative and metrics paths on a restricted network. For multi-node deployments, the vLLM documentation states that inter-node communication is insecure by default, so the nodes need an isolated network. Ask the partner to show the allowlist and the network diagram as part of the handover.

Questions to put to an implementation partner before signing

Start with the classification method, because every routing guarantee depends on it. Ask who writes the sensitivity rules, who approves changes to them, and how a request is labeled before it reaches the router: by application tag, by key, by data source, or by a classifier that inspects the prompt. Ask how a misclassified request is caught, and whether the partner will keep the rule set in version control with a named owner from security or compliance who signs off on each change. A partner that answers only with the name of a classifier has not answered the question.

Next, ask how the partner proves that regulated prompts never reach a hosted endpoint. A credible answer names a negative test: synthetic regulated prompts sent through every application key, with the test asserting that the upstream call went to the self-hosted deployment and never to a hosted provider. Ask for the test plan, the cadence for rerunning it after each LiteLLM upgrade, and what happens to a failing test. Ask also how the router's failover order is locked so that a hosted provider cannot become a fallback for a regulated model group. For licensing, ask for a written list of which LiteLLM features the proposed design depends on and which of those sit in the enterprise/ directory under the license in enterprise/LICENSE. SSO for the admin UI beyond the free user limit, audit logs, team-level RBAC and team-level guardrails and logging are the usual candidates. Confirm the current terms with the vendor, since the free SSO limit is tied to a specific release. Ask how the partner will keep the design within the free tier or price the license into the quote.

Ask where prompt and response logs are stored, who can read them, how they are encrypted, and how long they are kept, with retention set by your legal and records team rather than the default. Ask whether logs for regulated traffic stay in your own PostgreSQL or object storage, and how the design meets the minimum retention for high-risk logs under Article 19 of the EU AI Act if that regime applies to you. Finally, ask how the vLLM tier is isolated. vLLM's security documentation states that node-to-node traffic is insecure by default, and that its --api-key option does not cover every endpoint, so the partner should explain the network segment, the reverse proxy allowlist and how /metrics and /health are kept off user-facing paths. Ask how the exit path is tested: which hosted model is swapped for the self-hosted one, how often the swap is rehearsed, and what evidence a rehearsal leaves behind.

How DeployOpen scopes, builds and supports this gateway

DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside the client's own infrastructure. For a gateway that sits between applications and both in-house and hosted models, we start from a written scope that names each model route, data classification rule and log store before any configuration begins.

How we scope

  • Inventory the applications, teams and model providers that will call the gateway, with the endpoints each one may use
  • Agree in writing which data classes route to self-hosted vLLM and which may reach hosted APIs, with named owners
  • Define the log schema, retention location and access roles for prompt and routing records before build starts

How we build

  • Deploy LiteLLM or Apache APISIX in your cloud account or data center, backed by PostgreSQL and Redis where more than one instance runs
  • Configure routing, cooldowns and failover priority, then test each rule with synthetic requests labeled by data class
  • Place vLLM and Ollama endpoints behind a reverse proxy that exposes only the paths meant for users, and verify that nothing else is reachable

How we hand over

  • Deliver runbooks for adding models, rotating master keys in a secret manager and changing classification rules
  • Document the model inventory, routing table, budget and key mapping, and the evidence each log field supports
  • Walk your platform, security and compliance staff through failover tests and log queries before handover is signed off

How we support

  • Offer ongoing support, upgrades and continuous testing as a retainer after handover
  • Re-test routing and guardrail rules after each gateway, vLLM or provider change that affects data flow
  • Prepare the system, controls and log evidence that your independent auditors will examine, without issuing any audit opinion ourselves

Related

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.
Book a callhello@deployopen.com