Why keep the OpenAI API behind a gateway you run
If you are staying on the OpenAI API for now but need control over keys, spend and logging, a self-hosted LiteLLM gateway in your own infrastructure gives you that control, and it turns a later move to other models into a routing change instead of an application rewrite. Applications keep calling one OpenAI-compatible endpoint, while the gateway holds the OpenAI credentials, issues virtual keys, enforces per-key, per-user and per-team budgets and TPM or RPM limits, tracks spend and writes logs. Each of these controls sits with your team, not inside a vendor's console.
This is a staging step, not a replacement. OpenAI remains the model provider behind the gateway, and nothing about the hosted API changes for your users on day one. The gateway itself is MIT licensed everywhere outside the enterprise directory, so the core proxy can be run, reviewed and modified without a commercial contract. Features such as SSO beyond the free user limit, audit logs and SCIM sit in the separately licensed Enterprise code, so confirm with the LiteLLM project which of those controls your governance requirements actually need before you design around them.
The exit path depends on the gateway's routing layer. LiteLLM supports fallbacks, priority ordering and several routing strategies, so a second deployment can be added as a fallback first and promoted to primary later. Decide early which teams own the gateway configuration, where its PostgreSQL database and Redis instance run, and how the LITELLM_MASTER_KEY is stored, since those choices determine how much work a future model change will take. Check the current release notes in the LiteLLM releases page before you pin a version.
What the gateway layer must own
A LiteLLM gateway only pays off if it owns the controls that would otherwise live in each application: who may call the model, how much they can spend, how traffic is shaped, and where the records go. Each card below is one duty, the LiteLLM capability that covers it, and the infrastructure it depends on.
Per-team keys and budgets
Virtual keys let you issue a separate key to each application, user or team, with its own budget and a list of models it may call. Keys are only useful if each one maps to an owner you can name, so decide the ownership model before rollout. Virtual keys need a PostgreSQL database, so budget that database as a production dependency from day one. Confirm how budget increases are approved and logged, because temporary increases are a supported feature and will be requested.
Rate limits by TPM and RPM
Per-key limits on tokens per minute (TPM) and requests per minute (RPM) protect the shared OpenAI quota and stop one runaway job from starving other teams. Set limits per key, user or team so a single integration cannot consume the whole account. When you run more than one proxy instance, limits are shared through Redis, so a limit that works in a single-pod test can behave differently in production. Test limits under realistic concurrency before you rely on them.
Spend tracking and reporting
The gateway records spend per key, user and team, which gives finance and platform owners one place to answer who is consuming tokens and on which model. Spend updates are written to the database, and the production guide recommends batching those writes every 60 seconds at scale, so expect spend figures to lag slightly in dashboards. Reconcile gateway totals against the OpenAI usage invoice each month to catch missing or duplicated calls.
Logging and telemetry
Logs are the record of which prompts left the network and which model answered, so decide what is stored, for how long and who can read it before enabling callbacks. LiteLLM can send logs to Langfuse, Datadog, OpenTelemetry and Prometheus. Confirm that prompt bodies, which may contain personal data, are masked or excluded where your policy requires it, and that the logging backend has its own access controls.
Fallbacks and routing
The router chooses a deployment for each request and can retry, cool down failing endpoints and fail over by priority. Fallbacks are what make a later move to self-hosted models a routing change: an OpenAI deployment and a vLLM deployment can share one model group. Choose the routing strategy deliberately, since simple-shuffle is the recommended default and latency or cost-based routing change which backend gets traffic. Test each fallback path, because a fallback to a model with different output behavior is still a change in production.
Master key and secrets handling
The LITELLM_MASTER_KEY must begin with sk- and grants administrative control of the gateway, so it belongs in a secret manager, not in a configuration file or container image. Provider API keys for OpenAI and any self-hosted endpoints should also live in the secret store. Enterprise secret manager integrations include AWS, Azure, Google, HashiCorp Vault and CyberArk. Confirm who holds the master key, how it rotates and whether the admin UI is reachable only from your internal network.
Which LiteLLM features and companion tools fit, and at what trade-off
This table compares the LiteLLM features and logging destinations you may need for a gateway in front of the OpenAI API, with the license and cost position of each. Read the Enterprise column as the set of capabilities that sit behind a separate license, and check each row against your own identity, audit and licensing requirements before you commit to a design. The main takeaway is that the open-source gateway covers keys, budgets, routing and logging, while identity federation, audit trails and RBAC are where an Enterprise license usually enters the plan.
| Capability | LiteLLM open source (MIT) | LiteLLM Enterprise or companion tool |
|---|---|---|
| Virtual keys, budgets and rate limits | Per-key, per-user and per-team budgets, TPM and RPM limits, requires PostgreSQL | Included in the open-source gateway; confirm which admin workflows your team needs before assuming parity |
| SSO for the admin UI | Free for up to 5 users from v1.76.0 onward, per LiteLLM's SSO documentation | Required beyond 5 users, with Google, Microsoft Entra ID, Okta and generic OIDC providers supported |
| SCIM provisioning | Not in the open-source feature set | Enterprise adds SCIM beyond the free SSO limit; confirm with LiteLLM how it maps to your identity provider |
| Audit logs and organization or team RBAC | Not in the open-source feature set; build audit evidence from logging destinations | Enterprise adds audit logs and organization and team RBAC; ask for the audit log retention and export options in writing |
| Logging to Langfuse | Supported; Langfuse core is MIT licensed, with enterprise features under a separate license key | Use the self-hosted Langfuse deployment, which needs PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible storage |
| Logging to Opik | Supported as a destination; Opik is Apache-2.0 and can be self-hosted without a commercial license | Confirm the logging integration version against the LiteLLM release you run |
| Logging to OpenTelemetry, Prometheus or Datadog | Supported as logging destinations, per LiteLLM's documentation | Datadog is a commercial service; OpenTelemetry and Prometheus keep telemetry inside your own stack |
| License scope of the code | Everything outside the enterprise directory is MIT licensed | Content under the enterprise directory uses the separate license in enterprise/LICENSE; have counsel review it before deployment |
How a request flows through the gateway to OpenAI
Applications send every LLM call to one OpenAI-compatible endpoint on the LiteLLM proxy, and never to api.openai.com directly. The request carries a virtual key rather than the OpenAI key itself. Before anything is forwarded, the proxy checks that the key is allowed to call the requested model, that the key, user or team budget has not been exhausted, and that the key's requests-per-minute and tokens-per-minute limits still have headroom. LiteLLM's virtual keys need a PostgreSQL database for this state, so the budget and limit decisions are made in one place for every application behind the gateway. Illustrative example: a support bot's key might be restricted to a single chat model with a monthly budget, so a runaway loop is stopped at the gateway instead of on the OpenAI invoice.
After the checks pass, the router selects a deployment. Its retries and cooldowns handle transient failures: a deployment that fails repeatedly is taken out of rotation for a period, and priority ordering decides which deployment is tried next. Your team should confirm which routing strategy fits its traffic (the routing docs list simple-shuffle, usage-based, latency-based, least-busy and cost-based options) and test that a forced OpenAI error actually triggers the expected fallback. Keep the configured model names and fallback order in version control, so a change to routing is reviewed like any other code change.
The same endpoint is what makes the later move cheap. A self-hosted vLLM server started with vllm serve exposes OpenAI-compatible routes such as /v1/chat/completions, so it can be added to the gateway as another deployment and placed first or second in priority order. Applications keep their base URL and model alias, and only the gateway configuration changes. Before ranking vLLM above OpenAI, confirm that the vLLM endpoint sits behind a reverse proxy that exposes only the routes users need, since the vLLM security documentation warns that its --api-key option does not protect every endpoint.
How to deploy the gateway under your controls
This sequence takes LiteLLM from a licensing check to a running gateway that holds your keys, budgets and logs, so applications keep one endpoint while you decide where requests go. Each step ends with a check you can verify before moving on.
Confirm licensing scope and required features
Your security or legal lead confirms which parts of the LiteLLM code you will run. Everything outside the enterprise directory is MIT licensed, while the enterprise directory uses its own license. Map each control you need (SSO, audit logs, SCIM, team RBAC, secret manager integrations) to either the open-source gateway or LiteLLM Enterprise, and ask the vendor to confirm current SSO terms for your user count. The step is done when you have a written list of features and the license that covers each one.
Provision PostgreSQL and Redis 7.0 or newer
Your platform team provisions PostgreSQL as the primary database, since virtual keys, budgets and spend tracking depend on it. Add Redis 7.0 or newer as soon as you plan more than one proxy instance, because rate limits, router state and cache must be shared across instances. Place both in your own cloud account or data center, restrict network access to the gateway, and confirm backups are configured. The step is done when both services accept connections from a test pod and their versions are recorded.
Deploy the proxy with one worker per pod
Your platform team deploys the LiteLLM proxy on Kubernetes using one Uvicorn worker per pod, then scales out by adding replicas rather than workers. LiteLLM suggests about 1 vCPU and 4 GiB of memory per worker as a starting point, which you should validate under your own load. Configure the OpenAI key and any future provider keys as secrets, not plain configuration. The step is done when the health endpoint responds on every replica and a test chat completion returns through the gateway.
Store the master key in a secret manager
Generate LITELLM_MASTER_KEY, which must start with sk-, and store it in your enterprise secret manager rather than in manifests or source control. Grant read access only to the gateway service account and the few administrators who issue keys. Rotate the value through your normal secret rotation process and confirm that pods pick up the new value after a restart. The step is done when a search of your repositories and cluster manifests finds no copy of the key in plain text.
Issue virtual keys with budgets and model limits
Create a virtual key for each team or application, not one shared key. Set budgets per key, user or team, TPM and RPM limits where needed, and restrict each key to the models it may call. Use temporary budget increases only through a documented approval step. Spend updates are batched to the database every 60 seconds at scale, so expect spend figures to lag slightly behind live traffic. The step is done when a test key is rejected for a model outside its list and blocked once its budget is exhausted.
Enable logging to your observability backend
Connect the gateway's logging to the backend your team already runs, such as Langfuse, OpenTelemetry, Prometheus or Datadog. Decide which request fields are stored, and confirm whether prompts and completions are written to the backend and who can read them. Note that some advanced masking, retention and audit features in Langfuse need a separate enterprise license key. The step is done when a test request appears in the chosen backend with the expected key and team attributes.
Configure fallbacks and routing rules
Define the model group that applications call, list OpenAI as the first deployment, and add a self-hosted open-weight deployment behind it as a fallback or later primary. The router offers simple-shuffle, usage-based, latency-based, least-busy and cost-based strategies, with retries, cooldowns and priority ordering for failover. Choose a strategy that matches your traffic and test it under realistic load. The step is done when a forced failure of the primary deployment routes traffic to the secondary and the switch appears in the logs.
Validate the cutover with applications
Point one non-critical application at the gateway endpoint and run its regression tests, including output checks against prompts it relies on. Confirm that budgets, model restrictions and logs behave as designed under that application's keys. Then move remaining applications one at a time, keeping the previous direct configuration available for rollback. The step is done when each application passes its tests through the gateway and your team has a written rollback procedure that someone outside the project has rehearsed.
Questions teams ask before running a LiteLLM gateway
What does running a self-hosted LiteLLM gateway cost?
The gateway software is MIT licensed, so the cost is the infrastructure, the database and cache, and the people who run it, while your OpenAI spend stays the same because the gateway forwards requests to OpenAI. The main recurring costs are compute for the proxy workers, a PostgreSQL database for virtual keys and spend records, and Redis once more than one proxy instance runs. LiteLLM's production guide suggests 1 vCPU and 4 GiB memory per worker as a starting point, and that sizing drives the bill. Ask any quote to separate one-time setup from monthly operating costs, and to state who pays for cloud accounts, backups and on-call coverage.
Does the OpenAI bill go down after we add a gateway?
Not by itself. A LiteLLM gateway controls how spend is attributed and limited, through virtual keys, per-team budgets and rate limits, but the tokens you send to OpenAI are still billed at OpenAI's prices. Savings appear only when you route some traffic to a cheaper model, either a hosted provider or an open-weight model you serve yourself on vLLM. Ask for the routing rules and fallback order in the quote so you can see which requests would move and what that would change.
How long does a LiteLLM gateway deployment take?
Duration depends on how many applications call the API, how many teams need their own keys and budgets, and how much of the surrounding platform you already run. A deployment usually covers the gateway itself, the PostgreSQL and Redis dependencies, identity integration for the admin UI, logging to your observability tools, and cutover of each application's base URL. DeployOpen's engagements are fixed-scope, so the timeline is agreed in writing before work starts and tied to a defined list of deliverables.
What does a fixed-scope engagement include?
A fixed-scope engagement sets out the systems to be planned, implemented, integrated, hardened, backed up and documented, along with the acceptance criteria and the timeline, all in writing before work begins. For a gateway, that typically means the deployment in your own cloud account or data center, virtual key and budget structure, router configuration with fallbacks, logging integration, secrets handling for the master key, and runbooks. Ongoing upgrades, monitoring and continuous testing can be added as a retainer after go-live.
What do we need to provide before the work starts?
You need a cloud or data center environment where the gateway, PostgreSQL and Redis can run, network paths from applications to the gateway and from the gateway to OpenAI, and a secret manager for keys. You also need an inventory of applications and the teams that own them, the budget and rate limits each team expects, and an identity provider if administrators will sign in with SSO. Name a technical owner on your side who can make decisions during the engagement. Confirm these items in the quote so the scope reflects what your environment already provides.
Can our own engineers run the gateway without outside help?
Yes, if you have people comfortable operating containers or Kubernetes, PostgreSQL, Redis and secret management, and you can staff on-call for a service that sits in the path of every model request. The LiteLLM documentation covers production practices such as running one Uvicorn worker per pod and scaling horizontally, keeping the master key in a secret manager, and batching spend updates at scale. The harder part is ongoing operation: upgrades within the four most recent minor versions, incident response and key rotation. Ask your team to estimate that operating load honestly before deciding.
Do we need an enterprise license for SSO beyond five users?
Yes, according to the supplied LiteLLM documentation. From v1.76.0, SSO for the admin UI is free for up to 5 users with providers including Google, Microsoft Entra ID, Okta and generic OIDC, and an Enterprise license is needed beyond that. Enterprise also adds SCIM, JWT authentication, audit logs, team RBAC and secret manager integrations. Confirm the current terms with LiteLLM before you commit, since licensing details can change between releases.
Evidence to keep for your own security and compliance reviews
A self-hosted LiteLLM gateway becomes the single place where OpenAI access is issued, limited and recorded, so its configuration and logs are the evidence your reviewers will ask for. The four records below connect key handling, spend controls, request logs and deployment settings into one reviewable trail. DeployOpen prepares this evidence, but auditors and certification bodies are the ones who issue any assurance.
Key issuance and rotation records
Keep a dated record of every virtual key the gateway issues, who owns it, which models it may call and when it was rotated or revoked. Virtual keys need a PostgreSQL database, so the key table and its audit history are the primary source for this record. Check that the master key is held in a secret manager, that it starts with sk-, and that no application still uses the upstream OpenAI key directly, since a direct key bypasses every control in this record.
Budget and rate limit configuration exports
Export the per-key, per-user and per-team budgets, TPM and RPM limits, and model restrictions as they stand at each review date. Record every temporary budget increase with its approver and expiry, because a temporary increase that is never reverted is a common gap in reviews. Compare the export against your cost approval process so that the reviewer can see that the limits in force match what the business approved.
Spend and request logs routed to your logging backend
Route gateway logs to your own backend, such as Langfuse, Datadog, OpenTelemetry or Prometheus, and keep retention settings written down with the owner of each store. Confirm what each log captures, especially whether prompt and response bodies are logged, and whether personal data is masked before storage. Test that you can retrieve one request end to end, from the application call through the virtual key to the upstream OpenAI model and its spend entry.
Deployment configuration and secret handling record
Version-control the gateway configuration, the Kubernetes or container manifests, and the routing and fallback rules, and record each change with its reviewer. Document how database credentials, the master key and upstream provider keys reach the pods, and who can read them. LiteLLM's production guidance runs one Uvicorn worker per pod and scales horizontally, so keep the replica count and Redis configuration in the same record, since shared rate limits depend on them.
What changes when you later move to open-weight models
The application keeps its OpenAI request format, so the change happens in the gateway. A later move adds a vllm model entry in LiteLLM that points at a vLLM server through its OpenAI-compatible /v1/chat/completions endpoint, then changes the routing for chosen model names. Application code, SDK calls and virtual keys stay as they are. The work that does not disappear is everything behind that endpoint: output validation, GPU capacity and running vLLM as a service. Open-weight models can return different wording, tool-call formats and refusal behavior than the hosted models, so prompts and output parsers need regression tests before any traffic is routed to them. Confirm with your vLLM operators which model versions, context lengths and throughput targets they plan to support, since these decide how much capacity you must buy or rent.
Operating vLLM is a separate security task. Its --api-key option protects only the endpoints under /v1, /v2, /inference and /cohere. It does not protect /invocations, /pause, /resume, /abort_requests, /metrics or /health, and the vLLM security documentation warns against relying on that option alone. Place a reverse proxy in front of the server that allowlists only the endpoints your users need, keep the gateway as the only caller, and keep multi-node traffic on an isolated network, because node-to-node communication is insecure by default. Ask your team who owns that proxy configuration and who reviews changes to it.
Plan the operations the gateway does not cover. vLLM exposes Prometheus metrics at /metrics, which you should scrape and alert on, and the gateway's own logging must cover both providers so cost and latency stay comparable. Assign an on-call owner for GPU nodes, model loading and upgrades, since vLLM ships minor releases frequently. Choosing when and how traffic moves across providers is covered in the migration guide for teams that have decided to move.
How DeployOpen deploys a LiteLLM gateway in your environment
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For a LiteLLM gateway in front of the OpenAI API, we scope the keys, budgets, routing and logging your applications need, then build and hand the gateway over so your teams can operate it.
How we scope
- We agree the applications, teams and virtual keys that will route through the gateway before any build work starts.
- The written scope lists which OpenAI models, budgets, rate limits and fallback targets are in and out of the engagement.
- We confirm where PostgreSQL, Redis and logging will run, and which identity provider will sign administrators in.
How we build
- We deploy the LiteLLM proxy in your environment with PostgreSQL for virtual keys and a shared Redis once more than one instance runs.
- We store the master key in your secret manager and configure per-team budgets, TPM and RPM limits, and model restrictions.
- We test fallback and retry behavior against a failed upstream, then document the routing config so a later move to self-hosted models is a configuration change.
How we hand over
- You receive the deployment code, configuration files, runbooks for key issuance and budget changes, and a record of the tests we ran.
- We walk your platform and security staff through logging destinations, upgrade steps and how to add a new model route.
- The handover covers what your team must own after go-live, including upstream API key rotation and gateway patching.
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- Before you sign, ask us to confirm which LiteLLM features your license tier includes, since SSO beyond the free user limit and audit logs need an Enterprise license.
