Why an AI gateway for SaaS products is a different build
An AI gateway for a SaaS product is a different build from an internal gateway because the traffic it carries belongs to your customers. Each tenant's keys, spend, rate limits and failover behavior affect a paying account, so the gateway has to meter and isolate per tenant from day one. A reasonable core for this is LiteLLM as the proxy, with PostgreSQL holding virtual keys and spend records and Redis holding shared rate limits, router state and cache once more than one proxy instance runs. The LiteLLM GitHub repository is MIT licensed outside its enterprise directory, and the production guide names PostgreSQL as the primary supported database and Redis 7.0 or newer for multi-instance setups.
Three things change compared with an internal-only gateway. First, tenant isolation: a virtual key per customer or per customer application, restricted to approved models, with per-key budgets and TPM and RPM limits, so one tenant's runaway integration cannot consume another's allowance. Second, metering: cost attribution must be recorded by tenant and by product feature, not just by team, and the spend data must be reliable enough to bill from or to show customers. Third, provider failover: an outage at one model provider now reaches customers directly, so retries, cooldowns and priority ordering must be tested against each provider you use, and you should confirm how session affinity behaves for multi-turn conversations during a switch.
Before you fix the design, confirm the licensing of the features you need. LiteLLM's SSO for the admin UI is free only up to a small number of users from v1.76.0, and SCIM, audit logs, organization and team RBAC, JWT authentication and some secret manager integrations sit in the Enterprise license. Ask the vendor which of these your admin team and tenant-facing controls depend on, and get the answer in writing, because the cost and the identity architecture both follow from it.
Requirements a SaaS gateway must meet and what they mean for the build
This table maps the requirements a software vendor typically faces when exposing model access to customers, the source that creates each requirement, and what it means for the gateway build. Read each row as a design input: the source tells you who will ask for the control, and the build column tells you what to configure or confirm. The main takeaway is that per-tenant keys, budgets and logs are contract and compliance questions as much as engineering ones.
| Requirement | Source | What it means for the build |
|---|---|---|
| Per-tenant virtual keys and budgets | Customer contracts and plan terms | Issue keys per tenant and per feature, backed by PostgreSQL, so spend can be capped and reported per customer. Confirm the key hierarchy before onboarding the first tenant. |
| TPM and RPM limits per tenant | Customer contracts and fair-use terms | Set token and request limits per tenant key so one customer cannot exhaust shared provider quota. Shared rate limits across instances depend on Redis being in place. |
| Model allowlists per plan | Plan tiers and customer contracts | Restrict each key to the models its plan includes. Keep the allowlist in configuration under change control so plan changes are auditable. |
| Failover and retries | Customer SLAs and contract availability terms | Define provider priority order, retry counts and cooldowns in the router. Any availability commitment must be confirmed against the providers' own terms, not assumed. |
| Prompt and response logging with retention | GDPR Article 28(3) processor terms and EU AI Act Article 12 where the product is high-risk | Log only what the documented instructions allow, set retention per tenant and enable redaction of personal data. Confirm with counsel which records the product must keep and for how long. |
| PII filtering before external calls | GDPR Article 28(3) and Article 32 security of processing | Apply input filters at the gateway so personal data reaches only providers listed in the processor terms. Test filters against representative tenant data before release. |
| Admin SSO and role-based access | Customer contracts and enterprise security questionnaires | Connect the admin UI to the vendor's identity provider. Confirm which SSO and SCIM features require a commercial license for the chosen gateway. |
| Horizontal scaling and portability | EU Data Act switching rules (Articles 23 and 25) if in scope, and customer exit clauses | Run several stateless proxy instances on Redis and PostgreSQL. Confirm with counsel whether the product is a data processing service under the Data Act, since that determines whether switching duties apply. |
Which open-source platforms fit a SaaS gateway and why
A SaaS gateway usually combines three jobs: calling model providers, enforcing per-tenant keys and budgets, and recording cost and traces by feature and customer. The platforms below cover different parts of that job, so the choice depends on what your team already runs and which features your license terms allow.
LiteLLM as the model gateway
LiteLLM works as both a Python SDK and a proxy server that calls more than 100 LLM providers in the OpenAI request format. Its virtual keys support per-key, per-user and per-team budgets, TPM and RPM limits, and restriction to specific models, which maps directly to per-customer access. The core code is MIT licensed, but the enterprise directory uses a separate license. Confirm which features need an Enterprise license before committing, including SSO beyond the free user limit, SCIM, audit logs and team-level guardrails. The current release is v1.105.1.
Apache APISIX for existing API estates
Apache APISIX is a cloud-native API gateway that also positions itself as an AI gateway, licensed under Apache-2.0. It fits when your company already runs APISIX for customer-facing APIs, because authentication, rate limiting and observability policies can be extended to model traffic under one operations model. Check whether the AI-specific plugins you need, such as provider routing, token-based limits and per-consumer budgets, are available in the open-source build or only in a commercial distribution. Latest release: 3.19.0.
Kong Gateway for existing Kong estates
Kong Gateway is Apache-2.0 licensed and suits SaaS teams that already manage their public APIs with Kong. Routing model calls through the same control plane keeps authentication, quotas and logging consistent across product APIs and AI endpoints. Before choosing it for model traffic, confirm which AI plugins, token accounting and provider integrations are included in the open-source release, and which sit behind Kong's commercial offerings. This is a fit decision for your existing platform rather than a reason to adopt Kong from scratch. Latest release: 3.9.4.
Langfuse for traces and cost analytics
Langfuse is an LLM observability platform that records traces, prompts, outputs and token costs, so you can attribute spend to a feature, a tenant or a release. The core is MIT licensed, but directories under ee/ and related paths carry an enterprise license. Self-hosting it means running Langfuse Web, Langfuse Worker, PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible storage. Confirm which features you need, such as project-level RBAC, data retention policies, audit logs and server-side masking, since several require an Enterprise license key. Latest release: v4.63.0.
Deployment pattern for scaling the gateway with Redis and PostgreSQL
A SaaS gateway that issues per-customer keys and budgets becomes a stateful service once it runs on more than one instance. The cards below cover the database, cache, worker, secret and spend-tracking choices that carry that state, plus the observability stack that records it. Every sizing decision should be tested against your own tenant load before it is treated as settled.
PostgreSQL holds virtual keys
LiteLLM names PostgreSQL as its primary supported database, and virtual keys cannot be issued without one. Per-customer keys, per-user and per-team budgets, TPM and RPM limits and model restrictions are all stored there. Confirm backup, point-in-time recovery and failover for this database before any tenant receives a key, because losing it removes the budget and access state for every customer at once.
Redis for shared state
Once more than one proxy instance runs, shared rate limits, router state and cache must live outside any single pod. LiteLLM's production guide requires Redis 7.0 or newer for this. Without it, a customer's rate limit can be enforced separately on each instance, so the effective limit grows with the replica count. Ask your partner to test that a tenant limit holds across all replicas during a load test.
One worker per pod, scaled out
LiteLLM recommends one Uvicorn worker per pod on Kubernetes, with horizontal scaling for capacity. The production guide suggests 1 vCPU and 4 GiB memory per worker as a starting point. Scale on measured latency and queue depth for your traffic mix, not on the starting figure. Large SaaS workloads with long streaming responses may need a different ratio, so confirm it under realistic concurrency.
Master key in a secret manager
The LITELLM_MASTER_KEY must start with sk- and grants administrative control over every virtual key. Store it in a secret manager rather than in manifests, images or environment files checked into source control. Rotation procedures, access logging on the secret store and restricted read rights for the platform team belong in the runbook. Enterprise deployments can integrate AWS, Azure, Google, HashiCorp Vault or CyberArk secret managers, so confirm which one your organization already operates.
Batched spend writes at scale
Writing spend to PostgreSQL on every model call creates write pressure that grows with tenant count. LiteLLM recommends batching spend updates to the database every 60 seconds at scale. The trade-off is that budget enforcement and cost attribution can lag by up to that interval, so a tenant may briefly exceed a budget. Decide whether that lag is acceptable for your contracts and whether hard caps need a faster check.
Langfuse self-hosting components
A self-hosted Langfuse deployment runs Langfuse Web, Langfuse Worker, PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible blob storage. Minimums in the scaling guide are 2 CPU and 4 GiB for web, worker and PostgreSQL, 1 CPU and 1.5 GiB for Redis or Valkey, and 2 CPU and 8 GiB for ClickHouse, with at least 16 GiB of ClickHouse memory for larger deployments. Treat these as floors. Ask your partner to size ClickHouse from your expected trace volume and retention period.
How per-customer keys, budgets and cost attribution map to tenants
The core design choice is how a tenant maps to LiteLLM virtual keys. A key per tenant is the simplest model: each customer gets one virtual key with its own budget, RPM and TPM limits and an allowlist of models, and the gateway enforces those limits on every call. A key per tenant and feature gives finer control, for example a cheaper model and a tighter budget for a summarization feature than for an agent workflow, but it multiplies the number of keys to provision, rotate and reconcile. Many SaaS teams start with per-tenant keys and add feature keys only where a feature needs its own limits. Map plans to budgets through a team or a shared policy, not by hand-editing each key, and confirm with the vendor how team and per-key budgets interact when both apply to the same call.
Cost attribution depends on what each call carries. The application should send tenant, feature and plan identifiers as metadata on every request, and the gateway writes spend records to PostgreSQL against the key that made the call. Temporary budget increases suit a tenant that needs headroom for a launch or an incident, and they should expire automatically so that a one-off exception does not become a permanent change to the plan. Verify that spend updates are batched as LiteLLM's production guidance describes, since a short delay between a call and its recorded cost affects any customer-facing usage meter. Reconcile gateway totals against provider invoices monthly, because the gateway records only what it routed.
Customers who bring their own provider keys need a separate rule. Their keys should be stored in a secret manager, referenced by the gateway rather than pasted into application configuration, and tied to that tenant's virtual key so that their usage is never billed to your shared budget. Decide whether the customer's traffic still passes through your gateway for logging, and document the answer, because it determines whether you hold prompt records for that tenant. Ask any implementation partner how they will test that a BYOK tenant cannot reach another tenant's key or model, and confirm which secret manager integrations your LiteLLM edition supports.
Rollout phases for a SaaS AI gateway and the output of each
The sequence moves from a scoped tenant model and provider list to a single test gateway, then to metered keys, shared state, identity, observability and tested isolation, before any customer traffic is routed through a feature flag. Each phase ends with a named artifact that the next phase depends on.
Scope the tenant model and provider list
The product team and the platform team agree how tenants, workspaces, features and environments map to LiteLLM teams, keys and budgets, and which model providers and self-hosted endpoints are approved for each feature. Your security and legal reviewers confirm where prompts and completions may go. The output is a signed-off tenant model document and a provider register that lists every endpoint, its data handling terms and its owner. The step is done when every feature in the product has an assigned route and an accountable owner.
Stand up a single-instance test gateway
The implementation partner deploys one LiteLLM proxy in your test environment, connects two or three providers and runs the OpenAI-format request path from a staging copy of one feature. Configuration lives in version control, and the master key is held in a secret manager from the first day. The output is a working test endpoint, a baseline configuration repository and a short record of request latency and error behavior. The step is done when a test request from the staging application succeeds through each configured provider.
Add PostgreSQL keys, budgets and metering tests
Virtual keys, per-tenant and per-team budgets, rate limits and model restrictions are created against a PostgreSQL database, which LiteLLM requires for virtual keys. The team then runs metering tests that replay known request volumes and compare recorded spend with the provider invoices for the same period. The output is a metering test report and a key issuance procedure that tells the product team how a new tenant receives a key. The step is done when recorded spend matches expected spend within the tolerance your finance team agrees, and a budget breach blocks further calls as configured.
Add Redis and a second instance with failover tests
Redis 7.0 or newer is added for shared rate limits, router state and cache, then a second proxy instance runs behind the load balancer. Engineers simulate provider errors, timeouts and instance loss, and confirm that retries, cooldowns and priority-ordered fallback behave as the routing configuration describes. The output is a failover test record with each failure scenario, the observed route and the recovery time you measured. The step is done when a failed provider or a stopped instance does not stop tenant traffic and does not double-count spend.
Wire SSO for the admin UI and secrets management
Administrator access to the LiteLLM admin UI is connected to your identity provider through one of the supported SSO options, and provider API keys move from configuration files into your secret manager. The team confirms which SSO and secret manager integrations sit in the free feature set and which need an enterprise license, using the current LiteLLM documentation and a written answer from the vendor. The output is an access matrix for administrators, a secret inventory with rotation owners and a licensing decision record. The step is done when no provider key appears in plain configuration and each administrator signs in through identity federation.
Add logging and tracing
Request logging, spend events and traces are sent to an observability store that you control, such as a self-hosted Langfuse deployment that uses PostgreSQL, ClickHouse, Redis or Valkey and blob storage. Retention periods, redaction rules for personal data and access roles for prompt content are set before any real traffic passes. The output is a logging specification, a retention schedule and a dashboard showing spend and errors per tenant and per feature. The step is done when a test call can be traced from the tenant key to the provider response with the agreed fields and nothing beyond them.
Run load and tenant-isolation tests
Load tests run at the request volumes your product forecasts, with the worker count and horizontal scaling settings the production guide describes, and a separate test confirms that one tenant cannot read another tenant's keys, spend, logs or model access. Results feed back into worker sizing, Redis capacity and database connection limits. The output is a load test report with the bottleneck identified and a tenant-isolation test report with each attempted cross-tenant access and its result. The step is done when the agreed load runs without breaching budgets or latency targets and every isolation test fails as intended.
Cut over traffic behind a feature flag
Each product feature reads the gateway endpoint through a feature flag, so traffic moves in stages from the existing direct provider calls to the gateway, starting with internal tenants and then selected customers. Rollback is a flag change that your operations team can perform without a code release. The output is a cutover runbook, a rollback procedure tested in staging and a daily report of traffic share, errors and spend during the transition. The step is done when all production traffic for the migrated features passes through the gateway and direct provider keys are retired from the application.
Evidence customers and auditors ask for, and the component that produces it
This table maps each evidence item a SaaS customer or auditor typically asks about to what it shows and the gateway component that produces it. Read it as a checklist for the design phase, since each row should have a named owner and a retention setting before go-live. The main takeaway is that the gateway produces most of the operational evidence, while the SOC 2 report itself comes from an independent auditor, not from DeployOpen or the software vendor.
| Evidence | What it shows | Component that produces it |
|---|---|---|
| Tenant key and budget configuration | Which tenant or feature holds which virtual key, with per-key and per-team budgets, rate limits and allowed models | LiteLLM virtual keys backed by PostgreSQL, exported as configuration snapshots |
| Spend records by tenant and feature | Model spend attributed to each tenant and product feature, supporting usage billing and cost reviews | LiteLLM spend tracking in PostgreSQL, reviewed alongside Langfuse cost traces |
| Routing and fallback configuration | Which providers a request may reach, the failover order and retry and cooldown settings in force on a given date | LiteLLM router configuration kept in version control and deployed through the change process |
| Prompt and response logs with retention settings | What data entered and left the gateway, how long it is kept and who can read it | LiteLLM logging callbacks and Langfuse storage, with retention policy and masking settings documented |
| Access records for the admin UI | Who signed in to the gateway admin console, when, and which roles they held | LiteLLM admin UI SSO and audit logs, which the enterprise license covers (confirm the scope with the vendor) |
| Change records for gateway upgrades | What version ran in each environment, when it changed, who approved it and how it was tested | Git history, CI/CD pipeline records and the upgrade runbook for LiteLLM, PostgreSQL and Redis |
| Security of the gateway itself (CC1 to CC9) | Evidence of control design and operation for the common criteria, mapped to the systems in scope | Control mapping maintained by the client, using the outputs above; DeployOpen prepares it, and the independent auditor issues the SOC 2 report |
What SaaS teams ask before building a self-hosted AI gateway
What drives the cost of a self-hosted AI gateway for a SaaS product?
Cost comes from people and infrastructure more than from the gateway software itself, which is open source. The main components are PostgreSQL for virtual keys and spend records, Redis for shared rate limits and router state once more than one proxy instance runs, compute for the gateway workers, and the engineering time to integrate, test and operate it. LiteLLM's production guide names PostgreSQL as the primary supported database and requires Redis 7.0 or newer once more than one instance runs. Enterprise licensing matters if you need SSO beyond the free limit or audit logs: the LiteLLM admin UI SSO is free for up to 5 users from v1.76.0, and an Enterprise license is needed beyond that. Hosted gateway plans are typically priced by logged requests, so ask vendors to model your request volume against self-hosting. Ask any implementation partner for a fixed-scope quote that separates one-time build work from recurring operations and upgrades.
How long does a self-hosted AI gateway rollout take for a SaaS team?
The pace is set by your tenant model, the number of model providers you must connect, and how much of the control plane your team already runs. No duration is fixed in advance. The work usually moves in stages: deploying the gateway with PostgreSQL and Redis, mapping virtual keys and budgets to tenants and features, configuring routing and failover, adding guardrails and logging, then load testing and cutover from direct provider calls. Each stage depends on decisions about billing units, provider contracts and log retention. DeployOpen plans and implements self-hosted platforms in the client's own infrastructure under a written, fixed scope, so the timeline is agreed in that scope before work starts rather than estimated in advance.
What does our team need to provide before a gateway project starts?
Three inputs decide whether the project moves quickly: a documented tenant model, the provider contracts and data terms you already hold, and enough site reliability capacity to run the service. The tenant model defines how a customer, workspace or feature maps to a virtual key, team or budget, which is the basis for cost attribution. Provider contracts determine which models each tenant may call and which data may be sent to them. On the operations side, you need people who can own PostgreSQL and Redis backups, secret handling for the master key, upgrades and incident response. Confirm these owners early, because a gateway that no one runs becomes a single point of failure for every customer's model calls.
Can a small engineering team build and run a self-hosted AI gateway alone?
A small team can deploy the software alone, but running it reliably for paying customers takes ongoing ownership. LiteLLM is a Python SDK and proxy server with documented production practices, including one Uvicorn worker per pod on Kubernetes, horizontal scaling, and batching spend updates to the database every 60 seconds at scale. The harder work is the surrounding design: failover rules, budget enforcement logic, guardrail placement, log retention and access controls. Decide which parts your team will build and own, which you will buy as a fixed-scope implementation, and which you will retain for ongoing support, upgrades and testing. DeployOpen offers implementation, hardening, backup, upgrade and support retainers for self-hosted platforms in your own infrastructure.
How should we handle customer data in gateway prompt logs?
Treat prompt logs as personal and confidential data, because a gateway that logs every call will capture whatever customers type. Define what is logged, such as metadata, token counts, model and tenant identifiers, versus full prompt and response content, and set retention per category. Restrict who can read logs, encrypt the PostgreSQL and any observability storage, and use guardrails to redact personal data before logs are written where your policy requires it. If customers are in the EU, GDPR Article 28(3) requires processor contracts to bind the processor to documented instructions and to Article 32 security measures, and your own customer contracts will need to describe which providers receive their data. Confirm each point with your legal and privacy team before enabling full prompt logging.
Can a SaaS product offer self-hosted models to some customers?
Yes, a gateway can route some tenants to self-hosted models and others to hosted APIs behind one OpenAI-compatible endpoint. Routing is controlled by rules per tenant, model allowlists and budgets, with failover between providers configured in priority order. Self-hosted serving is a separate workload: vLLM is licensed under Apache-2.0 and Ollama under MIT, but you must size GPUs or servers, manage model updates and capacity, and decide whether a dedicated deployment is offered per customer. Check the license terms of every model you serve, since open-weight model licenses vary. Ask any partner to show how a single tenant's requests reach a chosen model and how that routing is logged for audit.
Which LiteLLM features need an enterprise license for a SaaS deployment?
The open source LiteLLM core covers routing, virtual keys, budgets and logging, while several controls sit behind the enterprise license. According to LiteLLM's enterprise documentation, the Enterprise license adds SSO and SCIM beyond the free limit, JWT authentication with identity provider tokens, audit logs, organization and team RBAC, IP allowlisting, secret manager integrations and team-level guardrails and logging. LiteLLM's enterprise directory is under its own license, while the rest of the repository is MIT. Confirm the exact feature list against the current enterprise documentation and your own requirements before design, because licensing terms change between releases.
Questions to ask an implementation partner about SaaS gateway scale
Start with tenant key mapping: ask the partner how each customer tenant maps to virtual keys, teams and budgets in LiteLLM, and who can create, rotate or revoke a key. Ask how a tenant's budget is enforced when requests arrive through several gateway instances, and how that enforcement is tested. For failover, ask which provider the router falls back to during an outage, how retries and cooldowns are configured, and what a tenant sees when every provider in its priority list is unavailable. Ask for the test evidence, not a description of the design. Confirm which LiteLLM routing strategies and priority settings the partner will use, since the choice changes both cost and latency.
On data stores, ask how Redis and PostgreSQL are sized for your request volume, what the partner tested under load, and how they will recover if Redis is lost. LiteLLM's production guide names PostgreSQL as the primary supported database and requires Redis 7.0 or newer once more than one proxy instance runs, so ask whether the partner validated those minimums against your traffic. Ask which features need an enterprise license, including SSO beyond the free user limit, SCIM, audit logs, team-level guardrails and secret manager integrations. Confirm each one in writing against the current LiteLLM enterprise documentation, because the list determines your recurring license cost.
Ask how prompt logs are stored, where they sit, and how a single tenant's logs are deleted on request, including backups and any observability store such as Langfuse. Ask how upgrades are tested, given that LiteLLM supports the four most recent stable minor versions, so every upgrade must land within that window. Ask for the staging process, rollback steps and who approves a release. Finally, ask how incidents are handled: severity definitions, response expectations written into the contract, who holds on-call duty, and whether support is a separate retainer. Scope and response times should be confirmed in the written scope, not assumed.
How DeployOpen scopes and supports a SaaS AI gateway
This sequence takes a software company from an unscoped idea of model access to a written, tested and supported self-hosted gateway that runs in its own infrastructure.
Confirm the product and tenant requirements
DeployOpen works with your product, platform and security leads to define which features call models, how tenants are isolated, which providers are approved and where the gateway will run, whether in your cloud accounts, data centers or an air-gapped network. The output is a requirements list covering per-customer keys, budgets, failover expectations, attribution by feature and tenant, and logging needs. The step is done when your leads have signed off on that list as the basis for scope.
Agree a fixed-scope statement of work
DeployOpen turns the requirements into a written scope with deliverables, responsibilities, assumptions and timelines before any work starts. Most engagements are fixed-scope, and a quote follows from that written scope rather than from a published price list. Ask for the scope to name which open-source components are in it, such as LiteLLM with PostgreSQL and Redis, and which features that depend on an enterprise license. The step is done when both parties have signed the scope.
Design the gateway architecture and data flows
Engineers document the reference architecture: gateway instances, the PostgreSQL database that holds virtual keys and spend records, the Redis layer used for shared rate limits and router state once more than one instance runs, routing and failover rules, and where prompt logs are stored. The design also records which identity provider signs in administrators and which secret manager holds the master key. The step closes with an architecture document and data-flow diagram that your security team reviews.
Implement, harden and integrate with your product
DeployOpen deploys the gateway in your environment, configures per-tenant virtual keys and budgets, connects approved model providers and self-hosted endpoints, and integrates the gateway with your application and billing or metering systems. Hardening covers admin access, secret handling, network controls and backup of the database. The step closes when test traffic from each feature reaches the intended model and appears in the attribution reports.
Test the gateway and AI systems
DeployOpen runs security testing against the gateway, its APIs and the AI features it protects, including prompt injection, key misuse, tenant isolation and budget bypass scenarios, and can add red teaming for agent behavior where your product uses agents. Findings are logged with severity, owner and a retest plan agreed in the scope. The step is done when the agreed test set has been run and every finding is either fixed and retested or accepted in writing by your team.
Hand over documentation and operating runbooks
DeployOpen documents the installed system, including configuration, routing rules, failover procedures, backup and restore steps, upgrade procedures and the evidence your compliance team can use in an independent assessment. DeployOpen prepares that evidence but does not issue audit reports or certifications. Your operators are trained on the runbooks. The step closes with a documented handover that your platform team accepts.
Provide ongoing support and continuous testing
After handover, support, upgrades and continuous testing can run under a retainer. Upgrades to LiteLLM and its database dependencies are planned and tested in a non-production environment before they reach production. Retainer terms, response expectations and the testing cadence are agreed in writing. Ask for the retainer scope to state which upgrade and testing activities it includes, so you know the operating model is covered after go-live.
