Open-source alternative to Kong AI Gateway: the short answer
If you pay for Kong's AI gateway features and face renewal or data residency pressure, the open-source AI Proxy plugin in Kong Gateway covers basic multi-provider proxying, while LiteLLM is the stronger self-hosted option when budgets, virtual keys and team-level controls matter most. The AI Proxy plugin supports a broad provider list, including OpenAI, Azure OpenAI, Amazon Bedrock, Anthropic, Gemini, Ollama and vLLM, and it needs Kong Gateway 3.6 or later, with some features requiring 3.10 or 3.11. LiteLLM, which can run as a self-hosted proxy in your own infrastructure, adds per-key, per-user and per-team budgets and TPM and RPM limits through virtual keys, but it needs PostgreSQL for those keys.
Two things stay commercial in Kong's stack. AI Proxy Advanced, with its semantic, lowest-latency and priority load balancing across models, is documented only in the AI Gateway Enterprise offering, and fully self-hosted gateways with a local control plane sit in Kong Gateway Enterprise. On the LiteLLM side, SSO beyond five users, SCIM, JWT authentication, audit logs and organization RBAC require a LiteLLM Enterprise license. Confirm the exact feature set you need against each vendor's current documentation before you decide.
Check the licenses first. The Kong Gateway repository is Apache-2.0, and LiteLLM's core is MIT with an enterprise/ directory under its own license. Treat the Kong AI Gateway decision and the Konnect decision as separate: you can drop Konnect while keeping Kong's AI features, keep Konnect while moving AI traffic to LiteLLM, or change both. Start by listing which AI features your teams use today, then map each one to the open-source option or the commercial tier that covers it.
Which Kong AI Gateway capabilities open source covers
This table maps the AI gateway capabilities teams typically buy from Kong to the open-source Kong AI Proxy plugin, to LiteLLM and to the remaining gaps. Read each row against your own Kong version and LiteLLM release, since plugin behavior and licensing change between versions and should be confirmed there. The main takeaway is that the open-source plugin covers provider proxying and basic routing well, while advanced load balancing, SSO and governance usually need a commercial tier or LiteLLM's own feature set.
| Capability | Kong AI Proxy (open source) | LiteLLM (MIT core) | Commercial or gap to confirm |
|---|---|---|---|
| Provider proxying (OpenAI, Azure OpenAI, Bedrock, Anthropic, Gemini, Vertex AI, Ollama, vLLM) | Supported through the AI Proxy plugin; needs Kong Gateway 3.6 or later | Calls more than 100 providers using the OpenAI request format | Confirm your provider list and minimum Kong version before committing |
| Request types (chat, embeddings, files, batches, audio, image, video) | Supported request types listed in the AI Proxy plugin documentation | Covers common chat and embedding calls; confirm coverage of newer request types | Some features need Kong Gateway 3.10 or 3.11; verify in your version |
| Basic load balancing across models | Available through the open-source plugin's configuration | Simple-shuffle, usage-based, latency-based, least-busy and cost-based routing | Round-robin, consistent-hashing, least-connections, lowest-latency, lowest-usage and priority balancing in AI Proxy Advanced need Kong AI Gateway Enterprise |
| Semantic routing (prompt similarity to model) | Not covered by the open-source AI Proxy plugin | Not a documented LiteLLM router strategy; confirm with the vendor | Semantic balancing in AI Proxy Advanced needs Kong AI Gateway Enterprise |
| Failover, retries and cooldowns | Confirm failover behavior for your provider set in your Kong version | Retries, cooldowns after repeated failures and priority ordering for failover | Confirm whether your Kong setup needs Enterprise features for advanced failover |
| Authentication and API keys | Kong authentication plugins (key auth, JWT, OAuth, ACL) protect routes | Virtual keys stored in PostgreSQL with per-key, per-user and per-team controls | Confirm how key lifecycle is managed in each option |
| Budgets, TPM and RPM rate limits | Rate limiting plugins apply at the gateway; per-model token budgets need confirmation | Per-key, per-user and per-team budgets, TPM and RPM limits and temporary increases | Confirm token-level budget enforcement in your Kong version |
| SSO for admin access | Not a core feature of the open-source gateway; use your identity provider in front of the Admin API | Google, Microsoft Entra ID, Okta and generic OIDC; free for up to 5 users from v1.76.0 | SSO beyond the free limit and SCIM need LiteLLM Enterprise |
| Audit logs and RBAC | Confirm logging options and any audit trail in your deployment | Organization and team RBAC; audit logs are an Enterprise feature | LiteLLM Enterprise adds audit logs and organization and team RBAC |
| Guardrails and secret manager integration | Confirm available guardrail plugins in your version | Team-level guardrails and secret manager integrations are Enterprise features | Plan for guardrail and secret integration effort in either option |
What the open-source AI Proxy plugin already does
The AI Proxy plugin in Kong Gateway covers the basic gateway job for LLM traffic: it accepts requests, routes them to a model provider and returns the response. It needs Kong Gateway 3.6 or later, and some features need 3.10 or 3.11, so check the minimum version against each feature you plan to use before you pin a release. The plugin supports providers including OpenAI, Azure OpenAI, Amazon Bedrock, Anthropic, Gemini, Vertex AI, Cohere, Mistral, Hugging Face, Ollama and vLLM. It handles chat completions, completions, embeddings, files, batches, assistants, responses, audio, image and video generation request types. Confirm in the plugin documentation which request types and providers work on your target version, since support differs by feature.
The plugin runs in the Apache-2.0 Kong Gateway 3.9 open-source line, so you do not need a commercial license to run it. Two operational facts matter for planning. Kong's version support policy covers only Kong Gateway Enterprise, so the open-source edition has no published support window, and your team owns patching, upgrades and security response for the gateway and its plugins. Ask your Kong contact or check the release notes for any open-source fix you need before relying on it, and set an internal review cadence for CVEs in the 3.9 line.
This is the strongest fit for teams already running Kong Gateway OSS for their APIs. The AI traffic then uses the same declarative configuration, the same decK workflow and the same operating model as the rest of the gateway estate, with no second control plane to learn. The trade-off is that advanced load balancing and governance sit outside the OSS plugin, so an existing open-source deployment covers routing to providers but not the team-level controls described elsewhere on this page. Review your current Kong deployment mode and monitoring before adding AI routes, because each added provider adds keys to manage and a new upstream to observe.
When Kong AI Proxy or LiteLLM fits better
If your API traffic already passes through Kong Gateway OSS, the AI Proxy plugin lets you route LLM requests through the same gateway, using the same Apache-2.0 software, configuration files and deployment pipeline. You add provider routing without standing up a second service. The plugin needs Kong Gateway 3.6 or later, and some features need 3.10 or 3.11, so check your version before planning. Its advanced load balancing is documented for Kong's AI Gateway Enterprise offering, not the open-source edition.
- One gateway to operate: AI requests use the same listeners, authentication and logging you already maintain.
- Declarative configuration: Provider settings can live in the same Git-managed files and decK workflow you use for other routes.
- Provider coverage: The plugin lists providers including OpenAI, Azure OpenAI, Amazon Bedrock, Anthropic, Gemini, Ollama and vLLM.
- Open-source limits: Advanced routing algorithms and semantic routing belong to AI Proxy Advanced, which is Enterprise only.
- Version pacing: The open-source line sits on 3.9 while Enterprise releases move ahead, so confirm which plugin features your OSS build includes.
Before you choose this path, confirm the Kong version you can run and whether basic provider routing covers your needs without the advanced load balancing or governance features.
Teams that must cap spend, limit tokens or requests, and assign model access to individuals or departments often find LiteLLM's virtual keys closer to the requirement. Virtual keys support per-key, per-user and per-team budgets, TPM and RPM limits, temporary budget increases and restriction to named models. These controls require a PostgreSQL database, so budgets have a stateful dependency that the gateway must keep available.
- Virtual keys: Per-key, per-user and per-team budgets with TPM and RPM limits are part of the open-source proxy.
- Database dependency: Virtual keys need PostgreSQL, so plan backup, restore and access for that database.
- Identity: The admin UI supports SSO with Google, Microsoft Entra ID, Okta and generic OIDC. From v1.76.0, SSO is free for up to 5 users, and more users require an Enterprise license.
- Enterprise governance: SCIM, JWT authentication, audit logs, organization and team RBAC, IP allowlisting and team-level guardrails sit under LiteLLM's Enterprise license.
- Secrets: Enterprise adds secret manager integrations such as AWS, Azure, Google, HashiCorp Vault and CyberArk.
Confirm with LiteLLM which governance features your audit requirements need, and whether the 5-user SSO allowance and Enterprise features match your user count and identity provider.
Teams whose developers call models directly from Python services often want the same provider abstraction in code and in a shared gateway. LiteLLM provides both a Python SDK and a proxy server that calls more than 100 providers using the OpenAI request format. Developers can test against the SDK locally, then point production traffic at the proxy without rewriting calls. The core project is MIT licensed, and the enterprise directory is covered by its own license file.
- Shared request format: Application code uses one OpenAI-style interface for providers such as Anthropic, Gemini, Bedrock and Azure.
- MIT core: Everything outside the enterprise directory is MIT licensed, so review the enterprise/LICENSE file before you adopt any Enterprise-directory code.
- Scaling requirements: PostgreSQL is the primary supported database. Once more than one proxy instance runs, Redis 7.0 or newer holds shared rate limits, router state and cache.
- Worker sizing: The production guide suggests 1 vCPU and 4 GiB memory per worker, with one Uvicorn worker per pod on Kubernetes.
- Secrets handling: Keep LITELLM_MASTER_KEY, which must start with sk-, in a secret manager.
If your developers already rely on Python libraries, verify that the SDK version you pin matches the proxy version you deploy and that your engineers can operate PostgreSQL and Redis alongside it.
Which advanced routing and governance features need Kong's commercial offering
Kong's open-source AI Proxy covers the basic multi-provider proxy. The features below sit in the commercial AI Gateway Enterprise tier or in LiteLLM's paid Enterprise license, so confirm which ones your team actually needs before choosing a path.
AI Proxy Advanced load balancing
AI Proxy Advanced adds round-robin, consistent-hashing, least-connections, lowest-latency, lowest-usage, semantic and priority balancing across models. The lowest-latency mode uses an exponentially weighted moving average to favor faster-responding models, and semantic routing matches prompts to models by similarity. Both are documented as available only in Kong's AI Gateway Enterprise offering, which needs Kong Gateway 3.8 or later. Check whether your traffic needs these algorithms or whether priority failover is enough.
Enterprise SSO and admin governance
Kong's documentation places the governance controls that come with the commercial gateway behind the Enterprise license, so SSO-backed admin access and role-based controls are not part of the open-source edition. Ask Kong to confirm, in writing, which admin authentication and RBAC features apply to your deployment model. If your identity team already mandates SSO for every admin console, treat this as a blocking requirement during evaluation.
Fully self-hosted Enterprise gateways
Kong's pricing page states that fully self-hosted gateways with a local control plane and data plane are part of Kong Gateway Enterprise. If you need that model, the cost is a commercial license rather than only infrastructure. Open-source Kong Gateway in DB-less or hybrid mode is a different path, and its 3.9 line is the newest open-source release, so confirm which model your compliance team accepts before committing.
LiteLLM routing without a paid license
LiteLLM's open-source router supports simple-shuffle, latency-based, least-busy and cost-based routing, plus retries, cooldowns after repeated failures and priority ordering for failover. These cover many load balancing needs that teams assume require a commercial tier. Test them against your own latency and failure patterns, since the router's behavior under your traffic is the deciding factor, not the algorithm name.
LiteLLM Enterprise for SCIM and audit
LiteLLM Enterprise adds SCIM provisioning, audit logs, organization and team RBAC, IP allowlisting, secret manager integrations and team-level guardrails. SSO is free for up to 5 users in the admin UI, and an Enterprise license is needed beyond that. If your auditors require audit logs or automated user provisioning, budget for that license and ask LiteLLM for current terms in writing.
Running LiteLLM beside existing Kong gateways
The main decision is whether Kong keeps owning the front door for AI traffic or whether LLM calls leave Kong for a separate gateway. In the first pattern, Kong keeps handling client authentication, rate limits and routing for every API, and forwards model calls to a LiteLLM proxy running as an upstream service. Kong authenticates the caller, LiteLLM holds the provider credentials and applies its own keys and budgets, and the two trust boundaries stay separate. In the second pattern, teams route AI endpoints directly to LiteLLM and leave Kong for conventional APIs, which avoids a second hop but means you must duplicate authentication policy in two places. Neither pattern is a built-in integration in the sense of a shipped Kong plugin that hands off to LiteLLM, so the wiring, header passing and identity mapping are your design work to confirm and test.
Virtual keys are the item to place carefully. LiteLLM virtual keys need a PostgreSQL database and carry per-key, per-user and per-team budgets, TPM and RPM limits and model restrictions. If Kong authenticates the user and LiteLLM issues the virtual key, decide which system is the source of truth for who may call which model, and make sure the Kong consumer identity reaches LiteLLM in a form it trusts. Keep the LITELLM_MASTER_KEY in a secret manager, not in Kong declarative files that may sit in Git. Check that a revoked user in your identity provider loses model access in LiteLLM within your accepted window, and record who owns that revocation step.
Test latency and failover before cutover, because each hop adds time and a new failure point. Measure time to first token and total completion time through the two-hop path against a direct call to the provider, using your own prompts and the models you plan to use. For failover, confirm that LiteLLM's router settings (retries, cooldowns and priority ordering, which are documented routing features) fire when a provider returns errors, and that Kong's health checks do not mark a healthy LiteLLM instance as down during provider-side slowdowns. Run the test with more than one LiteLLM instance, since the production guide requires Redis 7.0 or newer for shared rate limits and router state once more than one proxy runs. Ask your DeployOpen or internal team to script these tests so they repeat after every upgrade.
Moving AI routes, keys and budgets off Kong AI Gateway
This sequence moves AI routing, provider credentials and spend controls from Kong's AI plugins to a self-hosted LiteLLM gateway, with a parallel run that lets you confirm behavior before any client traffic changes.
Inventory AI Proxy and AI Proxy Advanced usage
The platform or API team lists every route, consumer and service that carries AI Proxy or AI Proxy Advanced configuration, including the provider, model, load balancing algorithm, semantic routing rules and any priority or fallback order. Note which features depend on Enterprise (AI Proxy Advanced and its load balancing modes) and which use only the open-source AI Proxy plugin. The step is complete when each AI route has a row in a shared inventory with its provider, model names, caller applications and owner.
Export declarative configuration with decK
The team runs decK to dump the current gateway configuration into a declarative file, then stores that file in Git so changes are reviewed. Check the export for plugin configuration, consumers and any references to provider secrets, which should be replaced with placeholders before the file leaves the source environment. The step is done when the export file reviews cleanly and every AI route in the inventory appears in it.
Map providers, models and request types
For each route, the team maps the Kong provider and model settings to LiteLLM model entries and confirms that the request types in use (chat completions, embeddings, files, batches, audio or image generation) are supported by the LiteLLM proxy for that provider. Features that Kong handles and LiteLLM does not, or the reverse, are recorded as gaps with an owner for each. The output is a model mapping table signed off by the application owners.
Stand up LiteLLM with PostgreSQL and Redis
The platform team deploys the LiteLLM proxy in the organization's own infrastructure with PostgreSQL as its database, since virtual keys need PostgreSQL. If more than one proxy instance will run, they also deploy Redis 7.0 or newer for shared rate limits, router state and cache. Following LiteLLM's guidance, they run one Uvicorn worker per pod on Kubernetes and set the LITELLM_MASTER_KEY, which must start with sk-, in a secret manager rather than a config file. The step is done when the proxy starts, reports healthy and accepts a test request against a non-production provider key.
Rotate provider credentials into a secret manager
Provider API keys used by Kong are treated as compromised once they have been stored in gateway configuration exports or plugin settings, so the security team issues new keys at each provider and stores them in the secret manager that LiteLLM reads from. The old keys are revoked after cutover, not before, so that the parallel run can proceed. The step is complete when no provider key appears in plain text in Git, the decK export or the LiteLLM config file, and the revocation date for old keys is written into the change record.
Recreate keys and budgets as virtual keys
The team creates LiteLLM virtual keys that match the consumers in the inventory, applying per-key, per-user or per-team budgets, TPM and RPM limits and model restrictions where Kong enforced them. Budgets for each team are checked against the spend history from Kong so that limits are not set blindly. The output is a register of virtual keys with owners, limits and the models each key can call, delivered to the application teams through a controlled channel.
Run parallel traffic and compare results
For a defined period agreed in the change plan, a share of non-critical requests is sent through LiteLLM while Kong continues to serve production. The team compares response status, latency, token counts, error rates and routing decisions between the two paths, and checks that retries, cooldowns and failover behave as the inventory expects. The step is complete when discrepancies are either fixed or accepted in writing by the application owners.
Cut over and retire Kong AI routes
Once parallel results are accepted, application endpoints are switched to the LiteLLM proxy in a single change with a documented rollback path back to Kong. After a stable period, Kong AI Proxy routes are removed from the declarative file, the old provider keys are revoked and the removal is recorded in the change log. The step is done when all callers use LiteLLM, Kong no longer serves AI routes and the revoked keys have been confirmed as inactive.
Questions buyers ask before replacing Kong AI Gateway
What does the open-source AI Proxy plugin cover compared with Kong's AI Gateway?
The open-source AI Proxy plugin covers basic routing of LLM requests through Kong Gateway, including chat completions, completions, embeddings, files, batches, assistants, responses, audio, image and video generation request types. It supports providers such as OpenAI, Azure OpenAI, Amazon Bedrock, Anthropic, Gemini, Vertex AI, Cohere, Mistral, Hugging Face, Ollama and vLLM, and needs Kong Gateway 3.6 or later, with some features requiring 3.10 or 3.11. Advanced load balancing across models, including semantic and lowest-latency routing, is documented for AI Proxy Advanced, which is part of the commercial AI Gateway Enterprise offering. Confirm the exact feature list against the Kong AI Proxy plugin documentation for the version you plan to run.
Is LiteLLM free for small teams, and what does Enterprise add?
LiteLLM is MIT licensed outside its enterprise/ directory, so the proxy and Python SDK can be run without a license fee. Its admin UI supports SSO with Google, Microsoft Entra ID, Okta and generic OIDC, and from v1.76.0 SSO is free for up to 5 users, with an Enterprise license needed beyond that. Enterprise adds SSO and SCIM beyond that limit, JWT authentication, audit logs, organization and team RBAC, IP allowlisting, secret manager integrations and team-level guardrails and logging. Your costs are the infrastructure, PostgreSQL, Redis once you run more than one proxy instance, and the people who operate it. Ask the vendor to confirm current license terms before you commit.
What does the migration effort depend on?
Effort depends mainly on how many AI routes, providers and applications call the current gateway, and on whether those applications use the OpenAI request format. LiteLLM uses the OpenAI request format across more than 100 providers, so clients that already speak it need changes mostly to their base URL and keys. Other drivers include how many AI Proxy Advanced policies you rely on (semantic or lowest-latency routing needs a different design), how many keys and budgets must be recreated, how identity is wired into the admin UI, and whether logging, guardrails and rate limits must match your current controls. Inventory these before you estimate.
Can our own team run LiteLLM without a partner?
Yes, a capable platform team can run LiteLLM alone if it has the skills to operate PostgreSQL, Redis, Kubernetes or container hosting, secret management and TLS. The production guide names PostgreSQL as the primary database, requires Redis 7.0 or newer once more than one proxy instance runs, recommends one Uvicorn worker per pod with horizontal scaling, and asks you to keep LITELLM_MASTER_KEY in a secret manager. Teams that lack that depth, or that must document controls for a compliance assessment, often bring in outside help for setup and hardening. DeployOpen implements, hardens and documents self-hosted platforms in your own infrastructure, but it does not issue audit reports or certifications.
How long does a pilot take?
The duration of a pilot depends on scope: the number of providers, the applications involved, the controls you need to prove, and the environments you must build. A narrow pilot that routes one application through a self-hosted gateway is much shorter than one that covers SSO, virtual key budgets, guardrails and an audit trail. DeployOpen's engagements are fixed-scope, so the timeline is agreed in writing before work starts. Ask for the scope, deliverables, entry criteria and acceptance tests in the written proposal, and compare each pilot against the same list.
What should we ask for in a written quote?
Ask for a written scope that lists the AI routes, providers, applications and identity integrations in the pilot, plus the controls to be configured, such as virtual keys, budgets, rate limits and logging. Require the deliverables, the environments in which work happens, the acceptance tests, and a clear split between one-time implementation work and recurring support or upgrade work. Confirm which licenses are needed and who buys them, such as LiteLLM Enterprise for SSO beyond five users, and which infrastructure costs fall to you. Ask how upgrades will be handled, since LiteLLM maintains support for the four most recent stable minor versions. DeployOpen does not publish a price list, and its quote follows from the written scope.
What the self-hosted AI gateway costs to run
The main cost decision is which license you pay for. Running the open-source Kong Gateway Apache-2.0 edition carries no license fee, but the AI Proxy Advanced load balancing and semantic routing features are only in Kong's AI Gateway Enterprise offering, so a team that needs them must budget for that license. On the LiteLLM side, the MIT-licensed core carries no fee, while SSO beyond five users, SCIM, audit logs, JWT authentication and secret manager integrations sit in the Enterprise license. Ask each vendor in writing which features your required list depends on and what the license covers per instance, per user or per gateway. Those answers, not the open-source price, usually set the total.
Most of the recurring cost is infrastructure and people. A LiteLLM proxy deployment needs PostgreSQL for virtual keys and spend tracking, and Redis 7.0 or newer once more than one proxy instance runs, so you pay for hosting, backups and patching of both. LiteLLM's production guide suggests 1 vCPU and 4 GiB memory per worker as a starting point, and it recommends one Uvicorn worker per pod with horizontal scaling. Request volume and the number of instances are the main drivers, since each added instance adds compute, database connections and Redis load. Engineering time for monitoring, key rotation, provider changes and upgrades is also recurring. Confirm your expected peak request volume and model mix before sizing anything.
Upgrades and support are where open-source costs tend to appear later. The Kong open-source edition stays on the 3.9 line, and Kong's version support policy applies only to Enterprise, so you should confirm your own patching policy for that line. LiteLLM supports its four most recent stable minor versions, which means your team must upgrade roughly on that cadence to stay supported. Budget a recurring test cycle for each upgrade, and ask any commercial support provider for the scope, response terms and whether it covers the exact version you run. Those items, plus your infrastructure and staff time, form the quote baseline you should compare against the current Kong AI Gateway bill.
