Which open-source Portkey alternative fits your LLM gateway needs
For self-hosted routing, virtual keys and budgets, LiteLLM or Portkey's own AI Gateway are the two open-source starting points. LiteLLM gives you a self-hosted server with one OpenAI-compatible endpoint for 100+ providers, plus virtual keys, budgets, rate limits, spend tracking, load balancing and fallbacks. Portkey's gateway is itself MIT licensed and can run with npx, Docker, Node.js, Cloudflare Workers or Kubernetes. If you also need tracing, pair either gateway with Langfuse or Opik, both of which can be self-hosted.
Teams usually start looking at alternatives for one of three reasons: a renewal or pricing change, a residency requirement, or lock-in to a control plane they do not operate. Portkey's repository lists prompt management, semantic caching, advanced logging and tracing, access control and PII redaction as enterprise features outside the open-source gateway, so a team that relies on those features will need either a commercial license or a replacement for each one. Portkey's hybrid deployment keeps the gateway in your VPC, but the control plane stays in Portkey's VPC and the gateway calls out over HTTPS to control.portkey.ai. Confirm with Portkey exactly which data and metadata that call carries before you treat the setup as fully self-contained, since your residency review will ask.
Before choosing, map each Portkey feature you use to a component. Routing, keys and budgets map to LiteLLM or the open-source Portkey gateway. Prompt management, semantic caching and PII redaction usually need a separate tool or custom code, and you should price that work with an implementation partner before committing. Ask any vendor or partner for a written scope listing components, hosting, upgrade ownership and support terms. Note that LiteLLM's enterprise directory has its own license, so confirm which SSO, audit log and multi-team features you need and whether they fall under it.
Open-source gateways that replace Portkey's core routing
Portkey's core routing role can be filled by self-hosted gateways, but each option covers a different slice of the feature set. The cards below compare LiteLLM and Portkey's own MIT-licensed gateway, then the tracing platforms that usually sit beside a gateway to record requests and costs.
LiteLLM AI Gateway
LiteLLM runs as a self-hosted server that gives applications one OpenAI-compatible endpoint across more than 100 LLM providers. It covers virtual keys, budgets, rate limits, spend tracking, load balancing and fallbacks, and it sends logs to Langfuse, Datadog, OpenTelemetry and Prometheus. The main trade-off is licensing: the code is MIT except the enterprise directory, and SSO, SAML, audit logs and multi-team management sit in LiteLLM Enterprise. Confirm which of your required controls live in that directory before you design around them.
Portkey's MIT-licensed AI Gateway
Portkey publishes its AI Gateway under the MIT license, and it can run with npx, Docker, Node.js, Cloudflare Workers or Kubernetes. Prompt management, semantic caching, advanced logging and tracing, access control and PII redaction are listed as enterprise features outside the open-source gateway. The main trade-off is that a team wanting routing alone can self-host it cheaply, while teams that need those enterprise features must plan to build or buy them separately. Check your deployment target against your data residency rules before choosing Cloudflare Workers.
Langfuse as companion tracing
Langfuse records traces, costs and prompt versions for requests that pass through a gateway. Its code outside the ee directories is MIT licensed, and some advanced features need an enterprise license key. Production self-hosting on Kubernetes with Helm depends on PostgreSQL, ClickHouse, Redis or Valkey, and S3 or blob storage, so plan for those services as recurring operating costs. The main trade-off is operating several stateful services in exchange for detailed trace history that a gateway alone does not keep.
Opik as companion tracing
Opik, built by Comet, is Apache-2.0 licensed, and its README states that the full platform, including tracing, evaluations and prompt management, can be self-hosted without a commercial license. It fits teams that want tracing and evaluation under a permissive license alongside a gateway such as LiteLLM. The main trade-off is that you must confirm the operational footprint, storage and upgrade path in the repository before committing, since those details change between releases.
Key and budget controls at the gateway
Virtual keys and budgets move credential issuance and spend limits into the gateway, so application teams never hold provider keys directly. Confirm how each candidate scopes keys per team or project, where budget counters are stored, and whether a budget breach blocks requests or only raises an alert. Test these behaviors in a staging environment before routing production traffic through the gateway.
Fallback and routing behavior
Fallbacks and load balancing decide what happens when a provider slows or fails. Measure how each gateway handles retries, timeouts and provider-specific errors, since a fallback that switches models can change output quality or data handling. Document the routing rules as configuration under version control so reviewers can audit them, and check whether prompt caching behavior differs from Portkey's semantic caching, which sits outside the open-source gateway.
How each Portkey capability maps to open-source coverage
This table maps each main Portkey capability to LiteLLM, Portkey's MIT-licensed gateway, Langfuse or Opik, and marks it as covered, partial or outside the open-source gateway. Use it to scope which features you must rebuild or replace, then confirm the exact behavior of each project's current release, since gateway and tracing features change often.
| Portkey capability | Open-source coverage | What to confirm |
|---|---|---|
| Virtual keys | Covered by LiteLLM virtual keys; partial in Portkey's open gateway, which focuses on provider routing | Key issuance, rotation and scoping per team or application in the version you deploy |
| Budgets and rate limits | Covered by LiteLLM budgets, rate limits and spend tracking | Whether budgets reset on your schedule and how limits apply across multiple gateway replicas |
| Fallbacks and routing | Covered by LiteLLM fallbacks and load balancing; partial in Portkey's open gateway | Which fallback triggers and routing strategies exist in your release and how they handle streaming |
| Logging and tracing | Partial: LiteLLM logs to Langfuse, Opik-compatible pipelines, OpenTelemetry, Datadog and Prometheus; Portkey's advanced logging is an enterprise feature | Which trace fields are captured by default and how much custom work your dashboards need |
| Access control | Partial: LiteLLM covers keys and teams; SSO, SAML, audit logs and multi-team management sit in LiteLLM Enterprise | Whether your identity provider requirements are met by the open tier or need the enterprise license |
| PII redaction | Partial: Portkey lists PII redaction as an enterprise feature; redaction in open tools usually needs a separate filter or custom callback | Where redaction runs, what it covers, and how you test it against your data classes |
| Prompt management | Outside the open-source gateway in Portkey's repository; Langfuse and Opik provide prompt management under their own licenses | Versioning, labels and deployment workflow in Langfuse or Opik, and how prompts are called from your gateway |
| Semantic caching | Outside the open-source gateway in Portkey's repository; no equivalent is assumed in LiteLLM or Langfuse | Whether you need semantic caching at all, and whether a self-built cache meets your data handling rules |
What you give up when you leave Portkey's managed features
Portkey's open-source gateway covers routing and fallbacks, but several capabilities sit in its enterprise tier or in its hosted control plane. Each one needs an owner after migration, whether that is a tool you add, a service you build, or a control you accept as a gap.
Prompt management and versioning
Portkey's prompt management is listed as an enterprise feature outside the open-source gateway. Without it, prompts live in application code or a repository, and teams lose a shared place to edit, version and roll back templates. The usual workaround is a self-built prompt registry in Git with review gates, or the prompt features in Langfuse or Opik. Confirm who approves prompt changes, how rollbacks happen, and whether prompt IDs in logs can be traced to a release.
Semantic caching
Semantic caching returns stored answers for prompts that are close in meaning, which can cut provider spend and latency. Portkey lists it as an enterprise feature, so the open-source path is usually exact-match caching or a Redis-backed semantic cache your team builds and tunes. Similarity thresholds decide whether users receive wrong or stale answers, so you need test sets and an owner for threshold changes. Confirm which caching behavior your gateway supports before you plan capacity.
Managed control plane and dashboards
In Portkey's hybrid model, the gateway runs in your VPC while configuration and dashboards stay in Portkey's control plane, reached over HTTPS. Self-hosting the gateway removes that dependency, but you then run the admin console, metrics and log storage yourself. Plan for a monitoring stack, retention rules and on-call coverage for the gateway. Confirm which dashboards you need on day one, such as spend by team or error rate by provider.
PII redaction
PII redaction is listed as an enterprise feature outside the open-source gateway. Without it, prompts containing names, account numbers or patient details can reach external model providers unchanged. Common workarounds are a redaction service placed in front of the gateway, such as a pattern and entity detector, or a policy that blocks certain fields from prompts entirely. Test redaction against your own data samples, and confirm which data classes must never leave your network.
Enterprise access control and SSO
Portkey's access control is an enterprise feature, and LiteLLM places SSO, SAML, audit logs and multi-team management in LiteLLM Enterprise under its separate enterprise license. Teams that want SSO without that license usually put an identity-aware proxy or an existing identity provider in front of the gateway and map groups to virtual keys themselves. Audit trails then depend on your proxy and log pipeline. Confirm whether your auditors need per-user attribution of model calls and who approves key issuance.
How data moves when the gateway runs in your own VPC
Portkey's hybrid deployment splits the product in two. The data plane, the gateway, runs in your VPC, so prompts and completions pass through infrastructure you control. The control plane, which holds the configuration and dashboard functions, stays in Portkey's VPC, and the gateway calls out over HTTPS to control.portkey.ai. That outbound call is the part your security team needs to model. Before you commit, confirm with Portkey exactly which metadata, configuration and logging fields cross that boundary, whether any request or response content is sent, and what happens to the gateway if the control plane is unreachable.
A fully self-hosted gateway removes that dependency. LiteLLM's proxy, or the open-source Portkey AI Gateway run with npx, Docker, Node.js or Kubernetes, serves requests from configuration you hold, so no control plane call is needed for routing or key checks. The trade-off is that you own what the hosted control plane provided: config distribution, key storage, dashboards and upgrades. Confirm in your own environment whether the gateway process needs any outbound route at all for your LLM providers, and whether it can run in an egress-restricted subnet with only provider endpoints allowed.
For residency-driven decisions, ask your security team to verify four things: the egress allow-list for each gateway environment, whether request logs or traces leave the VPC and where they are stored, how secrets for provider keys are injected and rotated, and how the logging destination is set so that prompts are not written to a third-party store by default. Test the design with a non-production gateway and capture the actual network flows, because a diagram alone rarely shows what a gateway sends.
How to move gateway traffic off Portkey in phases
This sequence moves LLM gateway traffic from Portkey to a self-hosted gateway in controlled stages, so each team can be checked before the Portkey keys and data are retired. Each step closes with a artifact or check that the next step depends on.
Inventory keys, providers, routing and budgets
Platform engineering lists every virtual key, upstream provider credential, routing rule, fallback chain, budget and rate limit currently configured in Portkey, and records which application or team uses each one. Prompt templates and any semantic caching or PII redaction rules are listed separately, because those features sit outside the open-source gateway and need a replacement decision. The step is done when the inventory maps each consuming application to its keys, routes and budgets, and the owners have signed off on it. Confirm with Portkey which of these objects can be exported in bulk and in what format before you rely on that export.
Export configuration and logs from Portkey
Export the gateway configuration and retained request logs that you need for audit, cost history or troubleshooting. Store them in a bucket or archive that your security team controls, and record checksums and the export date. Ask Portkey in writing which configuration objects, log fields and retention limits apply to your plan, and whether prompt templates are included in the export. The step is done when the exported files open successfully and a sample of records matches what the Portkey dashboard shows.
Map Portkey config to LiteLLM or the open gateway
Translate each inventoried route, fallback, virtual key and budget into the equivalent LiteLLM proxy configuration or the MIT-licensed Portkey gateway configuration, depending on which you choose. Where a feature has no direct equivalent, such as prompt management or semantic caching, record the decision: rebuild it elsewhere, drop it, or keep it in the application. The output is a version-controlled configuration file, a gap list and a short written decision for each gap, reviewed by the security and platform owners.
Pilot the gateway with one team
Deploy the self-hosted gateway in your own environment and move one low-risk application to it by changing only its base URL and key. Confirm that requests succeed, that budgets and rate limits fire as configured, that fallbacks trigger when a provider is forced to fail in a test, and that logs reach your chosen observability backend. The step is done when the pilot team has run its normal workload for an agreed period and has recorded any defects, each with an owner and a fix or accepted risk.
Run both gateways in parallel with traffic split
Route a defined share of production traffic through the self-hosted gateway while the remainder continues through Portkey. Compare response success rates, latency, spend per team and fallback behavior between the two paths, using the same providers and models. Increase the share in steps only when the comparison shows no unexplained differences. The step is done when the comparison report is approved by the application owners and the platform team agrees to proceed to cutover.
Cut over application base URLs
Change the base URL and credentials in each application's configuration or secrets store, team by team, using the mapped configuration from earlier. Keep the Portkey path available for rollback until the last application has moved. Verify that the new traffic appears in the self-hosted gateway logs and that spend tracking matches expectations for each team. The step is done when no application sends requests to Portkey endpoints and all teams have confirmed that their dashboards and alerts work.
Decommission Portkey keys and retained data
Revoke Portkey virtual keys and provider credentials that were routed through Portkey, rotating any upstream provider keys that Portkey stored. Ask Portkey to confirm in writing how and when it deletes your configuration and log data, and retain the copies your policy requires in your own archive. If you use Portkey's hybrid deployment, confirm which control plane connections and data flows must also be shut down. The step is done when the revocation and deletion confirmations are filed with the change record.
Which gateway fits each situation Portkey customers face
Teams that must keep prompts, responses and gateway traffic inside their own network usually start here. Portkey's AI Gateway is MIT licensed and can run with npx, Docker, Node.js, Cloudflare Workers or Kubernetes, so you can deploy the gateway in your own environment. Portkey's hybrid model is different: the data plane runs in your VPC, but the gateway still calls out over HTTPS to control.portkey.ai for the control plane. If your policy forbids any outbound call to a vendor control plane, a fully self-hosted deployment of LiteLLM avoids that dependency entirely.
- Control plane calls: Confirm with Portkey whether your hybrid setup needs the outbound path to control.portkey.ai, and whether your egress policy allows it.
- Observability sinks: LiteLLM can send logs to Langfuse, Datadog, OpenTelemetry and Prometheus, so decide which of those stays inside your boundary.
- Key storage: Virtual keys and provider credentials live in the gateway's configuration and database, so set where they are stored and who can read them.
- Model provider endpoints: Residency also depends on the LLM providers you call, which sit outside the gateway.
Before you choose, check the residency rules in your contracts and map every network path the gateway uses.
Teams facing renewal increases often want the gateway core without a per-seat or platform fee. The MIT-licensed Portkey AI Gateway and LiteLLM's open-source proxy both avoid a license charge for the core routing features. The cost you carry moves to infrastructure, operations and staff time. Those costs recur, while the initial setup is largely one-time work. Your main decision is whether your platform team can run the gateway, its database and monitoring reliably.
- Infrastructure: Compute for the gateway, a database for virtual keys and spend records, and storage for logs.
- Operations: Upgrades, patching, scaling and on-call coverage for the service.
- Feature gaps: Portkey features such as prompt management and semantic caching sit outside the open-source gateway, so you may need to build or buy them separately.
- Enterprise add-ons: LiteLLM Enterprise is a separate paid license, so price it only if you need its features.
- Quote questions: Ask any implementation partner for a written scope that lists each component and who operates it.
Compare total cost of ownership with your current invoice across at least one full billing cycle of usage.
Engineering teams that need custom routing logic, provider-specific handling or internal integrations benefit from open configuration and readable code. LiteLLM exposes one OpenAI-compatible endpoint for more than 100 LLM providers, with load balancing, fallbacks, rate limits and budgets set through configuration. The MIT license covers the code outside its enterprise directory, so you can fork or patch the parts you need. Changes to the core code add maintenance work with each upstream release, so keep custom code in the smallest possible layer.
- Configuration: Routing, fallbacks, budgets and rate limits are defined in config files that you can version in Git.
- Provider coverage: Check that every model you use is supported and tested in the version you deploy.
- Callbacks: Custom logging or policy hooks can be added to the proxy, which keeps application code unchanged.
- Upgrade cost: Each change to upstream code must be retested after upstream releases.
- Licensing boundary: Keep a record of which files are under the enterprise license so you do not ship them by accident.
Illustrative example: a team that needs one fallback rule for a single internal model can add it in configuration rather than modifying code, which keeps upgrades simple. Confirm the exact configuration keys against the version you plan to run.
Regulated organizations often need SSO, audit logs and multi-team administration before they can approve a gateway. LiteLLM Enterprise adds SSO and SAML, audit logs and multi-team management, which covers the most common identity and evidence requirements for an internal gateway. Portkey's enterprise features, including access control and PII redaction, also sit outside its open-source gateway. Choose the option whose enterprise feature set matches the control evidence your assessor will request, and do not assume a feature covers a requirement without testing it.
- Identity: Confirm SSO and SAML support in the edition and version you will license.
- Audit logs: Verify which events are recorded, where they are stored and how long they are retained.
- Access control: Map roles to the teams that manage keys, budgets and routing rules.
- PII handling: Portkey lists PII redaction as an enterprise feature, so an open-source deployment needs its own redaction step.
- Compliance support: DeployOpen prepares systems, controls and evidence for assessments, but independent auditors issue any report or certification.
Ask your assessor which evidence they need before you select the edition.
How long and how much effort does replacing Portkey take?
What drives the cost of replacing Portkey with a self-hosted gateway?
The cost is driven mostly by labor and by the infrastructure you already run, not by the gateway software, which is open source. The main components are the engineering time to deploy the gateway, re-point applications to its OpenAI-compatible endpoint, move virtual keys and budgets, and rebuild logging. Recurring costs include the compute and database capacity the gateway needs, observability storage, on-call time and upgrades. Costs rise with the number of applications, providers, teams and environments, and with any feature you must rebuild rather than reuse. Ask any quote to itemize one-time work separately from recurring operations, and to state which Portkey features are assumed to be replaced and how. Quotes follow from written scope, so the more precise your inventory, the more precise the estimate.
Can our own team migrate off Portkey without outside help?
Yes, a capable platform team can migrate, especially for a limited number of applications and providers. The gateway can run through npx, Docker, Node.js, Cloudflare Workers or Kubernetes, according to Portkey's AI Gateway repository, so the basic deployment is straightforward. The harder parts are the operational ones: high availability, key management, budget enforcement across teams, alerting, and rehearsed rollback. Teams without Kubernetes or observability experience usually find those parts take longer than the initial setup. Be honest about who will keep the system running after launch. If that owner is not yet named, that gap is a larger risk than the migration itself.
What does our team need to provide before a migration starts?
Three inputs determine whether the work can be scoped accurately: provider credentials, a configuration inventory, and a named on-call owner. Provider keys for each LLM vendor must be issued or rotated into your secret store, with spend limits set at the provider level where available. The configuration inventory should list every route, virtual key, budget, fallback chain, rate limit and caller, along with the applications that depend on each one. Finally, name the person or team who receives alerts and approves changes after go-live. You should also confirm which environments the gateway may run in, such as your own cloud account, a data center or an air-gapped network, because that determines the network and identity work.
Can semantic caching and prompt management be replaced after leaving Portkey?
Semantic caching and prompt management are the two areas where you should expect the most rebuilding. Portkey's AI Gateway repository lists prompt management and semantic caching among enterprise features outside the open-source gateway, so you should not assume an equivalent is included in the open-source code. Exact-match response caching can be handled at the application or gateway layer, but semantic matching needs an embedding model, a vector store and a tuned similarity threshold, each with its own cost and quality tradeoffs. Prompt management can be replaced with a versioned prompt registry in a separate open-source platform, or kept in source control. Confirm which of these features your applications actually depend on before estimating effort, because many teams use them less than they assume.
What is included in a fixed-scope DeployOpen engagement for this migration?
Most DeployOpen engagements are fixed-scope, so scope, deliverables and timelines are agreed in writing before work starts. For a gateway migration, a typical written scope would cover planning the target architecture, deploying the chosen open-source gateway in your own infrastructure, configuring virtual keys, budgets, rate limits and fallbacks, integrating logging and monitoring, hardening, documenting the runbook, and cutting over application traffic. Items that may be excluded include rebuilding semantic caching, migrating prompt libraries, or changing application code, and these should be listed explicitly. Ongoing support, upgrades and continuous testing are available as separate retainers. Request the written scope with each deliverable and acceptance condition named.
How do we keep the open-source gateway running after cutover?
Plan for operations as a recurring cost from the start. After cutover, the gateway needs monitoring, patch and version upgrades, key rotation, capacity reviews and a tested rollback path. Portkey's hybrid deployment, by contrast, keeps the control plane in Portkey's VPC while the gateway calls out over HTTPS to control.portkey.ai, so an open-source deployment removes that dependency but moves those duties to you. Define service expectations internally, including who responds to outages and how quickly provider failover must work. If your team cannot staff this rotation, a retainer for ongoing support and upgrades is one option to price against the written scope.
What should we confirm about Portkey's current terms before deciding?
Confirm your current Portkey contract terms, renewal dates and the exact features in your plan before you set a migration budget. Ask Portkey which capabilities your account uses, including access control, PII redaction, advanced logging and tracing, because the open-source gateway does not include those enterprise features. Also ask what data export options exist and in what format, so you can estimate the migration of historical logs. Your own environment matters as well: confirm where your provider keys, logs and customer data may reside, and whether your security team needs to review the new deployment before production traffic moves.
How DeployOpen scopes and hands over a gateway migration
DeployOpen starts a gateway migration with a written, fixed scope. DeployOpen is a Chainscore Labs company that plans, implements, hardens, backs up, documents and supports self-hosted open-source platforms inside the client's own infrastructure, whether that is a cloud account, a data center or an air-gapped network. For a move off Portkey, the scope should name each piece in writing before work begins: which gateway runs (the open-source Portkey AI Gateway, LiteLLM, or both), where it is deployed, how the virtual keys and budgets are modeled, which provider credentials move and how they are stored, and which Portkey features are retired, replaced or kept. Features such as prompt management, semantic caching, advanced logging and tracing, access control and PII redaction sit outside the open-source Portkey gateway, so the scope must say explicitly how each one is handled. Ask for deliverables, acceptance criteria and a timeline in the same document, and confirm whether a Portkey hybrid setup, which keeps a data plane in your VPC and a control plane with Portkey, is part of the current estate.
Handover is where most of the value of a self-hosted gateway is either kept or lost, so the scope should specify the documentation set up front. A useful package covers the deployment topology, configuration files and secrets handling, the routing and fallback rules, the budget and key lifecycle procedure, backup and restore steps for any state the gateway keeps, and the runbook for rotating provider credentials. Request that the rules are written as versioned configuration so your platform team can review changes in the same way as other code. Agree in advance on how cutover is verified, such as comparing request logs and spend figures across a parallel period, and who signs off on each step. Clarify what the client team will run alone after handover and what they will need help with.
Ongoing work is usually handled through a separate retainer rather than folded into the migration fee. Retainers can cover gateway upgrades, provider and model changes, security patching, periodic testing and support requests, and they are scoped and priced in writing like the original engagement. When you request a quote, ask which upgrade and support activities are included, what response expectations apply, and what happens if your provider list or routing requirements change mid-contract. DeployOpen does not publish a price list, so the quote follows from the agreed scope. DeployOpen does not hold partner status with Portkey, LiteLLM or any other vendor, and it does not issue certifications or audit reports, so confirm any compliance evidence needs directly with your auditor.
