Open-source alternative to Portkey semantic caching: the short answer
If your team wants semantic caching but cannot send prompts to a hosted control plane, the open-source route is the LiteLLM proxy, which you run in your own infrastructure. Its caching layer supports exact-match caches and semantic caches on Redis, Valkey or Qdrant, so the cache, the keys and the stored responses stay in systems you operate. Portkey lists semantic caching as an enterprise feature outside its open-source gateway, so this is the point where you replace a paid capability rather than a convenience feature. The central trade-off is wrong answers against hit rate. A semantic cache serves the closest stored response once it clears a similarity threshold, so a lower threshold produces more hits and more answers to questions that only look similar, while a higher threshold is safer but saves fewer calls. Exact-match caching avoids that risk because it keys on a hash of the whole request, but it hits far less often. Before rollout, confirm the threshold with a test set of real prompt pairs from your own workloads, and decide which routes (for example, retrieval-heavy or regulated answers) must never be served from a semantic cache.
Before go-live, confirm three things in your own environment rather than assuming them. First, what the cache stores: prompts, completions, metadata and any embeddings used for matching, and for how long, since that determines whether personal or regulated data lands in Redis, Valkey or Qdrant. Second, the cache backend's deployment model: in-memory and disk caches are not shared across workers, so a multi-instance gateway needs a shared store, and Redis-based caches need a Redis version and high availability design that matches your uptime needs. Third, the access controls on the cache store itself, because a cache that holds model responses is a data store and needs the same network, authentication and retention rules as one. Confirm each with the vendor documentation for the version you deploy and with your security team.
Which Portkey caching capabilities map to LiteLLM and which need your own work
This table maps Portkey's caching features to LiteLLM cache types and the Redis Open Source or Qdrant store behind each, then lists the gaps you will close yourself. Confirm each Portkey feature against your current plan before you rely on the mapping, because enterprise feature boundaries change.
| Portkey capability | LiteLLM equivalent | Store behind it and what you run | Gap or work left to you |
|---|---|---|---|
| Simple (exact-match) caching | Exact-match cache keyed on a hash of the whole request | Redis Open Source or Valkey, self-hosted with the proxy | Confirm whether your Portkey plan includes simple caching; the cache key covers the full request, so any prompt difference is a miss |
| Semantic caching | redis-semantic, valkey-semantic or qdrant-semantic cache serving the closest match above a threshold | Redis Open Source with its query engine, Valkey, or self-hosted Qdrant with an embedding model | You choose the similarity threshold and embedding model; validate that near-matches return correct answers for your prompts |
| Cache backend choice | s3, gcs, in-memory and disk caches | Object storage or local disk | In-memory and disk caches are not shared across workers, so they suit single-instance use only |
| Cache controls and TTL | Per-request cache settings in the proxy configuration | Set at the proxy and the store | Confirm which controls Portkey exposes per request and map each one to a LiteLLM setting during testing |
| Per-tenant or per-team cache policy | Not a documented native control in the caching docs | Separate namespaces or stores per tenant as a design choice | Build tenant isolation yourself and test that one team never receives another team's cached response |
| Cache observability | Logging goes through the proxy callbacks such as Prometheus or OpenTelemetry | Your metrics stack, for example Prometheus and Grafana | Define hit rate, miss rate and wrong-answer sampling as dashboards and reviews; the proxy does not provide this view alone |
| Data stored in the cache | Prompt and response content, plus embeddings for semantic matches | Redis or Qdrant on your infrastructure | Decide retention, encryption and access rules, since cached prompts can contain personal or confidential data |
| Shared cache across proxy instances | Redis-backed caches are shared when several proxy instances run | Redis 7.0 or newer for multiple proxy instances, per the LiteLLM production guide | Plan Redis high availability, for example Sentinel with at least three Sentinel instances, because the cache sits on the request path |
How exact-match and semantic caches decide what to serve
An exact-match cache serves a stored response only when the new request is identical to an earlier one. LiteLLM computes a hash over the whole request, so the model name, messages, parameters and any other request fields all have to match before the cache returns anything. That makes the exact-match cache predictable: a hit is the same request that produced the stored answer, so the risk of a wrong answer is low. The trade-off is a low hit rate for applications where users phrase the same question in different ways. Before you rely on it, pull a sample of real prompts from your logs and count how many repeat character for character. That count tells you what the exact-match cache can save.
A *semantic cache* works differently. It embeds the incoming prompt, searches a vector index for the closest stored prompt, and returns the stored response when the similarity score is above a configured threshold. LiteLLM supports this through redis-semantic, valkey-semantic and qdrant-semantic cache types. The threshold is the control that matters most. A high value limits hits to near-identical wording, while a lower value raises the hit rate and also raises the chance that a question about one product, date or account receives an answer written for another. Test the threshold against a labeled set of prompt pairs from your own traffic, including pairs that look alike but need different answers, and record the wrong-answer rate at each setting before you choose one.
In-memory and disk caches live inside a single proxy process or on one machine's disk, so they are not shared across workers. Two replicas behind a load balancer would each build their own cache, and a request that hit one replica could still miss on the other. That makes a shared store such as Redis, Valkey or Qdrant the practical choice once you run more than one instance. Confirm with your platform team whether the gateway will run several replicas, and if it does, plan the shared cache store, its high availability and its access controls at the same time as the gateway itself.
Choosing the cache store: Redis, Valkey, Qdrant or object storage
LiteLLM's semantic caches write each prompt and response to a store you run yourself, so the store choice decides the license you accept, the failover design you need and who can read cached content. The cards below compare the Redis, Valkey and Qdrant options and the object storage backends that do not support semantic lookup.
Redis Open Source licensing from 8.0
The redis-semantic cache runs on Redis. From Redis 8.0, you choose one of three licenses: RSALv2, SSPLv1 or AGPLv3. Versions 7.4 to 7.8 remain under RSALv2 or SSPLv1, neither of which is OSI-approved, and 7.2 and earlier stay under BSD-3-Clause. Before you pin a version, have legal confirm which option covers your use, especially if you modify Redis and offer it as a network service. Redis 8.0 also bundles a vector search engine, so confirm which features your cache actually needs.
Valkey as the BSD-licensed store
The valkey-semantic cache offers a Redis-compatible store for teams that do not want to adopt Redis licensing terms. Valkey is a community fork of Redis, so confirm the version your LiteLLM release supports and check your client library before you migrate. Your operators will need to learn Valkey-specific packaging and support channels, since the Redis Ltd. commercial options do not apply. Test cache hit behavior after switching, because semantic search behavior depends on the module and version you run.
Qdrant under Apache-2.0
The qdrant-semantic cache uses Qdrant's vector search engine, which is Apache-2.0 licensed and free to self-host. Its storage needs block-level POSIX file systems on SSD or NVMe, so it cannot sit on NFS or S3. Qdrant is a dedicated vector engine, so it adds a service to run but suits teams that want vector search separate from their key-value cache. Your cost comes from the compute, storage and the people who operate it, not from a license fee.
Secure Qdrant before exposing it
By default, self-deployed Qdrant listens on all network interfaces and has no authentication configured. Before any application reaches it, enable admin and read-only API keys, turn on TLS for REST and gRPC, and restrict collections with JWT-based read or write permissions. Turn on audit logging of authenticated API operations so you can show who queried or changed a cache collection. Treat the cache as sensitive, since it holds prompts and responses.
Qdrant production paths and gaps
Qdrant offers three production paths: managed Qdrant Cloud, a community Kubernetes Helm chart and an enterprise Kubernetes operator in Qdrant Private Cloud. The Helm chart lacks enterprise features such as zero-downtime upgrades and automatic shard rebalancing. For a cache, decide whether you can accept a maintenance window for upgrades, or whether you need the operator path and its quote. Ask Qdrant which path fits your upgrade process before you commit.
High availability for the cache tier
LiteLLM requires Redis 7.0 or newer once more than one proxy instance runs, and the cache is shared across those instances. Redis Sentinel gives automatic failover of a replica to master when you do not use Redis Cluster, but a robust deployment needs at least three Sentinel instances. For Qdrant, plan at least three nodes so quorum survives one node failure, and set the replication factor with its storage cost in mind. Confirm which layout your team can operate before you size the cluster.
How similarity thresholds create wrong answers
A semantic cache does not return answers to identical questions. It returns a stored response for a new prompt that is only similar to one it has seen, and the LiteLLM semantic cache serves the closest match once its similarity score clears the configured threshold. Two prompts can score as near-duplicates while differing in a detail that changes the correct answer. Illustrative example: a cached reply about the refund window for a card account from last quarter could be served to a question about the same account after a policy change, because the wording overlaps almost completely. The embedding model sees the shared phrasing, not the changed date, amount, account type or clause number.
Set the threshold per use case rather than once for the whole gateway. Start by listing the request types that reach the cache, then rank them by how costly a wrong answer would be. Content such as product descriptions or general help text can tolerate looser matching. Anything that quotes a figure, a policy clause, an account state or a regulatory position should either use exact-match caching or a strict semantic threshold, or skip the cache entirely. Keep the cache scoped by model, system prompt and, where relevant, tenant, so that a match cannot cross a boundary you care about. Confirm the exact parameters in the LiteLLM caching documentation for your version, since names and defaults can change between releases.
Before enabling any threshold in production, build a labeled test set of prompt pairs. Include pairs that should match, pairs that differ only in a material detail, and pairs on unrelated topics. Run each pair through the cache with the candidate threshold and record whether the served answer is correct for the second prompt. Repeat the test whenever you change the embedding model, the threshold or the cached content. Then log every cache hit with both the incoming prompt and the matched prompt, the similarity score and the cache key, and sample those hits on a schedule for human review. Ask your reviewers to flag any hit where the answer would be wrong for the new question. The review log is the evidence you need to tune the threshold and to show an auditor how wrong answers were detected.
Moving from Portkey semantic caching to LiteLLM caching
This sequence moves cached LLM responses from Portkey's enterprise semantic cache to LiteLLM's exact-match and semantic caches on a store you run, with thresholds, data rules and hit-rate evidence set before cutover. Cached entries do not transfer from Portkey, so the cache rebuilds from live traffic after the switch.
Inventory applications that rely on Portkey caching
The platform team lists every application, route and virtual key that sends requests through Portkey with a cache setting enabled, along with the owning team and the prompt types each one sends. Include internal tools, batch jobs and evaluation runs, because they often reuse cached answers without anyone noting it. Ask Portkey support or check your Portkey configs to confirm which settings are semantic and which are simple caching, since the simple mode is part of the open-source gateway. The step is done when each cached route has an owner and a classification of its prompts, such as public text, internal documents or personal data.
Run exact-match caching before semantic caching
Point the LiteLLM proxy at a Redis-compatible store and enable the exact-match cache first, since it keys on a hash of the whole request and returns an answer only when the request is identical. This removes the risk of a near-match returning a wrong answer and shows how much repeat traffic your applications actually generate. The platform team confirms in the LiteLLM configuration that the cache type is set for the proxy and that repeated identical requests return from the cache. The step closes when exact-match hits appear in logs for at least one non-critical application.
Define similarity threshold and data rules
Security, compliance and application owners agree in writing on the similarity threshold for semantic matching, the routes allowed to use it, and what the cache may store. Set the threshold conservatively and test it with pairs of prompts that look alike but need different answers, such as different account numbers, dates or jurisdictions, because a loose threshold serves a wrong answer that looks confident. Decide whether prompts and responses containing personal or regulated data are excluded from caching, and set retention for entries. The step is done when the agreed threshold, excluded routes and retention rule are documented and reviewed.
Stand up the cache store with access controls and backups
Deploy the Redis, Valkey or Qdrant store inside your network with authentication enabled, TLS for client connections and restricted network exposure. Qdrant ships without authentication by default, so set an API key before any application connects, and confirm the same for Redis or Valkey. Configure replication or high availability for the store and a backup schedule that matches your retention rule, noting that cache data may hold sensitive prompt content. The platform team tests a restore into a separate environment and records the result. The step is done when an unauthenticated connection is refused and a test restore succeeds.
Pilot caching with hit and miss logging
Route one application through LiteLLM with caching enabled and log every cache hit and miss, along with the route, model and threshold in use. Compare answer quality for hits against a sample reviewed by the application owner, and keep a record of any answer that was wrong or out of date. Confirm that cached responses carry no data the owner has excluded from caching, and that logs do not expose prompt text beyond what your policy allows. The pilot closes when reviewers have signed off on the sampled hits and the hit and miss rates are recorded for each route.
Cut over and retire the Portkey cache setting
Move each inventoried application to the LiteLLM endpoint in waves, starting with the routes that passed the pilot, and turn off the Portkey cache setting for each route once its replacement is confirmed. Expect a cold cache after cutover, because Portkey cached entries do not transfer and the LiteLLM cache fills from live requests, so hit rates start low and rise over time. Keep the Portkey setting in place for a fallback window that the owners agree on, then remove it. The step closes when every inventoried route runs through LiteLLM caching, the Portkey cache setting is disabled and the change record is filed.
Running the cache alongside the rest of your LLM stack
The cache does not run alone. It sits inside a LiteLLM proxy that also handles virtual keys, spend tracking, routing and logs, and each of those components adds a service you operate. The cards below cover what each shared piece costs to run and where its decisions touch the cache.
LiteLLM proxy as the cache host
The semantic cache is configured on the LiteLLM proxy, so every request that uses it passes through the same server that enforces keys and routes to providers. Plan the proxy's capacity around that role, not around the cache alone. The production guide suggests 1 vCPU and 4 GiB memory per worker and recommends one Uvicorn worker per Kubernetes pod, so horizontal scaling is how you add capacity. Confirm the sizing against your own traffic before you commit to a node count.
PostgreSQL for keys and spend
Virtual keys need a PostgreSQL database, and the same database records per-key, per-user and per-team spend. Cache hits still reach the key layer, so a budget check runs before a cached answer is returned. Your team operates backups, upgrades and connection limits for this database. Ask how the proxy batches spend writes at scale, since the guide suggests batching every 60 seconds.
Redis 7.0 or newer for shared cache
With more than one proxy instance, Redis 7.0 or newer holds shared state for rate limits, router state and the cache. Without it, in-memory and disk caches stay private to each worker, so a repeated prompt can miss on one pod and hit on another. Choose a Redis distribution whose license you have reviewed, since Redis 8.0 and later are offered under a choice of RSALv2, SSPLv1 or AGPLv3. Ask your provider which version and high availability setup they run.
Logs to Langfuse or OpenTelemetry
Cache hits and misses appear in the proxy's logs only if you send them somewhere. LiteLLM can forward logs to Langfuse, Datadog, OpenTelemetry and Prometheus. Logged prompts and responses are the main data the cache exposes outside the cache store, so check what each destination retains before enabling full-payload logging. A self-hosted Langfuse adds PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible storage to your operating scope.
Cache keys and virtual key budgets
An exact-match key is a hash of the whole request, which includes the model and parameters, so keys issued to different teams do not share hits unless their requests match. Verify whether your cache key includes the virtual key or team identity before assuming a hit saves budget for another team. A cache hit usually costs no provider tokens, so it reduces spend for the key that made the request. Test this with a sample key set before rollout.
Routing and fallbacks after a miss
On a miss, the router chooses a deployment using its strategy, such as simple-shuffle, latency-based or cost-based, and it applies retries, cooldowns and fallbacks. Cached answers bypass that path, so a model you retired or moved can still appear in cached responses until entries expire or are cleared. Set expiry and flush procedures as part of the rollout plan, and confirm them with the team that owns model changes.
Questions teams ask before replacing Portkey caching
What does it cost to run a self-hosted semantic cache?
The cost is mostly the infrastructure and the people who run it, and the software itself is free to use. LiteLLM is MIT licensed outside its enterprise directory, and Qdrant's open-source engine is Apache-2.0 and free to self-host. The recurring costs are compute for the proxy, a Redis or Valkey instance for the exact-match and state layer, a vector store if you choose a semantic cache on Qdrant, and the storage and backups for each. Redis 7.0 or newer is needed once more than one proxy instance runs, because shared cache and router state depend on it. Ask your cloud or data center team to price these components, and ask any vendor quoting Qdrant Cloud or Redis Cloud to separate the managed fee from the self-hosted option. The one-time costs are design, deployment and testing.
How long does it take to put a semantic cache into production?
The timeline depends on how many applications call the gateway, whether the cache must be highly available, and how much testing the team needs before trusting cached answers. A single proxy with an in-memory or disk cache is the quickest path, but those caches are not shared across workers, so they do not fit a multi-instance gateway. Shared Redis or Valkey caches add a deployment, a Sentinel or failover design and monitoring. Qdrant adds block storage planning, authentication and, for production, at least three nodes so quorum survives a node failure. Ask for a written scope with named milestones before work starts, because these choices change the schedule more than the caching code does.
What does our team need to provide before the work starts?
You need a clear inventory of the applications and prompts that will pass through the cache, the cloud or data center where it will run, and an owner who can approve the cache policy. The team also needs a PostgreSQL database for LiteLLM virtual keys if keys and budgets are part of the same gateway, identity provider details for admin access, and a secret manager for LITELLM_MASTER_KEY, which must start with sk-. Confirm storage classes for Qdrant, since it needs block-level storage with a POSIX-compatible file system and cannot run on NFS or S3. Finally, prepare a test set of real but approved questions so the team can measure cache behavior before cutover.
Can our own engineers run the cache without an outside partner?
Yes, if your team already operates Kubernetes, Redis and a vector database and has capacity for on-call work. Teams with that experience can deploy LiteLLM with one Uvicorn worker per pod, scale horizontally and follow the production guide. Gaps usually appear in threshold tuning, evaluation of wrong answers, security hardening and failover testing. Qdrant is not secure by default, because it listens on all interfaces without authentication, so you must configure API keys, TLS and network policy. Redis Sentinel needs at least three Sentinel instances for a robust deployment. An outside partner can take on any of these as a fixed-scope piece, and DeployOpen's role is to plan, implement, harden and document the work in your environment.
How do we measure wrong answers from a semantic cache?
Measure wrong answers by sampling cache hits, comparing each served answer with a fresh model response on the same prompt, and grading the difference against criteria your domain experts agree on. Track the similarity score for every hit, since a semantic cache serves the closest match above a configured threshold, so a looser threshold increases hits and increases the chance that a different question receives an old answer. Run this test separately for each application and each prompt family, because a threshold that works for FAQ text can fail for questions with numbers, names or negations. Keep the threshold change log in version control so you can tie an error rate to a specific setting. Note that LiteLLM documents the threshold behavior, but the accuracy of your application depends on your own test data, so confirm the chosen threshold with a representative sample before enabling it broadly.
Does caching LLM responses break our data residency rules?
It can, if the cache stores prompts or responses in a location your rules do not allow. The LiteLLM cache keys on a hash of the whole request for exact matches, and semantic caches store the matching content in the backend you select, so the location of Redis, Valkey, Qdrant or an S3, GCS or disk backend decides where that content lives. Keep every cache backend in the same region and network boundary as the gateway, and check which fields your data protection team classes as personal data, because a cached prompt can contain them. Apply the same retention limits to the cache as to logs, and confirm with your compliance lead whether the cache counts as a processing record. DeployOpen prepares the controls and evidence for compliance assessments, but independent auditors issue any attestation.
What changes if we later move the cache from Redis to Valkey or Qdrant?
The gateway configuration changes, but the application contract does not, because applications keep calling the same OpenAI-compatible endpoint. LiteLLM supports redis, redis-semantic, valkey-semantic and qdrant-semantic cache types, so switching is mainly a config change plus a plan for the existing cache entries, which you can usually discard because a cache can be rebuilt. The bigger decisions are licensing and operations. Redis 8.0 and later can be used under AGPLv3, but Redis 7.4.x to 7.8.x are under RSALv2 or SSPLv1, which are not OSI-approved, so confirm the version you run. Qdrant needs its own storage, backup through snapshots and, for production, a multi-node design. Test the new backend with your threshold set before you switch traffic.
How DeployOpen scopes and delivers a caching replacement
A caching replacement touches configuration, data handling, network exposure and ongoing upgrades, so the work is scoped in writing before any cache is switched on. The cards below follow that order, from the agreed scope through to the retainers that keep the cache current after handover.
Written scope and deliverables first
The engagement starts with a written scope that names which applications send traffic through the LiteLLM proxy, which cache types are in scope (exact-match, semantic or both) and which backend will hold the cache. Deliverables and timelines are agreed before work begins. Ask for the scope to state which requests are excluded from caching, such as personal data or time-sensitive prompts.
LiteLLM caching in your infrastructure
DeployOpen implements the cache in your own cloud account, data center or air-gapped network, configuring the LiteLLM proxy with a Redis, Valkey or Qdrant backend. Confirm which cloud regions, subnets and storage classes the cache will use, and whether the cache backend must sit inside the same boundary as the gateway.
Similarity threshold and data review
Threshold tuning is reviewed against sample prompts from your own traffic, so you can see where a semantic match returns an answer to a different question. The review also records what the cache stores about each prompt and response, how long entries live and who can read them. Ask for the review to produce a signed-off threshold and a written data-handling record.
Security testing of cache endpoints
Testing covers the proxy and the cache backend as deployed: network exposure, authentication on the cache service, TLS between components and access to cached content through the gateway. Qdrant, for example, ships without authentication enabled by default, so the test confirms that API keys, TLS and network rules were applied. Ask for findings to be tracked with severity and retest dates.
Documentation and handover
The handover pack describes the cache topology, the threshold settings, the key and access model, the cache eviction behavior and the steps to disable caching for a single application. It gives operators a runbook for clearing the cache after a prompt or model change. Ask for the documentation to be kept in your own repository so it stays with your environment.
Retainers for upgrades and testing
Optional retainers cover LiteLLM and cache backend upgrades, re-tuning of thresholds when models change and recurring security testing of the cache endpoints. Because LiteLLM and Redis publish frequent releases, ask how upgrades are tested against your cache configuration before rollout. Retainer scope and frequency are set in writing, separately from the initial fixed-scope work.
