LiteLLM vs Envoy AI Gateway (now Agent Router)

Agent Router's two-tier pattern splits gateway duties by who needs them. The tier one gateway handles authentication, top-level routing and global rate limiting, so every application call enters through one controlled point. The tier two gateway sits in front of self-hosted model clusters and provides fine-grained access to them, with endpoint picker support for inference optimization. The README describes this split, but it does not publish a configuration walkthrough, so your platform team should map each tier to specific AIGatewayRoute and AIServiceBackend resources in a test cluster before committing to the layout.

LiteLLM covers the same ground with a different shape. Inside one proxy, the router offers least-busy and latency-based strategies alongside simple-shuffle and usage-based routing, plus cooldowns after repeated failures and priority ordering for failover. Those settings steer traffic across deployments of the same model, which handles many cluster needs. What LiteLLM does not present as a separate tier is a second gateway that owns access to the cluster itself. If your clusters need a distinct team boundary, such as a platform group that owns the inference fleet, you will have to build that separation with network policy, separate proxy instances or separate keys, and you should confirm that design with your own engineers.

Confirm endpoint picker behavior in your own environment before relying on it. Ask how it reads inference load signals from your serving engine, how it handles a pod that is busy but healthy, and what happens to in-flight requests during scale-down. Run a load test that mirrors your prompt sizes and concurrency. Those results, not the README, should decide whether the tier two layer earns its place.

The LiteLLM side of this decision is mostly a stateful service to operate. LiteLLM's production guide names PostgreSQL as the primary supported database, and virtual keys need PostgreSQL, so the first sizing task is the database itself: connection counts, backups and restore testing. Once more than one proxy instance runs, Redis 7.0 or newer becomes required for shared rate limits, router state and cache, so plan that as a second stateful dependency. The guide suggests 1 vCPU and 4 GiB of memory per worker as a starting point, and recommends one Uvicorn worker per pod on Kubernetes, which means your scaling numbers come from load tests in your own environment. Secret handling is the next gate: LITELLM_MASTER_KEY must start with sk- and belongs in a secret manager. SSO setup for Google, Microsoft Entra ID, Okta or generic OIDC is the step most teams underestimate, and the free tier covers only up to 5 users from v1.76.0, so confirm your user count against that limit before the identity design is fixed.

The Agent Router side runs through Kubernetes. A team needs a cluster that meets its own availability needs, Envoy Gateway installed, and then the custom resources that define the platform: AIGatewayRoute, AIServiceBackend and BackendSecurityPolicy, all under the aigateway.envoyproxy.io API group. Most of the time goes into authoring those resources and designing policy around them, including how credentials are stored, how the two tiers divide authentication, global rate limits and fine-grained access to self-hosted model clusters. Cluster operations add their own work: an external etcd cluster or a stacked control plane with at least three control plane nodes, etcd backups, and Kubernetes upgrades on a schedule of roughly three minor releases per year. Confirm who owns that cluster before committing. The delays that recur across both options are rarely technical in isolation. Identity mapping is the usual first stall, because application users, team admins and platform operators need roles defined in both the identity provider and the gateway. Quota design across teams follows, since budgets, TPM and RPM limits, or Envoy-enforced rate limits all require agreement on who owns which allocation. For Agent Router, etcd and cluster operations can absorb a platform team's sprint capacity before any AI traffic flows. Ask any implementation partner, including DeployOpen, for the written scope, deliverables and timeline before work starts, since most engagements are fixed-scope and that document is where these risks should be assigned.

DeployOpen plans, implements, hardens, backs up and documents self-hosted open-source platforms inside your own infrastructure, including your cloud accounts, data centers or air-gapped networks. For a gateway decision, that means the team can work from your requirements rather than from a preferred product. If the answer is LiteLLM as a shared proxy with virtual keys in your own PostgreSQL, or an Agent Router deployment on Kubernetes with Envoy Gateway, the engagement covers the same disciplines either way: identity wiring, secret handling, network placement, upgrade planning and runbooks. DeployOpen is not a vendor partner, reseller or certified provider of LiteLLM, Agent Router, Envoy or Kubernetes, and says so up front so the recommendation is judged on fit.

Most engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. For this decision, a useful written scope names the teams that will call the gateway, the providers and self-hosted model clusters in use, whether MCP servers must be routed, the rate limit and quota model you need, and who owns the Kubernetes cluster and its etcd backups. Ask for the scope to state which items are confirmed against the current LiteLLM and Agent Router documentation and which are assumptions to test in your environment. Ongoing support, upgrades and continuous testing are available as retainers, and that recurring part should be priced separately from the one-time build.

DeployOpen does not publish a price list. A quote follows from the written scope, so the inputs that move cost are the number of clusters and environments, the identity and secret manager integrations you require, whether you need licensed LiteLLM Enterprise features such as SCIM or audit logs, and how many model backends must be validated. Ask any provider for the quote to separate one-time implementation from recurring operations, and to list which upgrades, Redis or PostgreSQL components and testing are included. Treat any figure not written into the agreed scope as something to confirm before signing.