Langfuse and LiteLLM proxy integration

Each LLM call follows one path. The application sends an OpenAI-format request to the LiteLLM proxy instead of directly to the model provider. The proxy translates that request for the upstream provider it selects, such as OpenAI, Anthropic, Bedrock or Azure, and returns the response to the application. When the call completes, the langfuse_otel callback enabled in litellm_config.yaml builds the telemetry for that call and sends it to the Langfuse OTLP endpoint at /api/public/otel on the host set by LANGFUSE_OTEL_HOST. The application code does not change, which is the main reason to place the gateway in this position. Before rollout, confirm that the proxy host can reach the Langfuse host on the network path your security team has approved, and that the callback is active on the proxy version you run.

Two Langfuse containers handle what arrives at that endpoint. Langfuse Web serves the UI and APIs, and it is the component that receives the OTLP request from the proxy. Langfuse Worker performs the asynchronous event processing that turns received events into stored trace data. Because processing is asynchronous, a trace can appear in the UI after a short delay, so teams should not treat the UI as a live view during testing. The Langfuse self-hosting documentation also lists PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible blob storage as parts of the deployment, and the vendor documentation should be consulted to confirm which of these each stage uses in your version. Plan Worker capacity against CPU load, following the scaling guide, and watch Web and Worker logs during the first load test to see whether ingestion keeps pace with the proxy's request volume.

The vendor documentation states that the LiteLLM Proxy integration captures token usage, cost and latency for every LLM call routed through the proxy. Those three fields are the core of the chargeback and performance reporting most teams want, so they should be checked on real traffic rather than assumed. Cost needs particular attention. Langfuse calculates cost from the model definitions it holds, so models that are not in the Langfuse model list may show no cost or an incorrect one. Ask the vendor how cost is calculated for models outside that list, whether custom model prices can be defined in your self-hosted instance, and who maintains those definitions when provider prices change. LiteLLM also tracks spend in its own database through virtual key budgets, so finance teams should decide which figure is authoritative and reconcile the two during the pilot.

A short test plan shows whether the path works end to end. Send requests through each provider and model family the proxy exposes, including a failed request, and confirm that each one produces a trace with the expected model name, token counts, cost and latency. Check that the model name recorded in Langfuse matches the deployment the router actually used, because routing and failover can send a request to a different deployment than the one the application named. Confirm that the user, team or key metadata you need for reporting arrives on the trace. Record the delay between a request and its trace appearing in the UI so operations teams know what to expect. Document the results and the cost-model decisions, since these are the records a security or finance reviewer will ask to see first.

Langfuse documents three ways to get LiteLLM traffic into a self-hosted Langfuse: sending logs from the LiteLLM Proxy, using the LiteLLM SDK, or using a compatible client SDK such as the Langfuse OpenAI SDK wrapper. The proxy callback is the only one of the three that is configured once, centrally, and needs no change to application code. You set the Langfuse credentials as LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY, point LANGFUSE_OTEL_HOST at your self-hosted OTEL endpoint, and enable the langfuse_otel callback in litellm_config.yaml. Every LLM call routed through the gateway then carries token usage, cost and latency into Langfuse. For a platform team that wants organization-wide capture, this is the default choice.

The LiteLLM SDK and the client wrapper both work per application. The LiteLLM SDK is added to the Python code path that calls models, so each service must be changed and redeployed. A compatible client wrapper, such as the Langfuse OpenAI SDK wrapper, replaces the client import in the application and adds the same tracing at that point. Both give you finer control over trace names, user identifiers, session IDs and metadata, which the gateway cannot always infer. The cost is ownership: every team that adopts these methods has to keep its instrumentation current when the SDKs change, and a missed service leaves a gap in your telemetry.

The proxy callback only sees calls that pass through the gateway. Before you rely on it for organization-wide reporting, inventory where model calls actually originate. Teams that call providers directly with their own API keys, batch jobs that bypass the proxy, and notebooks used by data scientists are common sources of untraced usage. For each of these, confirm with the owning team whether the call can be routed through the gateway. If it cannot, plan one of the per-application methods for that path and record it as an exception, so that totals in Langfuse can be reconciled against provider invoices.

A practical split for most organizations is to enable the proxy callback for all gateway traffic, then add the LiteLLM SDK or a client wrapper only where the proxy cannot supply the context you need, such as a per-customer or per-feature attribute. Check that the proxy and any per-application method do not both report the same call, which would double-count usage and cost. Test this with a known request in a non-production Langfuse project before you roll out, and compare the trace count and token totals against the provider's own usage export for the same window.

Confirm the exact configuration keys, callback names and supported client wrapper versions against the Langfuse LiteLLM integration documentation and the LiteLLM release you run, since both projects change their interfaces between releases. Ask the owners of each application which integration method they will use, who maintains it, and how it is tested after upgrades. Recording those answers in a single table, with one row per gateway and per direct-call path, gives you the scope that the rest of the implementation and its acceptance checks depend on.

Scale the LiteLLM proxy and the self-hosted Langfuse as one pipeline. A trace is only complete if the proxy accepts each request, the OTEL export reaches Langfuse, and Langfuse processes what arrives. LiteLLM's production guide recommends one Uvicorn worker per pod on Kubernetes and scaling horizontally by adding pods. Its suggested starting point is 1 vCPU and 4 GiB per worker. Treat those figures as a starting point only, then load test with your own mix of streaming calls, long prompts and fallback routes, because memory use varies with request shape. Record the request rate at which latency starts to climb, since that number sets your replica count.

Redis 7.0 or newer is required once more than one proxy instance runs. Shared rate limits, router state and cache depend on it. Without shared state, per-key TPM and RPM limits, cooldowns after repeated failures and retry behavior are enforced separately on each replica, so a key can exceed its intended limit. Confirm with your platform team the Redis version, high availability design and failover behavior before the second replica goes live. Treat Redis as a gateway dependency with its own monitoring and alerts, and check that the proxy recovers cleanly when it restarts while Redis is briefly unavailable.

On the Langfuse side, Langfuse recommends scaling the worker containers on CPU load. Under heavy ingestion, it suggests splitting langfuse-web into separate ingestion and user interface deployments, so dashboard queries do not compete with incoming OTEL traffic. The scaling guide lists minimum resources of 2 CPU and 4 GiB for web, worker and PostgreSQL, 1 CPU and 1.5 GiB for Redis or Valkey, and 2 CPU and 8 GiB for ClickHouse, with at least 16 GiB of ClickHouse memory for larger deployments. These are floors, not sizing for your volume. Ask your team or partner to size ClickHouse and the workers from expected trace volume and retention.

LiteLLM batches spend updates to its database every 60 seconds at scale, so budget and spend views in the proxy can trail live traffic by roughly that interval. Langfuse receives its own trace stream, so the two views may disagree briefly. Plan budget headroom for each key and team so a short lag does not hide a real overrun, and set alerts on the Langfuse cost and token dashboards as well as the proxy budgets. During load testing, compare the number of requests the proxy reports with the number of traces Langfuse stores, and confirm that retries and failed calls appear as you expect before you rely on the pipeline for chargeback.