Why tracing AI agents needs a different setup from single-call LLM tracing
A team whose agent makes dozens of model and tool calls for each user request will find that a flat log stream hides the failure. A single bad retrieval, a tool that returns an error the agent quietly retries, or a planning loop that repeats three times before answering all look like ordinary lines in a file. Self-hosted Langfuse fed by OpenTelemetry can record these runs as nested traces in the organization's own storage, with each model call, tool call and retry as a child span under the request that caused it. The setup for a single prompt and response does not carry over unchanged, because the unit of analysis is now the whole run rather than the single call.
Agent traces change three things. First, span volume multiplies, so storage and ingestion sizing must be based on spans per request rather than requests per day. Second, per-step cost attribution becomes the key requirement: finance and platform owners need to know which model, which tool and which retry consumed the tokens, not only the total for the session. Third, the team must decide how the agent's tool arguments and results are captured, since they often carry client data. Before go-live, confirm with your partner the span count for a representative request, the expected ClickHouse disk growth from that count, and which attributes will be masked before they leave the application.
Arize Phoenix is also a candidate for agent tracing, but its Elastic License 2.0 prohibits providing the software to third parties as a hosted or managed service. That restriction matters only if the trace platform is offered to customers, partners or other external users. For an internal agent platform run for the organization's own staff, the license question is narrower, but legal review should still confirm it. Langfuse's MIT-licensed core covers the agent tracing features described here, while certain enterprise controls, such as project-level roles and data retention policies, sit behind a license key.
Requirements agent tracing places on the build
This table lists the requirements that agent workloads add to a self-hosted Langfuse build, with the source for each and what it means for the implementation. Read each row as a design input: confirm it against your own retention, masking and capacity decisions before the scope is fixed. The main takeaway is that agent tracing raises the bar on span volume, retention and masking more than on initial setup.
| Requirement | Source | What it means for the build |
|---|---|---|
| Nested spans per agent step | OpenTelemetry is a vendor-neutral framework for generating traces, and Langfuse supports OpenTelemetry ingestion | Instrument each model call, tool call and retrieval step as a child span under one request trace, so a single agent run can be read end to end |
| Cost and latency per span | Langfuse tracks model usage and cost on generations; exact per-model pricing must be confirmed in your Langfuse version | Record model name and token usage on every generation span, so cost can be attributed to the step and tenant that caused it |
| Loop and tool-failure detection | Derived from trace structure, not a supplied product feature | Set alerts or evaluations on repeated tool spans and error status in a single run; confirm which alerting options your deployment exposes |
| Automatic event logs over the system lifetime | EU AI Act Article 12(1) requires high-risk AI systems to technically allow automatic recording of events over their lifetime | Tracing must run continuously in production, not only during debugging, and must capture the events your risk classification requires |
| Six-month minimum log retention | EU AI Act Article 19(1) requires providers to keep logs for at least six months unless other law provides otherwise | Set the retention period in the storage design and confirm with counsel whether longer periods apply under other law |
| Retention policies and server-side masking | Langfuse lists data retention policies and server-side data masking among features that need an Enterprise license key | Decide early whether retention and masking are enforced in Langfuse or client-side in the SDK or Collector, and budget for the license if the first option is chosen |
| Span-volume storage sizing | Langfuse scaling guide: at least 2 CPU and 8 GiB for ClickHouse, with at least 16 GiB memory for larger deployments | Agent runs produce many spans per request, so estimate spans per run times daily runs before sizing ClickHouse disk and memory; the Langfuse v3 to v4 guide advises about 3x disk headroom for backfills |
| Backup and restore testing | ClickHouse backup documentation: replication does not protect against human error, and untested backups may not restore | Schedule BACKUP to object storage and periodic restore tests on a spare cluster, because trace history is an evidence store |
Which platforms fit agent tracing and why
Agent tracing needs a backend that stores nested spans, a collection layer that receives them from every service, and a license that allows your team to run the software the way your governance requires. The three options below cover most self-hosted designs, and each carries a caveat that changes the architecture or budget.
Langfuse as the trace backend
Langfuse is the most common self-hosted choice for agent traces because its core features and APIs sit under the MIT license and self-hosted deployments run the same codebase as Langfuse Cloud. Agent runs fit its trace and observation model, which can hold nested steps, model calls and tool calls under one request. Check the enterprise boundary early: project-level RBAC, data retention policies, audit logs and server-side data masking need an Enterprise license key, so confirm which of these your controls depend on before you design access.
Arize Phoenix for evaluation-led teams
Phoenix is a tracing and evaluation tool that suits teams that want evaluation runs close to their traces during development and testing. Its Elastic License 2.0 prohibits providing the software to third parties as a hosted or managed service, so it works for internal use but not for a customer-facing tracing offering. Confirm the license scope with counsel if the same deployment will serve tenants, partners or an external product team, and compare its storage and access features with your retention requirements.
OpenTelemetry Collector as the entry point
The OpenTelemetry Collector receives, processes and exports telemetry without tying your applications to one backend, which makes it a practical front door for agent spans. Run it as an agent beside services for local buffering, or as a gateway that accepts traces from many services and forwards them to the trace backend. It is Apache-2.0 licensed. Confirm which components your pipeline needs, because core components have mixed stability levels and each component's README documents its status.
Gateway pattern for masking and routing
A gateway deployment of the Collector lets one team apply attribute filtering, redaction and routing before traces reach storage, so prompt content can be masked in one controlled place. This pattern matters for agent traces because tool arguments and retrieved documents often carry personal or client data. Ask who owns the pipeline configuration and how changes are reviewed, since a missed rule sends sensitive content into the trace store. Authenticator extensions can also protect the receiving endpoint with OIDC or add credentials to outgoing exporter requests.
Agent frameworks and instrumentation path
The right instrumentation path depends on which agent framework, SDK and model provider your teams use, and that combination is different in every stack. Some frameworks emit OpenTelemetry spans directly, others need a wrapper or a gateway that logs model calls, and tool calls often need manual spans. Confirm per stack which framework versions are supported, how parent-child relationships are preserved across async tool calls, and whether token usage is captured for cost attribution.
ClickHouse as the storage layer
Agent traces with hundreds of spans per request grow quickly, and Langfuse stores its high-volume trace data in ClickHouse alongside PostgreSQL, Redis or Valkey and S3-compatible blob storage. The ClickHouse server is Apache-2.0 licensed, so the storage tier can stay inside your own infrastructure. Plan the sizing and backup work before launch: size ClickHouse memory and disk for your expected span volume, and schedule restore tests, because a backup never test-restored will probably not restore when needed.
How a tracing rollout for one agent proceeds
A tracing rollout for one agent moves from defining what the agent does to proving that traces can be stored, searched and restored in the organization's own infrastructure. Each phase closes with a concrete output that the next phase depends on, so problems surface before the observability platform goes into production use.
Scope the agent and its step types
The team lists the agent's entry points, model calls, tool calls, retrieval steps and handoffs, and records which of them carry personal or client data. DeployOpen works with the application owner and the security lead to agree which prompts, tool arguments and outputs may be stored, in which project, and who may view them. The step closes with a written span catalogue: one row per step type, with the attributes to record, the masking decision and the owner of each decision.
Instrument one agent path with nested spans
Engineers instrument a single representative request path first, using the Langfuse SDK or OpenTelemetry instrumentation so that the agent run is the root trace and each model call, tool call and retrieval becomes a child span. Cost and token attributes are attached to model spans, and masking runs in the SDK or the OpenTelemetry Collector before data leaves the application. The step is done when one end-to-end run shows a readable tree, per-step latency, and a cost figure on every model span.
Deploy self-hosted Langfuse for production
DeployOpen deploys Langfuse Web, Langfuse Worker, PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible blob storage in the client's own cloud account or data center. Docker Compose on a single VM is used only for a test environment, because the Langfuse documentation describes it as having no high availability. Production runs on Kubernetes with Helm or Terraform. The step closes when the instance accepts traces from the instrumented path, and the team has confirmed which features it uses and whether any of them require an enterprise license key.
Size ClickHouse and worker scaling
Using the Langfuse scaling guide as the starting point, the team sizes ClickHouse, the worker containers and the web tier against the expected span volume. Worker containers are scaled on CPU load, and under heavy ingestion the web tier can be split into separate ingestion and user interface deployments. The output is a sizing note that records the expected spans per request, requests per day, retention period and the ClickHouse disk headroom needed for backfills, which the v3 to v4 guide advises at about 3x.
Run a load test with representative span counts
Engineers replay synthetic or anonymized traffic that matches the production shape: many spans per request, long agent loops and a realistic share of failing tool calls. They measure ingestion lag, worker queue depth, ClickHouse memory and query latency in the trace explorer at the expected concurrency. The step closes with a written result that either confirms the sizing note or changes it, and with a list of the limits that were reached, so the production capacity is based on measured behavior rather than estimates.
Add loop and tool-failure dashboards
The team builds dashboards and alerts for the failure modes that matter in agent systems: repeated tool calls within one trace, step counts above an agreed ceiling, cost per request, and tool errors grouped by tool name and error class. Alerts are routed to the owners named in the span catalogue. The step is complete when a deliberately induced loop and a failing tool each appear on the dashboards and trigger the agreed alert, which confirms that the signals are being recorded end to end.
Test backup and restore before go-live
DeployOpen configures backups for the ClickHouse data and the PostgreSQL and object storage dependencies, then performs a restore onto a spare environment. ClickHouse BACKUP and RESTORE documentation notes that replication alone does not protect against human error or corruption, and warns that an untested backup will probably not restore when needed. The step closes with a successful restore report, a recorded recovery time observed in the test, and a schedule for repeating the restore test, handed to the client's operations team.
How hundreds of spans per request change storage and cost
A single agent request can expand into many observations: one parent trace, a model call for each planning or reasoning turn, a tool call for each action, retrieval steps, and retries. Each of these becomes a span or observation that Langfuse must ingest, store and query. The span count per request is the number that drives storage, so measure it before sizing anything. Run a representative set of agent tasks in a pre-production environment, count observations per trace, and multiply by your expected request volume and retention period. Treat that figure as the input to every sizing decision below.
In a self-hosted Langfuse deployment, that span volume lands in several places. Langfuse Worker processes events asynchronously, so worker CPU rises with ingestion. ClickHouse holds the observation data and needs both memory for queries and disk for storage, and the scaling guide lists at least 16 GiB of ClickHouse memory for larger deployments. Prompts and large payloads can also go to S3-compatible blob storage, which grows independently. When you plan a migration or backfill of historic data, the v3 to v4 upgrade guide advises about 3x ClickHouse disk headroom, so a backfill of a large trace history needs far more free disk than the steady-state data set. Ask your partner to show these three growth curves separately in the sizing model, and confirm the headroom figure against the current guide before the upgrade window is set.
Cost attribution per step depends on what the application writes into each span. Langfuse can record model name, token usage and cost when the model call is instrumented correctly, but step-level attribution needs consistent tags. Set stable identifiers for the agent name, step type, tool name, tenant or business unit, and a request or session ID that links child spans to their parent. Record latency at span level so slow tools and repeated model turns stand out. Use OpenTelemetry attributes consistently across services so that OpenTelemetry Collector pipelines can route and filter data before it reaches storage. Define the tag schema in writing before development starts, because adding tags later leaves historic traces without the attributes you need for cost reports.
Spotting loops, tool failures and runaway cost
A loop shows up in a trace as the same tool or model call recurring under one parent span, often with near-identical inputs. Because an agent decides its own next step, a loop may not produce an error at all. The request simply keeps going until a limit stops it, so the trace is the only place the pattern is visible. Each repeated span should carry the step index, tool name, arguments hash and the agent's stated reason for the call, so reviewers can tell a retry from genuine progress.
- What the trace shows: a parent agent span with many child tool spans of the same name, and a rising count of model calls before the final answer.
- What to alert on: the number of spans of one tool type within a single trace, measured against a limit your team sets after reviewing normal runs. Do not treat any threshold in this page as a recommended default.
- What must be instrumented first: a stable trace ID per user request, a span per model call and per tool call, and the agent step index as an attribute.
- Loop guards to confirm: whether your agent framework enforces a maximum step count, and whether that limit is logged when it fires.
- Repeated arguments: an identical hash across spans is a stronger loop signal than call count alone.
Confirm with the framework vendor how it reports a terminated run, so a stopped loop appears as a failed trace rather than a short successful one.
Tool failures are the most common cause of degraded agent answers, and they are easy to miss because the model often recovers by apologizing or guessing. A trace that ends in a confident answer may have skipped a failed lookup entirely. Recording tool status, error type, latency and the response size for each tool call lets operators separate an upstream outage from a bad prompt that sent malformed arguments.
- What the trace shows: a tool span with an error status or an exception event, followed by either a retry span or a model span that answers without the tool result.
- What to alert on: error rate per tool over a time window, and timeouts per tool compared with that tool's normal latency. Set these thresholds from your own baseline.
- What must be instrumented first: status codes and exception messages on every tool span, plus the downstream system name as an attribute so failures group by dependency.
- Sensitive payloads: tool responses often contain customer or employee data, so confirm masking rules before storing full responses.
- Silent fallbacks: check whether the agent catches exceptions and continues, because those runs will look successful unless the error is recorded on the span.
Ask your partner to test a deliberately failing tool in a non-production environment and confirm that the error appears in the trace and in the alert.
Agents multiply model spend because each request can contain many calls, and the expensive step is often not the one users notice. Cost attribution works only when token counts and model names are recorded on each generation, and when the request carries tenant, user or feature identifiers from the start. Without those tags, a spike shows as a total with no owner.
- What the trace shows: cost and token usage on each model span, summed up to the trace and to the tenant or feature tag.
- What to alert on: cost per trace, cost per tenant per day, and a sudden change in tokens per step. Set limits from observed usage, and confirm how your pricing data for each model is maintained.
- What must be instrumented first: model name, input and output token counts, and tenant or feature metadata on every generation span.
- Prompt growth: a tool result that is appended to the context on every step can grow input tokens quickly, so track input size per step, not only the total.
- Pricing source: confirm whether Langfuse computes cost from token counts using its model definitions, or whether your team supplies prices for private or custom models.
Illustrative case: a retry loop that doubles context size each step would look modest in request count but expensive in total tokens, which is why per-step input size matters.
Every alert in the other tabs depends on the same foundation: consistent spans, attributes and identifiers emitted by each step of the agent. Teams that alert before instrumenting usually find that their dashboards cannot group failures by tool, or that cost cannot be linked to a tenant. Standardizing the attribute names across services before building alerts avoids rework later.
- Span hierarchy: one root trace per user request, with agent, model and tool spans nested beneath it using OpenTelemetry context propagation.
- Attribute naming: agree on names for tool, step index, tenant, feature and model, and apply them in every service that the agent calls.
- Collector routing: the OpenTelemetry Collector can add or remove attributes before export, which is a useful place to enforce naming and masking.
- Test traffic: run scripted loops, failures and high-token prompts in staging to confirm each signal appears.
- Ownership: name who receives each alert and who reviews changes to thresholds.
Before a rollout, ask your implementation partner to show a sample trace for each of the three signals, produced from your own applications.
Evidence auditors ask for from agent traces
This table lists the evidence an assessor is likely to request for a self-hosted agent tracing deployment, the component that produces each item, and what to confirm before the request arrives. Read it as a checklist to build before the assessment, not after. DeployOpen prepares this evidence inside your environment but does not issue audit reports, attestations or certifications; independent auditors and certification bodies do.
| Evidence | Produced by | What to confirm |
|---|---|---|
| Automatically generated event logs for agent runs | Langfuse (nested traces and observations), with OpenTelemetry SDK instrumentation and the OpenTelemetry Collector as the export path | Every model call, tool call and retrieval step in a request is captured as a child span, and the trace IDs link back to the originating request |
| Retention configuration and log lifetime | Langfuse data retention policy (an Enterprise license key feature) and ClickHouse table TTL settings | The retention period set for traces meets the record-keeping duties that apply to your system, and deletion is recorded |
| Access logs and audit trail for trace viewers | Langfuse audit logs (Enterprise license key feature) and ClickHouse query logs | Who viewed or exported traces containing client or employee data, and whether the audit log is retained outside the trace store |
| RBAC and project role settings | Langfuse project-level roles (Enterprise license key feature) and ClickHouse access control with external authenticators | Role assignments are exported on a schedule, and each engineer's access traces back to an approved identity source |
| Masking configuration for prompts and outputs | Client-side masking in the SDK or OpenTelemetry Collector processors, plus Langfuse server-side masking (Enterprise license key feature) | Test traces with sample personal data show masked values in the stored records, and the masking rules are under change control |
| Backup and restore test records | ClickHouse BACKUP and RESTORE logs and the restore test reports your team keeps | Backups cover ClickHouse and PostgreSQL, and a restore has been run on a spare cluster, since an untested backup may not restore |
| Replication and Keeper health records | ClickHouse Keeper and replica status, monitored by the platform team | Keeper runs with at least three nodes in production, and replica recovery events are retained as evidence of availability |
| Version upgrade and migration records | Change tickets, the Langfuse upgrade guides and the ClickHouse changelog reviewed before each upgrade | Each upgrade records the version pulled, the migration output, the pre-production test results and the rollback plan |
| Collector configuration and change history | OpenTelemetry Collector configuration files stored in version control, with signed release artifacts and SBOMs from each Collector release | Every change to processors, exporters and authenticator extensions has an approver and a date, and the Collector build matches a published release |
What do buyers ask about agent tracing before hiring help?
How much effort does instrumenting an AI agent take?
Effort depends mostly on how many frameworks, model providers and tool integrations the agent uses, and on whether each step already emits structured telemetry. An agent that calls a model, several tools and a retrieval step per request needs spans for each of those calls, plus parent-child links so a single user request appears as one nested trace. Teams that already use OpenTelemetry in other services usually reuse their conventions, while teams starting from scratch must also decide what to capture for prompts, tool arguments and token counts. Ask the partner to list every framework and call path to be instrumented before agreeing a fixed scope.
What infrastructure does self-hosted Langfuse with ClickHouse need?
A self-hosted Langfuse deployment runs two application containers, Langfuse Web and Langfuse Worker, alongside PostgreSQL, ClickHouse, Redis or Valkey, and S3-compatible blob storage. Langfuse's scaling guide lists minimum resources of 2 CPU and 4 GiB for web, worker and PostgreSQL, 1 CPU and 1.5 GiB for Redis or Valkey, and 2 CPU and 8 GiB for ClickHouse, with at least 16 GiB of ClickHouse memory for larger deployments. Docker Compose on a single VM is documented for testing without high availability, while production setups use Kubernetes with Helm or Terraform. Ask for a sizing based on your expected trace volume rather than the minimums.
Can our team run agent tracing without outside help?
A capable platform team can run Langfuse, but the ongoing work is broader than installation: upgrades, ClickHouse replication and Keeper nodes, backups, restore tests and access control all need owners. ClickHouse documentation warns that a backup never test-restored will probably not restore properly, so restore testing is a recurring duty, not a one-time task. Ask whether your team has someone who can own ClickHouse operations and whether you want a retainer for upgrades and support. Langfuse's Self-Hosted Enterprise plan adds a named support engineer and a support SLA for those who need contractual help.
Which Langfuse features need an enterprise license key?
The core features and APIs are available in the MIT-licensed open source version, but certain controls require an Enterprise license key. These include project-level RBAC roles, protected prompt labels, data retention policies, audit logs, server-side data masking, UI customization, the org management API with SCIM, and the instance management API. Anything that is a compliance control for personal or regulated data, such as retention or masking, should be confirmed against the current license key documentation before the design is fixed. Also confirm that the ee directories are excluded from any code you distribute.
How does span volume affect storage and cost?
Storage grows with the number of spans, and an agent that makes many model and tool calls per request produces far more spans than a single chat completion. Each span carries attributes such as token counts, inputs and outputs, so prompt length and payload size also matter. ClickHouse needs disk headroom beyond the stored data, and Langfuse's v3 to v4 guide advises about 3x ClickHouse disk headroom when backfilling historic data. Ask the partner to model spans per request, retention period and payload capture, then size ClickHouse disk and blob storage from those numbers.
How is the work scoped and quoted?
DeployOpen works from a written, fixed scope that lists the agents, frameworks and tool integrations to instrument, the environments to deploy into, the masking and access rules, and the deliverables and timeline. The quote follows from that scope, and DeployOpen does not publish a price list. Costs have two sides: the implementation work, which is one-time, and the infrastructure, licenses and operations, which recur. Ask for the quote to separate these, to name any Langfuse enterprise license separately, and to state what is excluded, such as evaluation design or upgrades after go-live.
What to ask an implementation partner about agent tracing
Start with instrumentation, because it determines whether an agent trace is useful at all. Ask the partner how each model call, tool call, retrieval step and sub-agent handoff is wrapped in a span, and which parent-child structure they will produce over OpenTelemetry. Ask them to show a sample trace from a multi-step agent run in your environment, including how a retried tool call and a loop are visible. Ask which instrumentation libraries they will use for your agent framework, and how they will confirm that token usage and model names are captured on every generation so that cost per step can be calculated.
Next, ask how the storage will be sized and protected. Request a written sizing model that starts from your number of requests, average steps per request and payload size, and ask how that model maps to ClickHouse disk, memory and object storage for large payloads. Ask how they will test restores: the ClickHouse backup documentation warns that a backup never test-restored will probably not restore properly, so ask for the schedule, the spare cluster and the evidence they will hand over. Confirm that they plan for ClickHouse Keeper with at least three nodes if you need replication, and ask which ClickHouse release line they will pin and how it will be patched.
Finally, ask about masking, upgrades and licensing. Ask where prompts and tool outputs are masked, in the SDK, in the OpenTelemetry Collector or in Langfuse, and how they will prove that secrets and personal data never reach storage. Ask how they will handle Langfuse upgrades, including the v3 to v4 path, which requires a ClickHouse version and a backfill with extra disk headroom, and whether they will rehearse the migration on representative data first. Ask them to list every feature they expect you to license, such as project RBAC, data retention policies, audit logs and server-side masking, and to explain which of those you need before go-live.
How DeployOpen scopes and supports agent tracing
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, so agent traces, prompts and cost records stay in storage you control. For agent tracing, the work starts with a written scope that names the agents, tools and storage in play before any instrumentation begins.
How we scope
- Inventory every agent, model provider, tool and framework that will emit traces, with an estimate of spans per request agreed with your team
- Define which spans carry prompts, tool arguments and outputs, and which fields must be masked before storage
- Agree the ClickHouse and object storage sizing basis, retention periods and the Langfuse edition needed for your roles and controls
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts
How we build
- Deploy Langfuse Web, Langfuse Worker, PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible storage in your cloud account or data center
- Instrument agent steps with OpenTelemetry, routing traces through an OpenTelemetry Collector where masking and authentication are needed
- Configure nested trace hierarchies so each model call, tool call and retry shows its own cost, latency and error state
- Run security testing on the trace pipeline, including access paths to prompts and outputs stored in traces
How we hand over
- Document the instrumentation pattern, span naming conventions and the trace fields your teams are expected to populate
- Deliver a tested ClickHouse backup and restore runbook, with a restore test on a spare environment before handover
- Record the upgrade path between Langfuse and ClickHouse versions, including the migration steps for major releases
- Provide evidence packs that show how logging, retention and access controls are configured for independent assessors
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover
- Review trace volume and ClickHouse disk usage as agent workloads grow, and adjust worker scaling accordingly
- Re-test masking rules and access roles when agents, tools or model providers change
- Quotes follow from the written scope, so ask for the component list, storage assumptions and support hours to be itemized
