Dify Enterprise telemetry export with OpenTelemetry: what it answers and when to use it
OpenTelemetry export sends Dify Enterprise metrics and traces to a collector you run, which gives a platform team the raw signals it needs for cost allocation and service level objectives across workspaces. The situation this answers is familiar: several workspaces are building workflows and agents, model spend is rising, and nobody can say which team drives latency, which provider is failing, or which application is burning tokens. A shared telemetry stream lets you attribute usage and measure reliability in one place, instead of comparing per-app log screens by hand. Before committing to this design, confirm in the Dify Enterprise documentation which metrics and traces the product emits, which labels or attributes identify workspace, app and model, how export is enabled, and whether it is configured through the admin console, Helm values or environment variables.
OpenTelemetry itself is an observability framework and toolkit for generating, exporting and collecting telemetry, and it is explicitly not an observability backend. (What is OpenTelemetry explains the split.) That means you still choose where metrics and traces are stored, queried and alerted on, and that choice carries its own licensing, retention and staffing cost. Your collector can be deployed as an agent next to workloads or as a gateway that receives data from many sources, and the collector configuration needs authentication on receivers and exporters, which the collector's authenticator extensions support. Ask the Dify Enterprise team whether export can be authenticated and how credentials are supplied, because an unauthenticated endpoint exposes usage data about every workspace.
Treat structured event logs as a separate pipeline with its own retention and access rules, not as a side effect of metric export. Confirm which event fields Dify Enterprise makes available, how often they are emitted, and whether their volume and personal data content fit your storage and privacy obligations. Decide early who owns the collector, who can read cost and latency data per workspace, and how these signals will feed SLO dashboards. A pilot against one staging workspace will show whether the attributes you need are present before any production dashboard depends on them.
Export options and when each fits
This table compares collector placement, signal type and authentication choices for exporting Dify Enterprise telemetry to your own OpenTelemetry Collector. Read each row against the goal it serves, whether that is cost allocation, SLOs or audit evidence, and confirm the listed items against your Dify Enterprise version before you commit. The main takeaway is that metrics and traces answer cost and reliability questions, while structured event logs usually need their own pipeline to meet audit needs.
| Option | Best fit | Trade-off | What to confirm in your Dify Enterprise version |
|---|---|---|---|
| Per-node collector agent | Node-level host and pod metrics next to Dify workers and the sandbox | More collector instances to configure, upgrade and monitor | Which Dify services emit OTLP natively and which need scraping or host-level collection |
| Central collector gateway | Cost allocation and SLO dashboards built from one tenant-aware pipeline | Single point of failure unless run highly available, and a larger blast radius for misconfiguration | Whether telemetry includes workspace or app identifiers you can use as attributes for cost splits |
| Metrics export | Token usage, request rates, latency and error budgets for SLOs | Cardinality grows fast if workspace or user labels are added without limits | Metric names, units and labels exposed in your release, and whether token counts are included |
| Trace export | Diagnosing slow workflow nodes, retrieval steps and model calls | Higher volume, so sampling and retention need a decision before rollout | Span names and attributes per node type, and whether prompts or payloads are captured by default |
| Structured event logs | Audit trails of who ran which app, from where and with what outcome | Logs are not the same signal as OpenTelemetry metrics or traces, so they usually need a separate pipeline | Which invocation and admin events are exposed as structured records, and the export method your version supports |
| Push from Dify to collector | Simple setup when the collector accepts unauthenticated traffic only on a private network | Credentials for exporters must be managed in Dify configuration and rotated there | Which export settings are available in the Enterprise configuration and whether custom headers can be set |
| Collector-side authentication | Requests reaching the collector are checked with OIDC or credentials added on outbound exports | Adds extension configuration and token lifecycle work to the collector | Whether the collector distribution in your deployment includes the authenticator extensions you plan to use |
Metrics, traces and event logs need different designs
Metrics are aggregated numbers sampled over time, so they suit dashboards and alerting rather than investigating a single request. For Dify Enterprise, the questions usually are how many runs each workspace or application makes, how much model capacity is consumed, and whether service levels such as response time or error rate hold over a month. Those answers depend on consistent labels, so the design work comes before the first dashboard.
- Cost allocation labels: Decide which dimensions must exist on every series, such as workspace, environment and application, before any chargeback report depends on them.
- Cardinality limits: Avoid per-user or per-conversation labels on high-volume series, since they inflate storage and slow queries.
- SLO definitions: Write each objective as a good-event ratio over a fixed window, and confirm which signal provides the numerator and denominator.
- Aggregation point: Decide whether counters are aggregated in the Collector or in the backend, so that restarts and retries do not distort totals.
- Metric names and units: Agree on a naming convention with the platform team so that dashboards survive a Dify upgrade.
Before relying on any metric for billing or SLO reporting, confirm with the vendor which metrics the Enterprise build emits and their exact names and units.
Traces follow one request through its spans, which makes them the right tool for finding where time is spent and where a failure starts. In an LLM application the interesting spans often sit around model calls, retrieval, tool calls and sandboxed code execution. A trace shows whether a slow answer came from the model provider, the vector store or an external tool, and that distinction changes who must act.
- Failure paths: Trace errors and retries as distinct spans so that an intermittent provider timeout is visible without reading application logs.
- Sampling strategy: Decide between head and tail sampling. Tail sampling keeps slow or failed traces, but needs more Collector capacity.
- Attribute hygiene: Prompts, documents and personal data should not become span attributes by default, since traces are often retained longer than teams expect.
- Context propagation: Confirm which hops pass trace context, including the proxies and sandboxes between the Dify services.
- Retention: Keep trace retention short unless an investigation requires longer, and document the exception.
Ask the vendor which spans and attributes the Enterprise services emit before designing sampling rules, because the design depends on what actually exists.
Event logs record discrete facts, such as who ran an application, when a workspace changed, or which credential was used. They serve audit and investigation, and their requirements differ from metrics and traces. An event record may need to be kept unchanged for a defined period, searched by a security team and exported to an evidence store, so it often warrants its own pipeline rather than sharing the telemetry path.
- OpenTelemetry log maturity: Log SDK maturity varies by language, and several languages are still in development, so check the status of each runtime before depending on OTLP logs.
- Separate pipeline: Route audit events through a distinct Collector pipeline, exporter and backend so that a telemetry outage or sampling change cannot silently drop them.
- Retention and immutability: Set retention from the organization's audit and records obligations, and confirm who can alter or delete stored events.
- Access control: Limit who can read event logs, since they can reveal user activity and operational detail.
- Completeness checks: Compare event counts against the invocation records in the Dify console to detect gaps.
Confirm with the vendor which audit events the Enterprise edition produces and in what format before writing retention or export requirements.
How to set up the export pipeline
This sequence takes Dify Enterprise telemetry from an unverified idea to a production pipeline your team owns. Each step ends with a check, so label mapping, credential handling and content exposure are confirmed before any data reaches a shared backend. The result is a pinned, version-controlled Collector configuration and a documented signal register.
Inventory the signals Dify Enterprise exposes
Start by listing every metric, trace and log stream that the Dify Enterprise release you run can emit. Confirm the names, units and resource attributes with Dify Enterprise documentation or your vendor contact, because the exposed set can change between releases. For each signal, record whether workspace or app identifiers appear as labels, and flag any attribute that could carry prompt text, model output or retrieved document content. Platform, security and cost owners sign the register together. Done when every attribute in a sample has a documented purpose and each content-bearing field has a removal or hashing decision.
Deploy a Collector in a test namespace
The platform team installs the OpenTelemetry Collector in an isolated test namespace. Use the contrib distribution from the Collector releases page, pin the version, and verify the signed artifact, SHA256 checksum and SBOM before deployment. Choose agent mode for a single cluster, or gateway mode when several clusters or sources send data to one point. Leave the test namespace without any exporter to production backends for now. Done when the Collector pod is healthy, its own internal telemetry is scraped, and outbound traffic is limited to endpoints you have approved.
Configure receivers, processors and exporters
Define a receiver for the signal path you confirmed in the first step, typically OTLP, and set any scrape or push configuration that Dify Enterprise documentation requires. Add processors that drop or hash content-bearing attributes, along with a memory limiter and batching. Point the exporters at a test backend. Keep the configuration file in version control so every change has a reviewer. Done when a sample sent through the debug exporter shows the workspace label on each record and contains no prompt, completion or document text.
Add authenticator extensions for credentials
Protect the inbound endpoint with an authenticator extension, such as OIDC on the receiver, if anything outside the namespace can reach it. For the path to your backend, use the oauth2client extension so the Collector obtains and refreshes tokens instead of storing a static secret in its file. Load client secrets from your cluster secret store. Review the configuration best practices before go-live. Done when an unauthenticated request is refused, the exporter writes successfully with a token, and no credential value appears in the rendered configuration.
Validate the pipeline with test traffic
Run synthetic traffic in a staging environment through two test workspaces, each with a known app and a known number of invocations. Compare per-workspace metric totals with the invocation records Dify Enterprise keeps for the same window. Confirm that trace parent and child spans link correctly across a workflow run. Watch the Collector counters for refused, dropped and queued data. Done when the counts reconcile within the tolerance your finance and SRE owners agreed on, and the drop counters stay flat during a sustained run.
Promote the pipeline to production
Deploy the same pinned, reviewed configuration to the production namespace through your normal change process. Use production-only credentials and a backend tenant that the test environment cannot write to. Enable cost allocation views and SLO recording rules only after production data arrives for every workspace. Keep the previous configuration version so you can roll back if data is refused or labels drift. Done when each production workspace appears in the backend, its labels match the signal register, and Collector health alerts are live.
Route structured event logs separately
Build the event log path as its own pipeline, so high-volume structured records share no queues or retention settings with metrics and traces. Decide which fields your audit and cost reviews need, exclude prompt and response bodies, and set retention in your logging backend to a period your compliance owner has approved. Confirm which Dify Enterprise log exports your release provides, since the set may differ between releases. Done when a sampled record carries workspace and operator identifiers, passes the redaction check, and lands in the intended retention tier.
Security controls for telemetry that carries LLM activity
Telemetry from Dify Enterprise can reveal prompts, user identifiers, model usage and internal endpoints, so the export path needs the same care as the application. The cards below cover the collector, the data it carries and the Dify settings that must stay untouched while you add the pipeline.
Authenticate every receiver
The collector accepts telemetry from Dify and anything else that can reach its ports, so an unauthenticated receiver lets any host on the network inject or read data. Use authenticator extensions such as OIDC on the receivers you expose, and confirm which clients are allowed to send. Test that a request without credentials is rejected before you point Dify at the endpoint.
Encrypt traffic with TLS
Spans and metrics often carry model names, workflow identifiers and request paths, which are useful to an attacker even without prompt text. Terminate TLS on every receiver and on every exporter that leaves your network, and manage certificates through your normal rotation process. Confirm the Dify side trusts your internal certificate authority.
Restrict exporter destinations
A collector that can export anywhere can also forward data to a destination you never approved. Limit exporters to named backends in your own environment, and set outbound credentials through the collector's authenticator extensions, such as the oauth2client extension, rather than embedding static tokens in configuration files. Review the exporter list whenever a pipeline changes.
Keep prompts and secrets out of telemetry
Prompt text, retrieved knowledge snippets, API keys and end-user input can appear in span attributes or log bodies if instrumentation captures them. Decide which attributes are allowed, drop or hash sensitive fields in Collector processors, and test with a sample workflow that uses a fake credential to confirm nothing leaks. Ask the vendor which Dify fields are included by default before enabling export.
Treat the collector as a sensitive system
Once the collector holds traces from every workspace, it becomes a concentrated source of organizational AI activity. Give it its own host or namespace, limit administrative access, patch it on the release cadence you set, and include it in backup and access reviews. Verify the release artifacts you deploy by checking the signatures, checksums and SBOM files published with each release.
Leave Dify security settings intact
Export settings do not change how Dify signs sessions or filters outbound requests, so SECRET_KEY and the SSRF_PROXY settings must stay as deployed. Changing SECRET_KEY after deployment logs out all users, and the SSRF proxy blocks internal and private IP ranges on outbound requests. Record these values in your change process and check them after each upgrade, when the .env.example file may add variables.
Questions teams ask before exporting Dify telemetry
What drives the cost of exporting Dify telemetry to an OpenTelemetry collector?
The cost depends on four things: which signals you export (metrics, traces, or both), how many workspaces and apps produce data, which backend stores and queries the data, and how authentication to the collector is handled. Each signal adds volume and retention needs, and each workspace multiplies the number of series or spans you must allocate. Backend choice drives recurring infrastructure and licensing cost, while the one-time work covers instrumentation mapping, collector configuration and dashboards. Ask any partner to price these components separately in the written scope, and to state which costs recur after handover, such as storage, collector hosting and retainer support.
Can our own platform team build this pipeline without a partner?
Yes, if your team already runs OpenTelemetry and understands Dify's runtime components. The OpenTelemetry Collector is Apache-2.0 licensed and vendor-agnostic, so the collector itself carries no license fee. The harder parts are mapping Dify's signals to your cost allocation keys, securing the export path, and keeping dashboards aligned with service level objectives. Confirm with Dify directly which telemetry options your Enterprise version exposes before you design around them, because the available signals determine how much mapping work remains. A partner is useful when the team lacks capacity or needs an independently documented build.
How long does a telemetry export project usually take?
Duration depends on the fixed scope agreed in writing. A narrow scope that exports one set of metrics to an existing collector and one backend takes far less time than a project that adds traces, workspace-level cost tags, a separate structured event log pipeline and new dashboards. Other factors include the readiness of your collector and backend, change approval windows, and how many environments must be tested before production. Most engagements are fixed-scope, so ask for a timeline tied to specific deliverables and acceptance tests rather than a generic estimate. No duration is given here because none has been researched for this work.
What do our platform and security teams need to provide?
Plan for access to the Dify Enterprise cluster, the namespace where Dify runs, and the collector and backend you intend to use. Platform engineers need to confirm network paths from Dify to the collector, certificate handling, and where collector configuration lives under change control. Security teams should define how collector connections are authenticated. OpenTelemetry supports authenticator extensions that can verify incoming connections, for example with OIDC, and add credentials to outgoing exporter requests, for example through the oauth2client extension. Agree on who owns secrets for those credentials and how they rotate before work starts.
Is the OpenTelemetry Collector production ready for every signal?
Not uniformly. OpenTelemetry trace APIs and SDKs are stable in most languages, but log SDK maturity varies, with several languages still in development. Core Collector components also carry mixed stability levels, which each component's README documents. Check the stability of every receiver, processor and exporter in your pipeline before you commit to it, and pin the Collector release you test. The Collector releases ship contrib distributions with signed artifacts, SHA256 checksums and SBOM files, which helps your supply chain review. Track new releases, since the project ships several versions per month.
How does cost allocation from telemetry differ from invocation logs?
Telemetry export gives you time series and traces that show resource use, latency and error rates across the platform, which suits cost allocation trends and service level objectives. Invocation logs, covered separately in the Dify Enterprise invocation logs topic, record each app run with fields such as status, duration, tokens, operator and source. Telemetry aggregates usage, while invocation logs attribute usage to individual runs. Many teams need both: telemetry for platform-wide dashboards and alerts, and a structured event pipeline for per-run chargeback and audit questions. Decide which question each source must answer before you choose the fields to export.
Should structured event logs go through the same pipeline as metrics and traces?
Usually not in the same path. Metrics and traces serve dashboards and alerting, while structured event logs serve investigation, audit and chargeback, so they often need different retention, access controls and schemas. Building a separate pipeline lets you apply stricter access rules to records that may contain operator identifiers or source details, and it avoids overloading trace storage. Confirm which fields each record contains, how long records must be kept under your policy, and who may read them. Treat the two pipelines as separate deliverables in the written scope so each has its own acceptance test.
Running SLOs and cost reports after go-live
Once telemetry is flowing, the ongoing work is keeping the pipeline trustworthy enough that SLO and cost numbers can be relied on. Review collector health on a fixed schedule, covering dropped spans, exporter failures, queue saturation and memory pressure on the gateway. A cost report built on a silently failing exporter is worse than no report, so alert on the collector itself and on gaps in the metrics that feed your allocation tables. Confirm the collector's own stability level for each component you use, since the status documentation notes that core components vary in maturity and each README records it.
Keep the Collector current with care. Releases ship the contrib distribution with signed artifacts, SHA256 checksums and SBOM files, so verify those against your artifact policy before each rollout and test the new build against a copy of your configuration. Pair Collector updates with Dify upgrades. Dify ships minor releases roughly monthly, and any release that changes span names, metric labels or log fields can break dashboards and chargeback rules without an error. Read the release notes for telemetry changes, and follow the upgrade guide for the target version on the GitHub Releases page, taking a backup before every upgrade and checking .env.example for new variables afterward.
Align the structured event log pipeline with governance. Ask your compliance and records owners how long invocation-level events must be kept, who may read them and where they may be stored, then set those retention rules in the log backend rather than relying on defaults. Re-check these settings after each Dify upgrade, because new event fields may contain prompts or user identifiers that your classification scheme treats differently. Record each review so the evidence of retention decisions exists when an assessor asks for it.
How to verify cost and SLO data is correct
This sequence checks that the token counts, request counts, error signals and workspace labels reaching your OpenTelemetry backend match what Dify Enterprise records, so cost reports and SLO alerts can be trusted before anyone acts on them. It finishes with a signed-off reconciliation record that names what was checked and what remains unverified.
Freeze a reference window for comparison
Choose a fixed time window and a known set of test apps in two or three workspaces, then ask the platform team to avoid changing those apps during the check. Record the workspace names, app names and model providers involved. Done when the team has a written reference list that both the telemetry dashboards and the Dify invocation logs can be filtered against for the same timestamps.
Reconcile token and request counts
Run a fixed set of test invocations through the reference apps, then compare the request counts and token totals in your metrics backend with the per-run figures in the Dify Enterprise invocation logs for the same window. Confirm with your Dify Enterprise contact which token fields the exporter emits, since the exact metric names and units are set by your configuration. Done when each app's totals match within a tolerance you have agreed in advance, and every gap has a recorded cause such as retries, streaming behavior or a dropped batch.
Check workspace labels against the admin panel
Compare the workspace identifiers and names attached to exported metrics and traces with the workspace list in the Dify Enterprise admin panel. Include a workspace that was renamed or archived during the test window if one exists. Done when every label on the reference apps maps to exactly one workspace, and any label that cannot be mapped is logged as a defect rather than assigned by hand in a dashboard.
Inject controlled failures into test apps
Use a test app and a deliberately unavailable dependency, such as a disabled model credential or an unreachable tool endpoint, to generate failed runs at a known rate. Confirm the failures appear as errors in the invocation logs, then confirm the error-rate query in your metrics backend reports a similar proportion. Done when the error-rate SLI moves as expected and the alert rule you configured fires and resolves once failures stop.
Verify alerts reach the right people
Trigger the alert route from the failure test and follow it through your notification and on-call tooling. Confirm the alert names the workspace and app, and that the link returns to the correct dashboard panel. Done when the responder can identify the affected workspace from the alert alone and the notification record is retained in your incident tooling.
Test the pipeline under collector interruption
Stop or restrict the OpenTelemetry Collector for a short, planned period while test invocations continue, then restore it. Compare the resulting gap in telemetry with the invocation logs to see what was lost and whether the exporter or Collector queued data. Done when the team has documented the data loss behavior, and the cost and SLO reports carry a visible note for any window with missing telemetry.
Record the reconciliation and open items
Write a reconciliation record covering the reference window, the tolerances used, each mismatch and its cause, the label mapping result, the failure test outcome and the interruption test. List items you could not confirm, such as vendor-specific field behavior or backend retention, and assign an owner to each. Done when the record is approved by the platform owner and the finance or SRE consumers of the reports, and is repeated after each Dify or Collector upgrade.
How DeployOpen scopes and delivers Dify telemetry export
DeployOpen is a Chainscore Labs company that plans, implements, hardens and supports self-hosted open-source platforms inside your own infrastructure. Telemetry export touches Dify, your collector, your backends and your cost and SLO reporting, so the work starts with a written scope that lists every signal and destination before anything is built.
Who DeployOpen is
DeployOpen is a Chainscore Labs company. It plans, implements, customizes, hardens, backs up, upgrades and supports self-hosted open-source platforms in your cloud accounts, data centers or air-gapped networks. For telemetry work, that means the collector, the pipeline and the Dify configuration run on infrastructure you control, and your telemetry stays in your own backends. DeployOpen does not hold vendor partner status or certifications for Dify or OpenTelemetry, so confirm any credential claim directly with the vendor.
Fixed-scope written agreement
Most engagements are fixed-scope. Scope, deliverables and timelines are agreed in writing before work starts. For this topic, the agreement should name the Dify version and deployment mode, the signals in scope (metrics, traces, structured event logs), the collector topology, and the acceptance checks for each signal. Anything not named in the agreement, such as new dashboards or extra backends, becomes a separate change to scope.
Signal and backend inventory
Before a quote is produced, DeployOpen recommends an inventory of what Dify will emit and where each signal must land. List the Dify services that generate telemetry, the metrics needed for cost allocation by workspace or team, the trace detail needed for SLO work, and the event log fields required by your audit or security teams. Record each destination, its owner, retention and access model. This inventory is what drives the size of the work.
Collector and pipeline build
DeployOpen configures the OpenTelemetry Collector as the single export point, with receivers, processors and exporters set for each signal. Collector authentication is planned with the collector's authenticator extensions, so incoming and outgoing connections carry the credentials your security team approves. The build also separates the structured event log pipeline from the metrics and traces path, so a backend outage in one does not block the other.
Verification and handover
Acceptance testing confirms that each signal arrives at its destination with the expected labels, that cost allocation totals match a known test workload, and that failed exports are visible. Handover covers the collector configuration, runbooks for version changes, and the list of open items. Because Dify ships minor releases about monthly, the handover should name who checks telemetry after each upgrade.
Optional retainer for ongoing support
Ongoing support, upgrades and continuous testing are available as retainers. A retainer usually covers collector version updates, Dify upgrade checks that affect metric names or trace attributes, and changes to dashboards and SLO definitions. Ask for the response expectations, the upgrade review process and what is excluded before you sign. Retainer cost recurs, while the initial build is a one-time fixed-scope cost.
