How to instrument self-hosted Dify observability with OpenTelemetry and Sentry
An operations team that runs Dify needs its errors and traces in its own collector and error tracker, not only inside the application. The settings that make this possible are environment variables, and Dify's environment variable reference documents what each one does. The design choices around them are yours: which collector receives the data, how much of it is kept, which Sentry project owns each service, and how environments are told apart. Decide these before the first production deployment, because changing them later splits your history across old and new settings.
Start with the export protocol. Setting OTEL_EXPORTER_OTLP_PROTOCOL to grpc makes Dify use gRPC exporters, and any other value makes it use HTTP, so confirm which protocol your collector accepts and match it exactly. Sampling is the second decision. API_SENTRY_TRACES_SAMPLE_RATE defaults to 1.0, which sends every trace. That is useful in a test environment and expensive in a busy production one, so set a lower value deliberately after you have measured trace volume. Also decide how you will handle the cost of log retention in the same review, since traces and stored workflow logs both grow with usage.
Use DEPLOY_ENV to label every environment, including staging, test and production, so that Sentry and OpenTelemetry data can be filtered without mixing environments. Its default is PRODUCTION, so a non-production instance that keeps the default will appear with production data. Dify also sends this value as the X-Env response header, which helps when you check a running instance. Plan a separate Sentry DSN for each service that reports errors, including the API, the web frontend and the plugin daemon, so an alert can be routed to the team that owns that service. Confirm in a test instance that each DSN receives events before you rely on the routing.
Which observability setting fits which operator question
Operators who need to follow requests across the API, worker and plugin services, or see where latency builds up, use OpenTelemetry export. Dify sends telemetry over OTLP to a collector that you run, so the data stays inside your monitoring stack. The protocol setting decides which exporter the process uses, and the environment variable reference lists the keys to confirm before you change anything.
- Protocol: Setting
OTEL_EXPORTER_OTLP_PROTOCOLtogrpcselects gRPC exporters. Any other value selects HTTP, so confirm which transport your collector accepts before you change it. - Network path: The collector endpoint must be reachable from the API and worker containers, which means your Compose network rules or cluster policies must allow the connection.
- Volume: Full export gives the most detail but raises collector load and storage cost, so pair it with a sampling decision.
- Data handling: Span attributes depend on what the application records, so review what reaches the collector with your privacy and retention owners.
After one test workflow run, confirm that the collector receives spans from both the API and the worker, and that its retention matches your policy.
On-call teams need exceptions with stack traces and release context so they can act on failures without reading raw container logs. Sentry captures error events for each service that you point at a DSN. Sentry answers the question of what failed and where, while traces and metrics answer how the system behaved over time, so the two are usually used together rather than swapped.
- Per-service DSNs: Give the API, the web service and the plugin daemon their own DSN so ownership, alert routing and quotas follow each service. Confirm the exact variable names in the environment reference for your release.
- Scope of capture: Sentry records errors and their context. It does not provide the capacity view that dashboards built on metrics give.
- Data minimization: Error events can include request details, so confirm the scrubbing options available in your Sentry setup and who can read events.
- Hosting choice: Decide whether Sentry runs as a service you operate or a hosted service, and check that the choice meets your data residency rules before you enable DSNs.
Trigger a test error in a non-production workspace and confirm that it appears in the correct project with the correct environment tag.
Sampling controls how many transactions are traced and sent to Sentry. The sampling rate is a cost and detail trade-off: a high rate gives complete visibility into slow paths, and a lower rate reduces event volume and storage. The documented setting for the API service is API_SENTRY_TRACES_SAMPLE_RATE, and the environment reference lists its default, so check that page before setting a value for production.
- Default of 1.0: The documented default traces every transaction. This suits early rollout and low traffic, but it can become expensive as usage grows.
- Lower rates: Reducing the rate lowers volume, but uncommon slow requests may never be sampled, so keep a higher rate while you investigate a specific problem.
- Service scope: The documented key applies to the API service. Confirm whether the web service or plugin daemon exposes an equivalent setting in your version.
- Change control: Treat the rate as an operational setting. Record each change and compare error and trace counts before and after it.
Compare the error picture from a sampled period with a period at full rate on your own traffic before you settle on a lower value.
When several stacks send data to the same Sentry project or collector, the environment tag is what tells them apart. Dify uses DEPLOY_ENV to tag Sentry and OpenTelemetry data by environment. The tag is also sent as the X-Env response header, which testers can use to confirm which stack answered a request. The setting does not change application behavior.
- Default value:
DEPLOY_ENVdefaults to PRODUCTION. A staging or test stack that keeps the default mixes its data into production views. - Naming scheme: Pick a fixed set of values, such as the names your change process already uses, and apply them the same way across every stack.
- Header checks: Use the
X-Envheader in smoke tests to confirm that a request reached the stack you intended, especially behind a load balancer. - Filtering: Dashboards and alert rules should filter on the tag so that test traffic does not page on-call staff.
Set DEPLOY_ENV on every stack, then confirm the value in both Sentry and the collector before you go live.
Configuration options for OTLP, Sentry, sampling and DEPLOY_ENV
This table lists the telemetry settings a self-hosted Dify exposes and when each one fits. Read it as a checklist for choosing the exporter protocol, sampling level and environment label before you enable anything in production. The main takeaway is that the defaults send traces at full sample rate and tag everything as PRODUCTION, so non-production stacks need explicit changes.
| Setting | What it controls | When to change it |
|---|---|---|
| OTEL_EXPORTER_OTLP_PROTOCOL | Transport for OpenTelemetry export. Set to grpc for gRPC exporters. Any other value uses HTTP. | Match the protocol your collector listens on. Confirm the collector endpoint and port with your platform team before setting it. |
| API_SENTRY_TRACES_SAMPLE_RATE | Fraction of API transactions sent to Sentry for tracing. The documented default is 1.0. | Lower it on high-traffic APIs to control event volume. Keep it at 1.0 in pilot environments where you need complete traces. |
| DEPLOY_ENV | Environment label attached to Sentry and OpenTelemetry data, also returned as the X-Env response header. The default is PRODUCTION. | Set a distinct value per environment, such as staging or test, so dashboards and alerts do not mix non-production data with production. |
| Per-service Sentry DSN settings | Separate Sentry project destinations for the API, web and plugin daemon. Confirm exact key names in the environment reference. | Use separate DSNs when different teams own each service or when you need separate retention and access rules per component. |
| Sentry and OTLP behavior change | DEPLOY_ENV does not change application behavior. Sampling and exporter settings change what leaves your network and how much. | Review data classification before export. Prompts and user content may appear in traces, so confirm what your collector and Sentry project store. |
Setting up OTLP export and Sentry in your Dify deployment
This sequence takes a self-hosted Dify from default settings to telemetry that reaches your collector and Sentry, with each environment tagged and sampled deliberately. Each step ends with a check so your team knows the previous change took effect before moving on.
Confirm the collector endpoint and protocol
Before editing any settings, your platform or observability team confirms the OpenTelemetry Collector endpoint that Dify will send traces to, and which transport it accepts. Dify uses gRPC exporters when OTEL_EXPORTER_OTLP_PROTOCOL is set to grpc, and HTTP for any other value, so the protocol must match the receiver's listener. Ask the collector owner for the exact host, port and whether TLS or authentication headers are required. Done means you have a written endpoint and protocol pair, and someone other than the Dify operator has agreed to it.
Set DEPLOY_ENV for each environment
Set DEPLOY_ENV in the
.envfile for each Dify instance, such as a value naming staging or production, so Sentry and OpenTelemetry data can be separated by environment. The default is PRODUCTION, which means a staging instance left unchanged will report as production. The value is also returned as the X-Env response header without changing application behavior. Done means each instance has a distinct value recorded in your configuration inventory, and you have not reused a production label on a test deployment.Set the OTLP exporter protocol
Set OTEL_EXPORTER_OTLP_PROTOCOL to match the endpoint you confirmed, then point the exporter at your collector using the Dify environment variables for OpenTelemetry. Keep the change in version-controlled configuration rather than editing containers by hand, because the value must survive upgrades that compare
.envagainst.env.example. Done means the variable is present in the environment file for the target instance and the value is exactly grpc or the HTTP value your collector expects.Create separate Sentry projects or DSNs
Create a distinct Sentry project or DSN for each service that reports errors: the API, the web frontend and the plugin daemon. Separate DSNs keep alert routing, ownership and issue volume apart, so a noisy plugin daemon does not bury API errors. Store each DSN as a secret in your secrets manager and inject it through the environment file for that service. Done means three DSNs exist, each is mapped to one service in your runbook, and a deliberate test error appears only in the expected Sentry project.
Set the API trace sampling rate
Set API_SENTRY_TRACES_SAMPLE_RATE to a value your team has chosen after weighing trace volume against investigation needs. The documented default is 1.0, which samples every transaction, so a high-traffic production instance may generate considerable data at that setting. Agree the value with the team that pays for trace storage and document the reason beside the setting. Done means the chosen rate is recorded per environment, and you have confirmed in your Sentry project that transactions arrive at the expected proportion.
Restart services and confirm the X-Env header
Restart the Dify containers so the new environment variables load, then check the response headers from the web entry point. The X-Env header should carry the DEPLOY_ENV value you set for that instance. This check confirms the variable reached the running process, not just the file on disk. Done means the header matches on a request to each instance, and any instance returning a different value is corrected before you continue.
Verify with a test trace and error
Generate one controlled workflow run in a non-production app, then confirm a trace appears in your collector and backend with the expected DEPLOY_ENV tag. Trigger one known test error to confirm it lands in the correct Sentry project for the service. Check that no sensitive prompt content is being captured beyond what your privacy policy allows before you enable this on production traffic. Done means the trace, the tagged error and the sampled transaction are all visible, and the results are recorded in your change ticket.
What sampling and retention choices change for operators
Sampling, payload handling, log retention and access decide how much telemetry you collect, how long it lives and who can read it. Each choice shifts cost, privacy exposure and incident-response value, so settle them with the security and compliance owners before turning on export.
Sampling rate trade-offs
The API_SENTRY_TRACES_SAMPLE_RATE setting defaults to 1.0, which keeps every API trace. Lowering it reduces volume and cost but can drop the rare failing request you need to investigate. Decide per environment: a full rate in staging may be reasonable, while production may need a lower rate plus error capture that is not sampled. Test the choice against a real incident before you rely on it.
Telemetry volume and protocol
Volume depends on how many API requests, workflow runs and plugin calls your instance handles, multiplied by the sampling rate. Setting OTEL_EXPORTER_OTLP_PROTOCOL to grpc selects gRPC export, while any other value uses HTTP, so confirm which your collector accepts. Ask your collector team for ingest limits and storage sizing before enabling full sampling in production.
Prompt data in traces and events
Traces and error events can carry request context, and LLM applications often receive personal data or confidential documents in prompts. Before export, decide which fields your collector and Sentry project may store, whether scrubbing rules apply, and whether production prompts should be excluded. Confirm each setting against your privacy and data classification policy, since the docs do not describe every payload field.
Workflow log and message retention
Workflow logs and conversation messages are stored in Dify itself, separate from OpenTelemetry and Sentry data. Their retention decides how long user inputs and model outputs remain in your database. Set a retention period that matches your privacy policy, turn on scheduled cleanup, and check that backups follow the same window, because deleted records may still exist in older backups.
Environment tagging with DEPLOY_ENV
DEPLOY_ENV defaults to PRODUCTION and tags Sentry and OpenTelemetry data by environment, also sent as the X-Env response header. Without distinct values per environment, staging errors and test traffic blend into production dashboards and alerts. Set a value for each instance and agree the naming with your operations team before the first export.
Who can read collector and Sentry data
Anyone with access to your collector backend or Sentry project can see the telemetry it holds, which may include prompts, identifiers and error context. Limit read access by role, separate production projects from development ones, and review memberships on the same schedule as other privileged access. Ask your identity team which groups receive access and how removals are handled.
Questions operators ask before instrumenting Dify
How much effort does instrumenting a self-hosted Dify take?
The configuration itself is short, and most of the effort sits in the surrounding work. The Dify environment variables are only one step. Effort rises with the number of services you export, whether you already run an OpenTelemetry Collector, how many Sentry projects you create, and how much validation you need before production. A typical sequence is to set the environment label, choose the OTLP protocol and sampling rate, add a separate DSN for each service, then confirm that traces reach your backend and errors reach the right Sentry project. Dashboards and alert rules often take longer than the variables. Ask for a written scope that lists the services, backends, environments and validation steps before agreeing on effort.
Can our team set up Dify observability without outside help?
Yes, if your team already runs Docker Compose and operates its own telemetry backend. The settings are documented environment variables, so changes are made in configuration and applied through a normal container restart. The harder parts are organizational and network work: agreeing on environment label names, opening network paths from the Dify containers to your collector and Sentry endpoints, and deciding who owns each Sentry project. Outside help makes most sense when you want a fixed-scope written plan, validation across the API, web and plugin daemon, or dashboards and alerts built to your operating model. Whichever path you choose, test the change in a non-production stack first.
What does the work need from us: collector, Sentry access and DSN ownership?
Three things are needed before configuration starts:
- An OTLP endpoint on your collector, with the protocol set to match it. The reference says
OTEL_EXPORTER_OTLP_PROTOCOLset to grpc uses gRPC exporters and any other value uses HTTP. - Sentry organization access with rights to create projects and issue DSNs, plus a named owner for each DSN.
- Network routes from the Dify containers to both the collector and Sentry.
DSNs are credentials, so store them in your secret manager, not in plain configuration files shared across teams. Assign an owner who can rotate a DSN and update the environment file, and record which Sentry project belongs to which service and environment.
How should we choose a trace sampling rate for Dify?
Start from the documented default, then lower it on purpose. The environment reference says API_SENTRY_TRACES_SAMPLE_RATE defaults to 1.0, which keeps every trace. That is reasonable during rollout, but trace volume grows with LLM traffic, workflow runs and plugin calls, and so do storage and ingestion costs. Choose the rate by comparing expected volume against how often your team needs a complete trace to debug a slow or failing workflow. Change one value at a time, measure the resulting volume, and record the chosen rate for each environment so that staging and production differ on purpose. Confirm in the reference which settings control error capture separately from traces before you tune either one.
How does workflow log retention relate to observability data?
They serve different purposes. Traces and errors help operators diagnose how the system behaves, and they usually go to your monitoring backends. Workflow logs and messages are application records of what users and triggers did, and they can contain personal data and prompt content. Retention for those records is controlled by Dify's own settings for scheduled cleanup of workflow logs and messages. Locate those settings in the environment reference for your version and confirm their names and behavior before you set them. Do not assume that a short trace retention period covers a privacy commitment, and do not keep traces that contain prompt content longer than your data retention policy allows.
Does DEPLOY_ENV change how Dify behaves?
No. DEPLOY_ENV labels data without changing application behavior. Its default is PRODUCTION, and the environment reference says it tags Sentry and OpenTelemetry data by environment and is sent as the X-Env response header. Use a distinct value for each stack, such as separate labels for development, test, staging and production, so dashboards, alerts and Sentry issues do not mix data from different environments. Because the default is PRODUCTION, an unset variable in a non-production stack will be labeled as production. Check the label on every stack after deployment, and agree on the naming convention with the teams that own the dashboards.
Do we need Dify Enterprise to export telemetry?
The telemetry settings covered here appear in the self-hosting environment reference, which applies to the self-hosted Docker Compose deployment. The supplied material does not link these settings to Enterprise, so confirm the exact behavior for your edition and version before you plan around it. For context, the pricing page lists the self-hosted Community edition as free and limited to a single workspace, while the Enterprise edition is custom priced and adds SSO, multiple workspaces, advanced security, a commercial license and professional support. Those additions are a licensing decision, separate from whether your telemetry backend is ready to receive data.
How DeployOpen scopes Dify observability work
DeployOpen scopes observability as part of a fixed-scope Dify engagement, so the telemetry work is written into the same statement of work as the deployment, with deliverables and timelines agreed in writing before work starts. In practice this means the scope names the exact settings to be applied: the OTEL_EXPORTER_OTLP_PROTOCOL choice, the collector endpoint and its authentication, the API_SENTRY_TRACES_SAMPLE_RATE value, separate Sentry DSNs for the API, web and plugin daemon, the DEPLOY_ENV value for each environment, and the workflow log retention settings. Each of these is a decision your team should confirm before the quote is final, because the collector, Sentry project layout and retention policy all sit with your organization.
The written scope should also state what counts as done. For observability, that usually means a test trace visible in your collector, errors from each service arriving in the correct Sentry project tagged with the right environment, and a documented check that the sampling rate matches the agreed value. Ask the proposal to list who provides the collector and Sentry organization, who holds the credentials, and what evidence is handed over at the end. Items your team owns, such as dashboards, alert routing and on-call procedures, should be marked as out of scope or as client-side work so there is no ambiguity about who maintains them.
Observability does not stop at handover. Telemetry changes when Dify releases new versions, when environment variables are added, and when your collector or Sentry plan changes, so many organizations keep a retainer for upgrade checks and ongoing support alongside the fixed-scope build. Ask any partner, DeployOpen included, to state in writing which tooling integrations it has built or tested and which it has not, rather than assuming a partnership or certification exists. Confirm the current Dify release and its environment variable reference with your own environment before work begins, since those details change with each release.
How DeployOpen works on this
DeployOpen plans, implements, documents and supports self-hosted Dify inside your own infrastructure, including telemetry export to your collector and error tracking through your Sentry setup. We start from the telemetry your operations team needs to answer real questions, then build and hand over the configuration in a form your team can run.
How we scope
- We confirm where your OpenTelemetry collector and Sentry projects run and who owns them.
- We agree which Dify services get their own Sentry DSN, including the API, web and plugin daemon.
- We record sampling targets, DEPLOY_ENV values and workflow log retention in the written scope.
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
How we build
- We set the OTLP protocol, endpoint and sampling variables in your Dify environment configuration.
- We separate DSNs and environment tags so staging and production errors never mix in one view.
- We test that traces and errors arrive at your collector and Sentry, and that the X-Env header matches each environment.
- We confirm that log retention settings match your privacy policy before the configuration goes live.
How we hand over
- We document every observability variable, its default, and the service it affects in your deployment.
- We provide a runbook covering how to change sampling rates, rotate DSNs and verify exports after an upgrade.
- We walk your operations team through dashboards, alert paths and what each signal does and does not show.
- We record the configuration in a form your team can keep under version control alongside the deployment.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers once the handover is complete.
- We re-check telemetry settings against each Dify release, since new variables arrive with versions.
- We review sampling and retention periodically with your team as traffic and compliance needs change.
- Retainer scope, response expectations and review cadence are confirmed in writing before they start.
