How Grafana connects to Tempo for distributed tracing
If your services already emit traces and your team needs to query them next to logs and metrics, the Tempo data source in self-hosted Grafana is the supported way to do it. Grafana is the query and visualization layer, and Tempo is the trace store it reads from. You add the connection inside Grafana under its data source settings, not in Tempo's own interface, and then build trace queries in Grafana's Explore view or on dashboards. The data source supports *TraceQL*, the query language Tempo uses to select spans and traces by attributes, duration and status.
The telemetry itself usually comes from *OpenTelemetry*, which is an observability framework for generating, exporting and collecting traces, metrics and logs. OpenTelemetry is vendor-neutral and is not a backend, so it does not store or display anything on its own. Your applications send spans through OpenTelemetry instrumentation and a Collector or Grafana Alloy pipeline to Tempo, and Grafana reads from Tempo. Before you configure the data source, confirm which component receives your spans, what storage backs Tempo, and which URL and network path Grafana will use to reach the Tempo query API from your infrastructure. Those answers decide the rest of the build order.
Plan the Grafana side as a separate piece of work from the trace pipeline. Check that the Tempo data source is available in your Grafana version, that the Grafana server can reach Tempo's query endpoint, and that access to traces follows the same identity and permission model you use for other data sources. The Grafana Tempo data source documentation lists the settings and the links the data source can build to other signals, and it is the right reference to confirm current field names before you write any configuration as code.
What a Tempo integration needs before configuration
A Tempo data source only works when the Grafana instance, the Tempo endpoint, the trace senders and the linked sources are all in place and agreed before anyone edits a data source. The cards below list what to confirm in your own environment, in the order that avoids rework.
Grafana on a production database
Confirm that Grafana stores its configuration in MySQL or PostgreSQL rather than the default SQLite. The installation docs say SQLite does not scale for production, and high availability requires a shared database. Check the database version against the supported minimums before you plan the Tempo rollout, since a data source change on a fragile instance is harder to roll back.
A reachable Tempo query endpoint
Grafana must reach the Tempo query API over the network path you intend to use in production, including any proxy, TLS termination or authentication layer. Confirm the Tempo version and its query API with the Tempo documentation for your release rather than assuming one, because TraceQL support and endpoint paths depend on that version. Test from the Grafana server itself, not from a laptop.
Trace senders and collection path
Identify which instrumented services send traces and whether they go directly to Tempo or through an OpenTelemetry Collector. A Collector gives you one place to batch, filter and authenticate traffic before it reaches storage. Confirm that every service uses the same trace context propagation, or spans will not join into complete traces in Grafana.
Loki source for trace-to-logs
If you want links from a span to its logs, a Loki data source must already exist in Grafana and log lines must carry a trace identifier that matches the one Tempo stores. Confirm the label or field name your services use for the trace ID, because the link is built from that value. Without a consistent identifier, the link opens an empty log view.
Prometheus source for metrics and service graph
Trace-to-metrics links and the service graph need a Prometheus data source that receives service graph metrics from Tempo. Confirm that a metrics generator or equivalent process writes those series to a Prometheus-compatible store Grafana can query. The service graph will be empty if that linked source is missing or does not hold the expected series.
Access model and secrets for each source
Decide which Grafana users may query traces, logs and metrics, and create the service accounts or credentials each data source needs. Store passwords and tokens in Grafana's secrets handling rather than plain configuration files, and confirm the read permissions on each backend match the access model your team uses.
Configuring the Tempo data source in sequence
This sequence takes Grafana from an unconnected Tempo backend to a working trace view with links into logs, metrics and profiles. Each step ends with a check so the next link is built on a verified foundation.
Confirm the Tempo endpoint and network path
Before touching Grafana, the platform team confirms the URL that the Grafana server will use to reach Tempo's query API, the port, and any proxy or firewall rule between the two. Run the check from the Grafana host itself, not from a laptop, because the Grafana server is the component that makes the request. Confirm with the team that runs Tempo whether the query endpoint is exposed through a gateway, whether TLS is terminated there, and whether authentication headers are expected. Done means a plain HTTP request from the Grafana host to the Tempo query endpoint returns a response rather than a timeout or a refused connection.
Add the Tempo data source in Grafana
An administrator opens Connections, adds the Tempo data source and enters the confirmed URL. Set authentication to match the gateway in front of Tempo, if one exists, and record the data source name, since dashboards and links refer to it. Save and test the connection. Done means the data source test reports success. If it fails, the most common causes are a URL that works from another host but not from the Grafana server, a missing authentication header, or a certificate the Grafana server does not trust.
Run a first TraceQL query on known traces
Choose two or three traces that you already know exist, such as a request you generated in a test environment, and find them in Explore with the Tempo data source selected. Start with a simple TraceQL filter on a service name or span attribute that appears in those traces, then open one result to view the span tree. Confirm that the trace ID matches the one your application logged or returned. Done means a known trace opens with the expected services and spans. An empty result usually points to the wrong tenant, a retention window that has passed, or instrumentation that does not set the attribute you filtered on.
Link traces to logs in Loki
In the Tempo data source settings, enable trace to logs and select the Loki data source the team already runs. Map the trace fields to log labels that your collector actually writes, such as service name, and set the time window around the span so the log query is bounded. Save the settings. Done means that opening a span in a trace shows a logs link, and clicking it returns Loki lines for that service and time window that include the trace ID. If the link returns nothing, compare the label names in Loki with the mapping you entered, because a mismatch is the usual cause.
Link traces to metrics in Prometheus
Enable trace to metrics in the same data source settings and select the Prometheus-compatible data source that holds your span-derived or application metrics. Map the service and span tags to the labels used in those metrics. Save and then open a span from a known trace and follow the metrics link. Done means the link opens a Prometheus panel scoped to the service and time range of that span. This link depends on the metric labels matching the trace tags, so confirm the label names with the team that owns the metrics pipeline before you rely on it.
Add trace to profiles links in Pyroscope
Add the trace to profiles setting only if a Grafana Pyroscope data source is already running in the same instance. Select the profiles data source, map the span's service and profile type fields, and save. Open a span from a trace that ran CPU-heavy code, then follow the profiles link. Done means the link opens a flame graph scoped to the service and time window. If Pyroscope is not deployed, skip this step and keep the logs and metrics links as the verified baseline. Confirm the profile labels with the team that collects profiles before you map them.
Record the verified configuration
Export or document the final Tempo data source settings, including the URL, authentication method, trace to logs, trace to metrics and trace to profiles mappings, and the test traces used for each check. If your instance is managed as code, move these settings into provisioning files so a rebuild restores them. Store the record with the runbook for the Grafana instance. Done means another engineer can repeat each check from the documentation and reach the same results.
How TraceQL queries find the right traces
TraceQL is the query language the Grafana Tempo data source uses to search stored traces by span attributes, service names and durations, so the way you write it decides which traces an on-call engineer can find during an incident. A query such as a filter on the service name plus a minimum span duration returns traces that match those conditions across the whole retention window. If the attribute you filter on was never set by the instrumentation, the query returns nothing, and the engineer may conclude that the failure did not happen. Before the integration goes live, check which resource and span attributes your services actually emit, how they are spelled, and whether they are consistent across teams.
Query breadth is a trade-off. A broad search, such as all traces above one second in a given service, is easy to write and useful for exploration, but it can return large result sets and load the backend heavily when the time range is wide. A narrow search, such as one route, one status code and a specific attribute value, is fast and precise, but it only finds traces that carry those exact values. Teams should agree on a small set of standard attributes, such as service name, HTTP route and error status, that every instrumented service sets, and document the TraceQL patterns that engineers should start from during triage.
Test these queries against your own telemetry before any dashboard or runbook depends on them. Run each pattern over a known incident window or a deliberately injected slow request, confirm that the expected traces appear, and check the result count and response time at the time ranges your responders actually use. Then review the query again after each instrumentation change or service rename. The Grafana Tempo data source documentation describes the query editor and TraceQL support, and it is the right reference for confirming the syntax your installed Grafana version accepts.
Which links need which data source
The trace to logs link takes a span and opens matching log lines in Loki or another log data source. Tempo does not store your logs, so the link only returns results when the application writes the trace ID into log records and the log store can filter on it. The Tempo data source supports Loki and other log sources, so the target is a separate data source you configure and test.
- Required data source: a Loki data source, or another supported log data source, configured in Grafana before you set the link.
- Trace ID in log lines: confirm that your services write the trace ID as a field or label that the log store can search, and that the field name matches what you enter in the link.
- Span attributes for filters: decide which span tags, such as service name or pod, should be mapped to log labels so the query narrows to the right stream.
- Time window: confirm the offset around the span start and end, since log ingestion delay can hide lines that arrive after the span closes.
- Multi-tenant access: if Loki runs with tenant isolation, confirm the Grafana data source sends the correct tenant header for each team.
Test the link with a trace from a known request and confirm the log lines for that request appear in the same time range.
The trace to metrics link moves from a span to a Prometheus query for the same service or operation. It depends on the metric labels you already emit, and the link is only as useful as the match between span attributes and metric labels. The service graph needs its own linked Prometheus source with service graph metrics, so plan that data source separately.
- Required data source: a Prometheus data source, or a Prometheus-compatible store, configured in Grafana and reachable from the dashboard users.
- Label alignment: confirm that the service and operation names used in span attributes match the label values in your metrics, including case and naming conventions.
- Service graph metrics: confirm that the metrics needed for the service graph are being generated and written to the linked Prometheus source, because the service graph reads from them rather than from Tempo.
- Query scope: decide which metric families the link should query, and confirm that high-cardinality labels are not added to the link by accident.
- Time alignment: check that scrape intervals are short enough that the metric window around the span shows meaningful values.
Open a span from a known service and confirm the metric panel shows the same service and operation you expect.
The trace to profiles link opens a continuous profile for the span from Grafana Pyroscope. A profile only matches a span when the profiler records the same service identity and time window as the trace. Pyroscope is a separate system, so the link is useful only after profile collection is running in the same environment.
- Required data source: a Pyroscope data source configured in Grafana, backed by a running Pyroscope deployment.
- Service identity: confirm that the profiler's service name matches the service name in span resource attributes, including environment or namespace labels if your profiles use them.
- Profile collection: confirm that profiles are collected for the same processes and instances that emit traces, through your chosen collector or agent.
- Profile types: decide which profile types, such as CPU or memory, are expected by the team reading the span, and confirm they are collected.
- Time window: check the profile retention and sampling interval against the span duration, since short spans may not have enough samples to show a useful flame graph.
Select a span from a service you know is profiled and confirm the flame graph loads for the same time range.
Custom correlations let you link a trace attribute to any other data source or URL, for example a ticket, a configuration store or a second log system. They are the right choice when a built-in link does not cover a system you run. Because you define the query or URL yourself, the checks focus on the attribute values that drive the link and the permissions of the target system.
- Source attribute: confirm which span attribute or tag holds the value to correlate, such as a request ID, customer reference or deployment version, and that it is set consistently.
- Target query or URL: confirm the exact field name and format the target expects, and test with values from real traces.
- Access to the target: confirm that the target data source or application is permitted to receive requests from Grafana users, and that the link does not expose data those users should not see.
- Null handling: decide what should happen when the attribute is missing, and confirm the link does not return misleading results.
- Ownership: name the team that maintains each custom link, since changes to the target system can break the link silently.
Test each custom correlation with at least one trace that contains the attribute and one that does not.
Mapping trace fields to log, metric and profile fields
This table lists the identifiers that Grafana's Tempo links depend on, where each one is usually produced, and what the team must align in instrumentation and label design. Read each row as a contract between the trace, the log, the metric and the profile, because a mismatch in name or format usually produces an empty panel or a missing link rather than an error.
| Field | Where it must appear | What to align | Failure when it differs |
|---|---|---|---|
| Trace ID | Span context, log lines, exemplars | Same hex format in every signal, written under the key the Loki link expects | Trace to logs link returns no matching lines |
| Span ID | Log lines and span context | Same identifier for the span that emitted the log entry | Logs open at trace level but cannot pinpoint the span |
| Service name | Resource attributes, Prometheus labels, service graph labels | One spelling and casing across traces, logs and metric labels | Service graph nodes and trace to metrics queries show no data |
| Span attributes used as labels | Prometheus series derived from TraceQL metrics | Only low-cardinality attributes promoted to labels, named consistently | Dashboards split into many series or filter to nothing |
| Service graph metric labels | Prometheus series for rate, error and duration | Label names the Tempo service graph expects, linked to the right Prometheus source | Service graph stays empty even though traces arrive |
| Profile labels | Pyroscope profiles and span tags | Matching service and span identifiers so flame graphs can be linked from spans | Profile links open an unrelated or empty profile |
| Data source UIDs | Tempo data source settings in Grafana | Each link points to the correct Loki, Prometheus or Pyroscope UID | Links are present but route to the wrong or missing source |
Why the service graph needs a linked Prometheus source
The Tempo service graph does not read its dependency map from trace storage alone. It draws service-to-service edges using rate, error and duration metrics that must already exist in a Prometheus-compatible store, and the Tempo data source has to be linked to that Prometheus data source before the graph renders. Plan this as two data paths from the start: spans flow into Tempo, and service graph metrics flow into Prometheus. Confirm in the Tempo data source settings which Prometheus source is linked, and confirm that the metrics include the service graph series the Tempo docs describe, since a graph built from the wrong source will look empty even when traces are arriving.
Provisioning the metrics side is a separate piece of work. Tempo's metrics generator produces the service graph series and remote-writes them to Prometheus, so the team has to enable the generator, set its remote write target, and make sure the Prometheus instance accepts that write path. Prometheus also needs retention and capacity for these series, and its storage is a single node that is not clustered or replicated by default, so a shared or long-term metrics backend affects how the graph behaves during failures. Ask the platform owner to confirm where these series will live, who owns the Prometheus instance, and how its retention compares with trace retention in Tempo.
If the metrics pipeline is missing or delayed, the service graph shows gaps or stale edges while trace search keeps working, which is easy to misread as healthy. An illustrative case: a new service starts sending spans, but the generator is not writing to Prometheus, so the service appears in search results and never appears on the graph. Build a check that compares a known service's span count in Tempo with its service graph rate in Prometheus, and treat a persistent mismatch as an incident in the observability pipeline, not as an empty dashboard.
Securing trace access and the Tempo endpoint
Trace data often carries request paths, service names and occasionally identifiers or payload fragments, so access to it needs the same care as logs. The cards below cover the endpoint, the Grafana side, the telemetry content and the Grafana settings that connect them. Confirm each control against the current vendor documentation for your versions before you rely on it.
Authenticate the Tempo endpoint
Tempo's query and ingest interfaces should not be reachable without authentication. Place an authenticating reverse proxy or gateway in front of Tempo, and confirm how Tempo's multi-tenant mode and tenant headers are handled before you expose it beyond the Grafana server. Test that unauthenticated requests to the query path are refused from every network segment that should not reach them.
Scope the Grafana service account
Grafana needs read access to traces, and nothing more. Give the Tempo data source credentials or a proxy identity that can query but cannot change tenant settings or ingest data, and do not reuse credentials shared with other data sources. Rotate those credentials on a schedule your team can enforce and store them in Grafana's configured secret handling rather than in dashboard JSON.
Keep secrets out of spans
Span attributes, events and resource fields are copied into storage and then displayed in Grafana to anyone with trace access. Review instrumentation for query strings, authorization headers, tokens, card numbers and personal identifiers, and drop or hash them in the Collector before export. Treat this as a design check at the instrumentation stage, because removing data later from stored traces is slow and incomplete.
Protect telemetry with Collector authenticators
OpenTelemetry Collector authenticator extensions can check credentials on incoming receivers, for example with OIDC, and attach credentials to outgoing exporter requests, for example with the oauth2client extension. Use them so only approved services can send spans and the Collector can authenticate to Tempo. Confirm the extension names and configuration keys against the Collector configuration docs for the release you run.
Apply Grafana hardening settings
Grafana's security hardening guidance covers secure and SameSite cookies, a Content Security Policy, Trusted Types, hiding the version from unauthenticated users, authentication on the metrics endpoint and domain verification against DNS rebinding. Set these in the Grafana configuration that serves the Tempo data source, then verify each one in a staging instance. Record the settings in version control so changes are reviewed.
Limit who can open trace data
Trace views can expose the same operational detail as logs, so restrict Explore and dashboard access to the teams that need it. Confirm which access controls your Grafana edition provides for data sources and folders, since some team and provisioning features depend on the edition. Review the list of users with data source edit rights on a regular cadence.
Common Tempo integration failures and their fixes
This table lists the failures teams most often hit when wiring Tempo into Grafana, with the likely cause and the fix for each. Read it by symptom: find the message or empty panel you see, then check the cause column before changing configuration.
| Symptom | Likely cause | Fix |
|---|---|---|
| Data source test fails on the Tempo URL | Grafana cannot reach the Tempo query endpoint from its own network, or the URL points at the wrong port or path | Confirm the URL from the Grafana host or container, check proxy and firewall rules, and point Grafana at the query frontend address rather than the ingest address |
| Search returns no traces for a known request | Retention has removed the traces, or the spans never reached Tempo because the collector or exporter dropped them | Check the Tempo retention setting against the age of the trace, then confirm spans arrive by querying the collector or Tempo ingest metrics for the service |
| Trace ID lookup says the trace is not found | The trace ID was copied from a different system, or the trace is still being ingested and has not been flushed to storage yet | Retry after a short wait, verify the ID format matches the sending SDK, and confirm the service exports to the Tempo endpoint in use |
| Trace to logs link opens Loki with no results | Log lines do not carry the trace ID field, or the field name or label mapping in the data source does not match the one the application writes | Align the trace ID field name between the application log format and the trace to logs settings, then test with one known request |
| Trace to logs link opens the wrong Loki tenant | Loki multi-tenancy is enabled and the Tempo data source has no tenant header that matches where the logs are stored | Set the Loki data source to send the X-Scope-OrgID value used for those logs and check that the tenant can read the stream |
| Service graph panel is empty | No Prometheus data source is linked, or Prometheus does not hold the service graph metrics that Tempo generates | Link the Prometheus data source in the Tempo settings, confirm the metrics generator writes to that Prometheus, and check the time range |
| Trace to metrics panels show no data | The linked Prometheus data source does not contain the span metrics with the expected labels | Compare the metric and label names in Prometheus against the trace to metrics mapping and correct the tags in the data source settings |
| Alert on the Tempo data source never fires | The Tempo data source does not support alerting directly, so the rule has no query path that Grafana Alerting can evaluate | Create a TraceQL metrics query stored in Prometheus, then alert on that series, and verify the rule evaluates in the Alerting view |
Questions teams ask before adding Tempo to Grafana
What drives the cost of adding Tempo to Grafana?
The main drivers are the number of services that emit traces, how far their instrumentation has progressed, and how much telemetry you keep. Each service needs OpenTelemetry or another tracing library producing spans, and the collection path must reach Tempo. Costs fall into three groups: the infrastructure to run Tempo and its storage, the people who build and operate the pipeline, and the linked Prometheus and Loki sources that the trace views depend on. Infrastructure is recurring, while instrumentation and integration work is mostly one-time with ongoing maintenance as services change. Ask any vendor or consultant for a quote that separates these items, lists the services in scope, and states how long trace data will be retained.
How does the number of linked data sources change the effort?
Each link adds configuration and testing. Trace to logs needs a Loki or other log data source with a label scheme that matches the trace attributes you want to join on. Trace to metrics and the service graph need a Prometheus data source that receives service graph metrics, so someone must confirm those metrics are generated and retained. Trace to profiles and custom correlations add further work. Every link also needs a check that users can move between views without empty results. Ask a vendor to list each link in the written scope, along with the data it depends on.
How long does a Tempo integration take?
Duration depends on scope rather than on Tempo alone. A single Tempo instance receiving traces from a few instrumented services and one Grafana data source is a much smaller job than a multi-team rollout that adds trace to logs, trace to metrics, the service graph and alert rules across many services. Timing also depends on how many services already emit spans, on access approvals, and on whether your Grafana and storage platforms are already running. No fixed duration applies to every environment, so ask for a timeline tied to a written list of services, links and environments.
What does our team need to provide before the work starts?
You need to provide access and information the work cannot proceed without. This includes access to the Grafana instance and the systems where Tempo will run, a list of services to instrument or already instrumented, and the trace attributes your teams already use. Agree on label conventions for service names, environments and namespaces, because trace to logs and trace to metrics links depend on consistent labels in Loki and Prometheus. You also need named owners who can approve changes to collectors, retention and access. Ask the vendor to put these prerequisites in the written scope.
Can our in-house team build and run this without a partner?
Often yes, if the team already runs Grafana, Prometheus and a log store, and has engineers who have instrumented services with OpenTelemetry. The harder parts are consistent instrumentation across teams, label design that makes links work, and keeping the service graph and trace alerts accurate as services change. Small pilots tend to be easier to staff internally than organization-wide rollouts. Before deciding, ask a partner for a scope that shows which tasks you would still own after handover, and compare that with the skills you can staff.
Can Grafana alert directly on Tempo traces?
No. The Tempo data source does not support alerting directly. The Grafana documentation recommends TraceQL metrics written to Prometheus as the basis for trace-based alerts. This means the alert rule evaluates a time series, which is easier to route, silence and test than a raw trace query. Dashboards can keep using the Tempo data source for trace exploration. Plan for the metric pipeline as a separate deliverable, and confirm the exact configuration with your Grafana and Tempo versions before you rely on it.
Does retention change the storage and the support load?
Yes. Longer trace retention increases object storage use and the work needed to size, back up and monitor Tempo over time. Retention also affects how far back users can open traces and how far back linked logs and metrics must still exist to make the links useful. Set retention per data type in the written plan, and confirm with the storage team which backup and restore process applies. Ask any vendor to state the retention assumed in their quote and what changes if your requirements differ.
Which parts of the work are one-time and which recur?
Installation, instrumentation of services, data source configuration, link setup and the first alert rules are mostly one-time work. Recurring costs include infrastructure for trace storage and query, upgrades to Grafana and Tempo, monitoring of the pipeline, retention changes, and updates when services or label conventions change. Ask a vendor to separate these in the quote and to say whether ongoing support is offered as a retainer or handled by your team.
How DeployOpen scopes and delivers a Tempo integration
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For Grafana and Tempo integration, we start from the trace sources, log and metric stores you already run, then deliver a documented, tested data source configuration that your team can operate after handover.
How we scope
- Inventory the services, trace sources and existing Loki and Prometheus stores that the Tempo data source must link to.
- Agree in writing which trace-to-logs, trace-to-metrics and service graph links are in scope for each Grafana instance.
- Confirm the network paths, identity model and retention targets for Tempo, Loki and Prometheus before any build starts.
- Record the alerting approach, including which alerts run on TraceQL metrics in Prometheus instead of the Tempo data source.
How we build
- Deploy Tempo and the Grafana Tempo data source in your environment, then configure provisioned data source definitions as code.
- Build and test TraceQL queries against representative services, checking that trace results return from each linked store.
- Configure trace to logs and trace to metrics links, then validate that label mappings resolve correctly across Loki and Prometheus.
- Set up the service graph with a linked Prometheus data source that receives the service graph metrics it needs.
How we hand over
- Deliver the written configuration, provisioning files, data source settings and a runbook covering each linked data source.
- Walk your platform and observability team through query patterns, link behavior and the TraceQL metric alert rules.
- Document known limits, such as the lack of direct alerting in the Tempo data source, so operators know where to look.
- Confirm acceptance tests against the scope before closing the engagement.
How we support
- Engagements are fixed-scope, and ongoing support, upgrades and continuous testing are available as a retainer.
- Retainers can cover Grafana, Tempo and linked data source upgrades, with release notes reviewed before each change.
- Ask for the support boundary in writing, including who owns Tempo storage, Loki retention and Prometheus capacity after handover.
