How Pyroscope adds continuous profiling to a self-hosted Grafana
A platform team with Grafana already running often wants to know where CPU and memory go inside its services. Grafana Pyroscope is a core Grafana data source in the profiles category, so it sits beside your existing dashboards rather than in a separate tool. Engineers reach profiling data in two ways: through the Pyroscope data source for direct profile queries, and through span links from the Tempo data source, which can embed flame graphs for a specific trace. Profiles are collected through Grafana Alloy, which handles profiles alongside metrics, logs and traces in one collector.
The integration does not replace tracing or metrics. A profile shows which functions consume CPU or allocate memory during a period, but it does not show the request path across services or the trend of a service-level indicator over time. Keep Tempo for traces and your metrics store for trends and alerts, and treat profiles as the step that explains a slow span or a rising resource curve. Plan that split before the build starts, because it decides which dashboards link to which data and who owns each data source.
DeployOpen plans and builds this stack inside your infrastructure: the Pyroscope service, the Grafana data source and Tempo links, Alloy collection on your hosts or clusters, and the retention and access settings for profile data. Before scoping, confirm with your team which services can expose profiling endpoints, how much profile data your storage can hold, and who may read profiles that can reveal internal code paths. Ask for these items to be written into the scope so the profiling work can be checked against them.
What a Pyroscope integration needs before the build starts
A profiling integration touches four parts of your stack: the Grafana instance, the Pyroscope service and its storage, the collector that gathers profiles, and the network paths between them. Settle each one before the build starts, because a choice made for Grafana or the network later constrains how Pyroscope and Alloy can be deployed.
Production Grafana database
Grafana keeps its configuration in SQLite 3 by default, but the installation docs state that SQLite does not scale for production. Plan MySQL 8.0 or later, or PostgreSQL 12 or later, for the instance that will host the Pyroscope data source and the Tempo links. Confirm that your Grafana version supports the database you pick, and check the high availability requirements if more than one Grafana node will serve profile dashboards.
Pyroscope service and storage placement
Decide where Pyroscope runs and where its profile data is stored, in the same infrastructure boundary as your other telemetry. Profile volume grows with the number of services, sampling rate and retention you choose, so ask the vendor for sizing guidance for your expected workload. Record which storage backend, encryption and backup approach your data governance requires before you choose a deployment target.
Alloy as the collection path
Grafana Alloy is the collector that gathers profiles from your workloads and forwards them to Pyroscope, alongside metrics, logs and traces. Decide whether Alloy runs as an agent on each host or node, or as a central gateway, and who owns its configuration. Confirm which profile types your language runtimes and Alloy release can collect, since support differs by runtime.
Tempo for span-to-profile links
Profiles linked from trace spans need a Tempo data source in the same Grafana instance, and the Tempo data source is what embeds Pyroscope flame graphs in a span view. If you only need standalone profile exploration, you can skip Tempo. If you do want span links, confirm that your traces carry the labels Grafana uses to match a span to a profile, and that Tempo and Pyroscope share a compatible time range.
Network paths from workloads to the collector
Map every path from profiled workloads to Alloy and from Alloy to Pyroscope, including any firewall rules, proxies and cross-cluster routes. Also map the path from users to Grafana. Confirm which traffic must stay inside a private network and which can cross a boundary, because profiles can expose function names and internal code paths. Plan TLS and authentication on these paths before you add collectors.
Versions and profile types to confirm
Pyroscope, Alloy, Grafana and your language runtimes all change independently, and profile support depends on the combination. Confirm the exact Pyroscope and Alloy versions for your release, the profile types each runtime emits, and the compatibility matrix with your Grafana version against the vendor documentation before you write the deployment plan. Record the results so the build team and the operators work from the same list.
Configuring Pyroscope, Alloy and the Grafana data sources in sequence
This sequence takes Pyroscope from a running server to a working profiling path in Grafana, with each step confirmed before the next one depends on it. The end result is profiles arriving from your services, queryable in Grafana, and linked from Tempo spans.
Confirm the target environment and storage
Your platform team decides where Pyroscope runs, such as a VM, container platform or Kubernetes cluster you control, and how its storage and network access will be handled. Grafana itself should already be running on a production-grade database, since the installation docs say SQLite does not scale for production. Done means you have a written list of the hosts, network paths and access rules for Pyroscope, Alloy and Grafana, approved by the team that owns each one.
Deploy Pyroscope and verify its endpoint
The infrastructure or platform team installs Pyroscope in the chosen environment and sets its storage location and retention settings. Confirm the server starts cleanly, listens on the address your collectors will use, and answers a basic health request from a host on the same network. Done means the service stays up after restart and a test request from the Alloy host succeeds.
Add the Pyroscope data source to Grafana
A Grafana administrator adds Pyroscope as a core data source, either through the Grafana UI or as a provisioned file loaded at startup. Provisioning keeps the data source definition in version control, which suits teams that manage Grafana as code. Done means the data source test in Grafana reports a successful connection to the Pyroscope URL.
Route profile collection through Alloy
The platform engineer configures Grafana Alloy on the hosts or pods running your services so it scrapes or receives profiles and forwards them to Pyroscope. Alloy also handles metrics, logs and traces in the same collector, so keep its configuration in the same review process you use for other telemetry. Done means Alloy reports its profiling components as running, with no repeated errors in its logs.
Confirm profiles arrive in Grafana
Open the Pyroscope data source in Grafana's Explore view and query the application you instrumented, selecting a time range where the service was active. A flame graph that renders with stack frames is the check that collection, storage and Grafana are working end to end. If the query returns nothing, check the Alloy output, the Pyroscope endpoint and the labels used in the query before going further.
Enable profile links from Tempo
A Grafana administrator opens the Tempo data source settings and sets the linked Pyroscope data source, so span views can reference profile data for the same service and time window. Save the change and confirm the Tempo data source test still passes. Trace-to-profile links depend on matching service names between traces and profiles, so confirm the names agree before you rely on them.
Verify a span opens an embedded flame graph
Choose a trace from a service that is both traced and profiled, open a span in Grafana, and use the profile link to open the embedded flame graph for that time window. Done means the flame graph matches the service and period of the span. Record which service names and labels were needed, so the same check can be repeated after upgrades or when new services are onboarded.
Where profiling cuts CPU and memory cost
Profiling shows where CPU and memory go before you decide to buy more of either. A flame graph built from profiles collected under real load displays which functions consume CPU time or allocate memory, with the widest frames marking the largest share. In Grafana Pyroscope, you can open these flame graphs directly in the profiles data source and compare them across time ranges, such as a window before and after a release or a period of peak traffic. That turns a vague sense that a service is expensive into a short, ranked list of code paths that deserve attention, which is the narrowing step that comes before any capacity change.
The savings are workload dependent. A hot function in one service may be a small share of the bill in another, and a fix that reduces allocations may do nothing for a service that is bound by I/O or by a downstream dependency. Profiles also reflect the sampling and collection settings you chose in Alloy, so confirm how the profiling agent samples and which profile types it collects for each runtime you run before you read anything into the widths of the frames. Treat the flame graph as a hypothesis about where cost comes from, not as a measured saving.
Measure before and after in your own environment. Record CPU usage, memory use and request latency for the affected service over a comparable traffic window, capture profiles for the same window, make one change at a time, then repeat the measurement under similar load. Keep the baseline and the change in the same change record so reviewers can see what moved and why. If the flame graph points at a function you cannot change, such as a vendor library or a runtime setting, the next decision is usually a configuration or sizing change, and that choice should also be checked against the same before and after numbers.
Mapping services, labels and profile types between Alloy, Pyroscope and Tempo
This table lines up where each workload identifier is set, how Alloy and Pyroscope carry it, and what Tempo must expose for a span to open the matching flame graph. Read it left to right before you write any collector config, and confirm exact attribute names against the current Tempo data source docs before you rely on them.
| Identity element | Where it is set | What must match downstream |
|---|---|---|
| Service name | Application instrumentation or resource attributes on the process | Same value used as the Pyroscope application name and the Tempo service name on spans |
| Namespace | Kubernetes labels or deployment metadata collected by Alloy | Carried as a profile label so teams can filter flame graphs to the same namespace as traces |
| Environment | Deployment configuration, often a resource attribute such as a deployment environment key | Identical across production and staging so a span never resolves to a profile from another environment |
| Alloy scrape targets | Discovery and relabeling rules in the Alloy profiling pipeline | Relabeled targets must keep the same service identity as the instrumented process, or links break silently |
| Pyroscope application name | Profile ingestion labels from Alloy or the SDK | Must equal the service identity Tempo uses, including the case and separator conventions |
| Profile types | CPU, allocation, heap and other types the runtime exposes | Choose the types the Tempo link should open, and confirm which profile types the Tempo data source can embed |
| Tempo span attributes | Span resource and span attributes emitted by the tracer | Tempo data source settings map span attributes to Pyroscope labels; confirm exact attribute names in the current Tempo docs |
| Validation span | A test request traced end to end | Open the span in Grafana and confirm the flame graph shows the expected service and time range |
Securing profile data and access
Profiles reveal how your code runs, including function names, file paths and internal service structure, so they deserve the same access controls as your logs and traces. The cards below cover who can query profiles, how Grafana signs users in, how Alloy talks to Pyroscope, and which Grafana hardening settings apply to the instance.
Decide who can query profiles
Flame graphs expose function names, library versions and call paths, which can reveal internal architecture to anyone with a Viewer role. Map which teams need profiles for which services before you grant access, and treat the Pyroscope data source like any other source that holds code-level detail. Check that viewers outside engineering have no reason to see production stacks, and confirm how your Grafana edition scopes data source permissions.
Sign users in through your identity provider
Configure Grafana to use your enterprise identity provider so access follows your joiner and leaver process rather than local accounts. Generic OAuth, Entra ID, Okta, GitHub, GitLab, Google OAuth, LDAP, auth proxy and JWT are available in every edition. SAML, team sync, active sync and SCIM provisioning are Grafana Enterprise and Grafana Cloud features, so confirm early whether group-to-team mapping for profile access depends on them.
Encrypt traffic from Alloy to Pyroscope
Alloy pushes profiles to Pyroscope over the network, and profile payloads carry the same sensitive detail as the flame graphs. Terminate TLS on the Pyroscope endpoint, configure Alloy with the certificate authority it must trust, and restrict which hosts may reach the ingest port. Confirm with your platform team whether mutual TLS or a network policy is required for collectors that run on other clusters.
Protect the Pyroscope ingest and query paths
Decide who may write profiles as well as who may read them, since an open ingest endpoint lets any workload inject data into your tenant. Where multiple teams share one Pyroscope, separate tenants and enforce the tenant identifier at the proxy or gateway in front of the service. Ask your team to document which network segments reach the query API and which service accounts hold write access.
Apply Grafana security hardening settings
Hardening settings apply to the Grafana instance that renders profiles, not to Pyroscope itself. Use secure and SameSite cookies, a Content Security Policy, Trusted Types, hide the version from unauthenticated users, require authentication for the metrics endpoint, and enable domain verification against DNS rebinding. Test flame graph panels after changing the Content Security Policy, because plugin scripts can be blocked by a strict policy.
Review access and retention together
Profiles that stay for long periods widen the exposure window if an account is compromised or a role is over-granted. Set a retention period that matches your troubleshooting needs, then review who holds access on the same schedule as your other observability tools. Ask your team to record the retention setting and the access review owner in the runbook so both are checked during audits.
Why profiles or span links fail to appear
This table lists the failures teams most often hit when profiles or Tempo span links stay empty, with the likely cause and the check that confirms it. Work through the rows in order, since a broken collection path hides every later symptom. The main takeaway is that most empty views trace back to the collector, labels or data source URL rather than to Pyroscope itself.
| Symptom | Likely cause | Fix and what to check |
|---|---|---|
| No profiles in Pyroscope | Alloy is not scraping or pushing profiles to the Pyroscope endpoint, or the component configuration points at the wrong address. | Confirm Alloy's profiling components are running and reporting no delivery errors, then check that the write endpoint matches the Pyroscope service address your network allows. |
| Profiles exist but the Tempo link is empty | Labels on profiles do not match the service and span attributes Tempo uses for the link, so no profile matches the selected span. | Compare the service name label on profiles with the service name on spans for the same workload, and align them in Alloy relabeling or in the application before tuning anything else. |
| Flame graph panel shows an error or no data | The Tempo data source is linked to a Pyroscope data source whose URL is wrong, unreachable from the Grafana server, or pointed at a different tenant. | Open the Pyroscope data source settings in Grafana, confirm the URL resolves from the Grafana host, and check the tenant or organization header if multi-tenancy is enabled. |
| Profiles appear only for some services | Collection targets are incomplete, often because discovery rules cover some namespaces, hosts or processes and not others. | List the targets Alloy discovers for each workload and compare them with the services expected in Grafana, then widen selectors or add the missing targets. |
| Collection slows or adds load on hosts | Over-collection: sampling rates or profile types are set higher than the investigation needs, or many workloads are profiled at once. | Start with the profile types your team uses, review the sampling settings with the platform owner, and widen coverage only after measuring the impact on the monitored hosts. |
| Pyroscope storage fills faster than planned | Retention was not set, profile volume grew with new services or labels, or high-cardinality labels multiplied series. | Review retention settings and label cardinality, confirm the volume estimate with the Pyroscope documentation for your version, and set a storage alert before the disk fills. |
| Span links work for some traces but not others | Spans lack the profiling labels or the time range of the trace falls outside the retained profile window. | Check that instrumented spans carry the same identifying attributes as profiles and that retention covers the time range users investigate. |
Questions teams ask before profiling is rolled out
What drives the cost of running Pyroscope beside Grafana?
The main cost drivers are how many services and environments you profile, how much profile data you keep, and how much collector capacity Alloy needs on each node. Each service you instrument adds profile volume, so the count of workloads and clusters sets the storage and compute footprint. Retention multiplies storage, so a shorter window for development and a longer one for production is a common split to test. Costs are one-time for design, instrumentation and rollout, and recurring for hosting, storage, upgrades and on-call time. Ask any quote to itemize instrumentation count, environments, retention periods, the Alloy footprint per node and who operates the stack after handover.
How long does a Pyroscope rollout take?
Duration depends on scope: the number of services and languages to instrument, whether Pyroscope is new or joins an existing Grafana and Tempo setup, and how many environments must be covered before production. A pilot on a few services is usually the fastest way to learn what the data looks like and what the team needs to change. Because most engagements are fixed-scope, DeployOpen agrees the timeline in writing before work starts. Ask for a written scope with phases, dependencies on your change windows and the acceptance checks for each phase rather than a general estimate.
What does the client need to provide before work starts?
You need to provide access and a clear list of what will be profiled. That means accounts or clusters where Alloy and Pyroscope will run, a Grafana instance with access to add data sources, and a named list of workloads and their languages. You also need owners for the labels that identify each service, team and environment, because profiles without consistent labels are hard to filter. Confirm change approval paths, network rules between collectors and the Pyroscope endpoint, and who can restart workloads to enable profiling. Missing access is one of the most common causes of schedule slip.
Can our in-house team roll out Pyroscope without a partner?
Yes, many teams can, provided they already run Grafana and a collector, and have engineers comfortable with application runtimes and performance tuning. The work is mostly configuration, label design and integration with traces, which is manageable for a platform team with time to spare. The harder parts are often sizing the backend, agreeing label conventions across teams and keeping the setup patched. Assess your team against those skills and your available hours. If you choose a partner, DeployOpen plans, implements and documents the work in your own infrastructure so your team can run it afterward.
What goes wrong with profiling in production?
The most common problems are excessive profile volume, missing or inconsistent labels, collector resource pressure on busy nodes and gaps in coverage when workloads restart. Profiling adds overhead to the systems it observes, so sampling and the set of profiled workloads need checking under real load. Label drift makes flame graphs hard to compare across releases. Verify that flame graphs appear for each target service, that the Tempo link opens the right profile for a span and that Alloy restarts do not drop coverage. Build these checks into acceptance criteria rather than discovering them after launch.
How does Alloy affect the footprint of the stack?
Alloy runs as the collector that gathers profiles and forwards them to Pyroscope, so every node or cluster that profiles a workload carries an Alloy process. Its resource use grows with the number of targets and the volume of data it handles. Alloy also handles metrics, logs and traces, so one collector can replace several agents, which can reduce operational overhead even though each instance needs CPU and memory. Ask for a measured footprint per node in your own environment during the pilot, and confirm how Alloy configuration will be version controlled and rolled out.
Does profiling need security and compliance review?
Yes. Profiles can reveal function names, file paths, library versions and sometimes data embedded in call stacks, so access to Pyroscope and Grafana should follow your existing role model. Restrict who can query production profiles, protect the endpoint that collectors write to and review how long profiles are retained. Profiling evidence can support operational and capacity records, but it does not on its own satisfy any framework requirement. Confirm your data handling and retention rules with your security and compliance owners before the pilot begins.
How DeployOpen scopes and delivers Pyroscope in your environment
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For Pyroscope beside Grafana, that means agreeing in writing which services are profiled, where Pyroscope and its storage run, how Alloy collects profiles, and how flame graphs connect to your existing Tempo traces.
How we scope
- Inventory the services, languages and hosts where CPU and memory profiling would answer a real question
- Confirm the Grafana, Tempo and Alloy versions already running and the data source links to be built
- Agree in writing the deliverables, storage retention, access model and acceptance checks before work starts
How we build
- Deploy Pyroscope in your infrastructure and add it as a core Grafana data source through provisioning files
- Configure Alloy to collect profiles from the agreed workloads and route them to Pyroscope
- Set up the Tempo data source link so spans open embedded flame graphs, then test each path end to end
How we hand over
- Document the architecture, configuration files, collection targets and the steps to add a new service
- Walk your platform and application teams through reading flame graphs and checking collection health
- Deliver runbooks for storage sizing, retention changes, upgrades and removing a collection target
How we support
- Engagements are fixed-scope, and changes to the agreed scope are written up before they are built
- Ongoing support, upgrades and continuous testing are available as a retainer
- Confirm with the Pyroscope and Alloy release notes which upgrade steps your version needs before each change
