Monitoring Backstage with OpenTelemetry: what it gives platform teams
A Backstage portal that teams depend on usually reveals its failures through complaints: a catalog entity that has not updated, a sign-in that stopped working, a template that hangs. Instrumentation lets the platform team see those problems first. The instrumentation file loads OpenTelemetry into the Backstage backend so that requests and plugin calls emit telemetry. Prometheus metrics then show how rates, errors and durations change over time, and OTLP traces show the path of a single request through the catalog, authentication and scaffolder code. Backstage is Apache-2.0 licensed and runs on PostgreSQL in production, so the monitoring stack is a separate component you choose, install and operate.
Each signal answers a different question, and none replaces the others. Prometheus collects metrics by pulling them over HTTP and stores them in a local time series database that is not clustered or replicated, with a default retention of 15 days unless you change it. OpenTelemetry does not store anything itself. It is a framework and Collector that move telemetry, so you need a trace backend such as Jaeger to view traces, and you should confirm that the Jaeger version you plan to run accepts OTLP in your environment. The Collector can run beside the Backstage pods as an agent or as a gateway for several sources, which affects where you put authentication and retries.
Settle these decisions before the portal has real users, because a baseline recorded during onboarding is the only reliable reference for telling normal load from degradation later. Decide how long metrics and traces must be kept, who receives alerts and through which channel, and where the Collector runs. Confirm that the metrics endpoint requires authentication or is reachable only from your monitoring network. Then run a test catalog refresh and a test scaffolder task and check that both appear in Prometheus and in your trace view. Metric names and alert thresholds should come from your Backstage version's documentation and your own measurements, not from memory or from this page.
Which signal answers which question
The first question is whether the process is running and answering requests. Start with the health endpoint the deployment exposes, because it gives a direct answer without analysis. Metrics show whether the target stayed reachable over time. OpenTelemetry overview explains that it generates and exports this data, but a separate backend must store it.
- Health endpoint: Confirm the exact health check path in the Backstage version you run, then point load balancer and Kubernetes probes at it.
- Scrape status: Prometheus pulls metrics over HTTP and records a failed scrape, which catches a hung process that still holds an open port.
- Startup logs: Database connection errors and configuration failures appear in logs before any metric is available.
- Recent changes: Compare the first failure time with the rollout time, image tag and configuration changes.
Before you trust these checks, stop the backend deliberately in a non-production environment and confirm that the health check and the scrape target both fail.
Traces answer the question of where one slow request spent its time. A span records a unit of work, such as a plugin route, a database query or an outbound call to a source control or identity system, and spans from one request share a trace. Metrics show that latency rose, while traces show which step caused it. Traces leave the Backstage process over OTLP to a trace backend such as Jaeger. OpenTelemetry does not provide that backend, so you run and maintain it separately and confirm that it accepts your export.
- Spans per call: Each span shows the time spent in one step, so a slow directory lookup stands out from a slow database query.
- Trace context: Confirm that requests arriving through proxies or ingress carry trace context, so their spans join the same trace.
- Sampling: Decide what share of traces to keep, because capturing every request from a busy portal increases storage.
- Latency histograms: Use histogram metrics to find the slow route, then open traces for that route only.
Before you build any dashboard, send one test request through the portal and confirm that the expected spans appear in the trace backend.
A stale entity is a processing question. The catalog reads descriptors through providers and locations, and an entity goes stale when a read or processing step stops completing. The errors recorded on the entity and the catalog's own processing state are the first places to look. Metrics help when many entities go stale together, such as after one provider stops finishing its runs. Traces help when a single refresh is unusually slow. Logs hold the provider's error text, including the source URL it tried to read.
- Entity errors: Check the processing errors on the affected entity, which usually name the failing location or processor.
- Provider runs: Confirm that each entity provider completed its last run, since one stalled provider leaves every entity it supplies out of date.
- Refresh logs: Search backend logs for the entity reference and its source URL to find the read that failed.
- Moved files: Confirm that the descriptor still exists at its location, because a moved or renamed file looks like a stale entity.
Before you alert on catalog freshness, list the catalog metrics your Backstage version exposes in its documentation and confirm each name against your own deployment.
A scaffolder run is a task made of steps, and the task log explains which step failed and what that action reported. Metrics show how failure rates change over time and across templates. Traces show whether a step waited on an external service, such as a Git platform or catalog registration. Backend logs hold the lower-level errors, including authentication and permission failures returned by the systems each action called. Most failures in this area come from publish or permission problems, so start with the action that failed.
- Task log: Open the failed step first, since its output usually names the action and the error it received.
- Failures by template: A rise in failures for one template points to a change in that template or its actions, so review recent template commits first.
- Outbound waits: Traces show whether a publish or registration step waited on a slow external API before it failed.
- Permission errors: Backend logs record the rejected request from the Git platform, which shows the token or scope problem.
Before you build a failure alert, confirm that your metrics can group failures by template name, since that grouping makes the alert actionable for the owning team.
Backend options for metrics, traces and storage
This table compares the main collection, trace backend and storage choices for a Backstage backend, with the fit and trade-off of each. Read each row against your own retention needs, team capacity and how many Backstage instances you run, then pick the simplest option that meets those needs.
| Decision | Option | Trade-off | When to choose it |
|---|---|---|---|
| Metrics collection | Prometheus pull from a /metrics endpoint | Prometheus must reach every Backstage pod or host, and the endpoint must be protected from unauthenticated callers. | Backstage runs in a network Prometheus can reach directly, and you want a simple setup with fewer moving parts. |
| Metrics collection | OTLP push to a Collector, then export to Prometheus | Adds a Collector to run, configure and upgrade, but centralizes filtering, auth and export settings. | Several services already send OTLP, or you need to add credentials or processing to outgoing telemetry. |
| Collector deployment | Agent beside each workload | Each agent adds configuration and upgrade work across every node, and offers local buffering close to the source. | Backstage runs on hosts where local buffering matters, and you can manage agents consistently across the fleet. |
| Collector deployment | Gateway shared by many sources | A single point of failure unless you run more than one, and it needs capacity planning for peak telemetry volume. | Many services send telemetry to one place, and you want one set of exporters, credentials and sampling rules. |
| Trace backend | Jaeger receiving OTLP traces | Another stateful component to store, query and upgrade, with its own retention and access settings to plan. | Your team already runs Jaeger or wants a dedicated trace UI for Backstage request paths. |
| Trace backend | Another OTLP-compatible trace store | Feature support, retention controls and access models differ, so confirm each capability with the vendor before choosing. | Your organization already standardizes on a trace platform and wants Backstage to follow that choice. |
| Metrics storage | Local Prometheus storage only | Single-node, not clustered or replicated, with default retention of 15 days, so history beyond that window is lost unless you change it. | Short-term troubleshooting and alerting are the main need, and you can accept a single node to back up and restore. |
| Metrics storage | Remote write to long-term storage | Adds an external system to size, secure and operate, and the data path must be monitored for gaps. | Dashboards or audits need metric history beyond the local window, and the storage platform is already run by your team. |
Setting up instrumentation, Prometheus and Jaeger
This sequence takes the Backstage backend from an uninstrumented process to metrics scraped by Prometheus and traces viewable in Jaeger, with a check after each stage. It is written for the platform team that owns the Backstage deployment and the observability stack. Each stage depends on the one before it, so verify each result before moving on.
Create the instrumentation file
The engineer who owns the Backstage backend creates a dedicated instrumentation file and makes sure it loads before the backend starts, so the SDK can patch libraries such as HTTP and database clients before they are used. Confirm the file name, the load mechanism and the SDK packages against the Backstage observability documentation for your release, because the exact pattern depends on the version you run. The step is done when a local start logs the SDK initializing without errors.
Enable the metrics and OTLP trace exporters
The same engineer turns on the metrics export and the OTLP trace exporter in the instrumentation file and sets a service name, so telemetry from this Backstage instance can be told apart from other services. Use the configuration keys documented for your Backstage version and avoid copying keys from older examples. Point the trace exporter at the OTLP receiver address of the Collector you will deploy in the next step. The step is done when a local request produces a trace in the exporter's debug output.
Deploy the OpenTelemetry Collector
The platform team deploys the OpenTelemetry Collector in your infrastructure, either as an agent beside the Backstage pods or as a gateway that accepts data from several sources. Configure an OTLP receiver for traces, a batch processor and an exporter to your trace backend. Collector releases ship signed artifacts, SHA256 checksums and SBOM files, so verify those before copying the image into your registry. The step is done when the Collector's health check passes and its logs show spans received from a test request. See OpenTelemetry Collector configuration for the receiver and exporter syntax.
Configure Prometheus to scrape the endpoint
The platform team adds the Backstage backend metrics endpoint to Prometheus, using service discovery in Kubernetes or a static target elsewhere. Prometheus pulls metrics over HTTP, so confirm the scrape path and port against the Backstage documentation and check the network policy between namespaces. Keep the endpoint internal or authenticated, since it reveals details about the service. Local Prometheus storage is single node and not replicated, and its default retention is 15 days, so plan remote write or backups if you need longer history. The step is done when the target shows as up in the Prometheus targets view.
Connect Jaeger to the trace pipeline
The platform team points Jaeger at trace data from the Collector, either by exporting spans from the Collector to a Jaeger OTLP endpoint or through the storage backend your deployment uses. Confirm the supported ingestion path and the Jaeger version against the Jaeger documentation for your environment before you choose one. OpenTelemetry does not include storage or visualization, so Jaeger or another backend must be selected and operated separately. The step is done when Jaeger lists the Backstage service in its service search.
Verify the metrics endpoint responds
The engineer runs a few catalog refreshes and scaffolder tasks, then checks that the metrics endpoint returns Backstage metric families. Run a PromQL query for one of those metrics and confirm it returns series from every backend replica. Check that the service and environment labels you set are present, so dashboards and alerts can filter by environment. The step is done when the query returns current data for each replica and no replica is missing from the results.
Confirm traces appear end to end
The engineer calls a Backstage backend API, then opens Jaeger and follows that single request through the backend and its outbound calls to the database and external systems. Check that span names and attributes identify the plugin and route, and that no tokens, credentials or personal data appear in attribute values. The Collector's own logs should show no export errors for the same request. The step is done when one request shows a complete trace with the expected child spans.
Check histogram units before alerting
Before building alerts, the engineer inspects a duration histogram and confirms its unit suffix and bucket boundaries match what the dashboard assumes, since a unit mismatch produces latency graphs that look plausible but are wrong. Compare the value Prometheus reports for a known slow operation with the duration of the matching span in Jaeger. Confirm the metric names and units in the Backstage observability documentation for your release. The step is done when the latency reported for that operation agrees with the span duration within the histogram bucket resolution.
Catalog and scaffolder metrics worth alerting on
Most alerts for a Backstage portal fall into five groups: catalog freshness, scaffolder reliability, backend request health, database connectivity and the health endpoint. Each group answers a different question, so each needs its own threshold. Set thresholds only after you have observed the metrics your instrumentation actually emits for a few weeks.
Catalog processing failures and staleness
Catalog failures mean entities go missing or stop updating, and engineers stop trusting the portal. Alert on the rate of processing errors and on the age of the last successful refresh per location, not on single failures, because a broken descriptor in one repository is usually a team problem rather than a platform outage. Confirm the exact catalog metric names and labels in your build before writing rules, and tune the staleness limit to how often your entity providers actually run. Route the alert to the owning team when a single location is failing.
Scaffolder task failures
A failed scaffolder task leaves a half-created repository or an unfinished service, so the cost lands on a developer who is trying to start work. Track the ratio of failed to completed tasks per template, and alert when one template starts failing more often than its own baseline. A global threshold hides a single broken template behind healthy traffic. Pair the alert with the step that failed, because publish and permission steps fail for different reasons and need different owners.
Backend request error rates and latency
Error rates and latency on backend routes show whether plugins are serving requests within acceptable time, and they often rise before users complain. Alert on sustained 5xx responses and on high percentile latency for the routes engineers use most, such as catalog reads and search. Use histogram buckets to set these thresholds, and remember that histogram units are a common source of mistaken alert values, so confirm the unit on each metric before you compare it with a number. Keep the windows long enough to avoid paging on short spikes.
Database connectivity
Backstage keeps a separate logical PostgreSQL database for each backend plugin, so one lost connection can affect catalog, scaffolder and auth at the same time. Alert on failed connections and on connection pool exhaustion, and check the database host's own availability alongside the Backstage metrics. A connectivity alert should point the on-call engineer toward the database and network path first, since restarting the portal rarely fixes it. Confirm which pool metrics the Knex integration exposes in your version.
Health endpoint failures
The health endpoint reports whether the backend process is up, and orchestrators use it to decide when to restart a pod. Alert when the endpoint fails for longer than a restart cycle would explain, and keep this separate from the readiness signal used during rollouts, so a slow catalog start does not look like an outage. Confirm the endpoint path and the probe settings in your deployment, and review them after every upgrade because startup time changes as the catalog grows.
Histogram units and why dashboards go wrong
A histogram only means something when the unit it records, the unit the Collector exports and the unit the dashboard query assumes all match. Latency histograms are the usual casualty: the instrumentation may record milliseconds while the panel labels its axis as seconds, or a query divides by 1000 when the metric never needed scaling. The panel still looks plausible but is off by a factor of a thousand. Set the unit in the Backstage instrumentation file, where the OpenTelemetry instrument is defined, and confirm it there before writing any alert rule. Record that unit next to the rule so the next editor does not have to reverse-engineer it from a graph.
Between the source and Prometheus sits the OpenTelemetry Collector, which can rename or transform metrics depending on its pipeline and exporter settings. Review the Collector configuration for any processor that changes names, and compare the series names Prometheus stores with the names you expect from the source. Histogram buckets are cumulative and labeled by upper bound in the le label, so a boundary defined in the wrong unit shifts every value that histogram_quantile computes. Confirm that a bucket boundary sits near each threshold you plan to alert on. Without one, the alert depends on interpolation between distant buckets.
Prometheus values reliability over precision, so a percentile on a panel is an estimate interpolated between buckets, not a measured value. Treat it as approximate and alert on a bucket boundary you have verified exists. Before trusting a new panel, generate a known slow request in a test environment and check that the panel shows the magnitude you expect. Repeat the check for count and rate panels, since a unit error there is easy to miss. Confirm the exact metric names, bucket layout and units with the Backstage and Collector versions you run, because these can change between releases.
Securing the telemetry path
Telemetry carries sensitive detail about your portal, its users and its catalog, and each hop from Backstage to the Collector, Prometheus and Jaeger is a place where it can leak or be tampered with. The cards below cover the access controls, data hygiene and network boundaries to set before the monitoring stack goes into shared use.
Restrict the metrics endpoint
Backstage exposes Prometheus metrics on an HTTP endpoint, and anyone who can reach it can read request paths, entity counts and error patterns. Decide whether the endpoint is served on a separate port or path, limit which network sources can scrape it, and confirm that Prometheus authenticates with a credential or that the endpoint sits behind a controlled network path. Test the restriction with an unauthenticated request from outside the scrape network before you rely on it.
Authenticate Collector receivers
An OTLP receiver that accepts spans from any client lets anyone inject false traces or flood the pipeline. Use a Collector authenticator extension, such as the OIDC option, on the receiver so that only the Backstage backend and approved workloads can send data. Review the configuration best practices page before exposing the receiver beyond the cluster, and confirm which authenticator each receiver uses in your environment.
Add credentials on exporters
Exporters that send traces or metrics to a backend need their own credentials, and a static token in the Collector configuration is easy to leak through version control or logs. The oauth2client extension can attach tokens to outgoing requests so that the backend accepts only authorized senders. Store the client secret in your secrets manager, and ask your platform team how rotation will reach running Collectors.
Keep sensitive data out of spans and logs
Backstage instrumentation records request URLs, headers and attributes that can contain tokens, user emails or entity details with personal data. Review what the auto-instrumentation and any custom spans capture, drop query strings and authorization headers, and do not log secrets from configuration on startup. Sample a set of traces and logs after each change to confirm that no credentials or personal data appear in Jaeger or the log store.
Segment Jaeger and Prometheus
Prometheus and Jaeger hold operational data that reveals your architecture and user activity, so place them on a monitoring network that only the Collector, Grafana and approved operators can reach. Restrict the Jaeger query interface to authenticated staff, and keep the storage backend off any path that users of the portal can reach. Document these flows in your network diagram so that the reviewers of your architecture can trace them.
Back up Prometheus as single-node storage
Prometheus local storage is not clustered or replicated, so the loss of its disk loses the metric history that your alerts and capacity reviews depend on. Plan backups of the data directory or a remote write target, and test a restore, because the default retention is 15 days and older data is deleted. Decide with your team whether long-term storage will sit in a remote system and who owns that system.
Questions teams ask before monitoring Backstage
How much effort does monitoring Backstage take?
Effort depends mostly on how many signals you collect and how much of the monitoring stack you already run. A team with a working Prometheus and an OpenTelemetry Collector mainly adds Backstage instrumentation, scrape configuration, dashboards and alert rules. A team starting from nothing also has to install and size Prometheus, the Collector and a trace backend, and design storage. Three drivers matter most:
- Number of signals: HTTP metrics only, or catalog and scaffolder metrics plus traces.
- Existing components: whether Prometheus and the Collector already run, who owns them and whether they accept OTLP.
- Retention: Prometheus keeps data for 15 days by default, and longer retention usually means a remote storage design.
Ask for the effort split by these items in any written scope.
Can our team set up Backstage monitoring without a partner?
Yes, if you have engineers who already run Prometheus or an OpenTelemetry Collector and can own the Backstage side. The core steps are enabling instrumentation in the backend, exposing metrics, sending traces to a Collector and writing alert rules. Teams without that experience usually struggle with the parts that are easy to get wrong: histogram units, label cardinality, Prometheus local storage that is single-node and not replicated, and deciding which catalog and scaffolder metrics should page someone. A partner can work from a written scope to fill those gaps or review your design. Be clear about who maintains the setup after launch, because Prometheus and the Collector both change on regular release cycles.
What does the work need from our environment?
Settle access and ownership before the engagement starts. Typical needs are:
- Cluster or host access: rights to deploy Prometheus, the Collector and Backstage configuration in the namespaces you use.
- DNS and network paths: Prometheus must reach Backstage to scrape metrics, and Backstage must reach the Collector's OTLP endpoint.
- Storage: persistent volumes for Prometheus and the trace backend, sized to your retention choice.
- An alert owner: a named team that receives Alertmanager notifications and agrees on thresholds.
Confirm which of these you already have. Prometheus should point at every Alertmanager instance rather than at a load balancer, so the alert route needs a defined owner from the start.
How long does it take to set up Backstage monitoring?
The timeline depends on scope and on your environment, and no fixed duration applies to every setup. The main drivers are how many components are new, how many environments need the setup (development, staging and production) and how quickly access, DNS and storage requests move through your change process. Alert tuning often takes longer than wiring metrics, because thresholds only mean something once they have seen real traffic. DeployOpen's engagements are fixed-scope, so the timeline and milestones are agreed in writing before work starts. Ask for milestones that match your environments and change windows.
Do Prometheus and Jaeger cost money to run?
Prometheus has no paid edition from its project, and its GitHub repository lists it under Apache-2.0, so the license itself costs nothing. Its costs are infrastructure, storage and the people who run it. Jaeger is also open source, but the sources for this page do not cover its license, current release or hosting options, so confirm those on the Jaeger project site before you choose it. Recurring costs include compute, disk for metrics and traces, backups, upgrades and on-call time. One-time costs include design, initial setup and dashboard work. High availability for metrics adds cost because local Prometheus storage is not clustered or replicated. Hosted dashboard and telemetry services are a separate decision with their own pricing.
What should we ask for in a written quote for this work?
Ask the quote to list each item in the scope, so you can compare proposals on the same basis:
- Signals: which metrics and traces are covered, which Backstage endpoints and plugins, and which environments.
- Components: whether Prometheus, the Collector, Alertmanager and the trace backend are installed, reused or only configured.
- Deliverables: the instrumentation file, scrape configuration, dashboards, alert rules, Docker image changes and a runbook.
- Assumptions: access, DNS, storage sizes, retention and who owns alerts after handover.
- Exclusions: upgrades, ongoing support and any audit evidence work.
- Change handling: how scope changes are priced and whether ongoing support is a separate retainer.
How do we keep the monitoring setup working after launch?
Monitoring is a running service, so plan for upgrades and periodic review. A new Prometheus minor release starts every six weeks, and a minor release generally stops receiving bug fixes after that period. Long-term support lines receive bug fixes for one year, so choose a line your team can keep current and confirm the current LTS line on the Prometheus release cycle page before you pin a version. Backstage ships a monthly main release line and a weekly next line, so instrumentation changes can arrive with routine upgrades. Review alert rules after significant traffic changes. Upgrades, support and continuous testing can run under a DeployOpen retainer, or your own team can own them.
How DeployOpen works on this
DeployOpen approaches Backstage monitoring as a bounded piece of work inside infrastructure you control. We scope which backend signals, Collector pipelines and dashboards your team needs, build them in your environment, and hand over documentation and runbooks your engineers can operate.
How we scope
- We confirm which Backstage backend plugins, catalog and scaffolder metrics, and trace sources the monitoring must cover.
- We agree where Prometheus, the OpenTelemetry Collector and Jaeger run, and who owns each component.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before any build work starts.
How we build
- We add the instrumentation file, configure OTLP trace export and Prometheus scraping for the Backstage backend.
- We define Collector pipelines, histogram units and alert rules for catalog and scaffolder signals your team agrees on.
- We test the setup in a non-production environment, then ship the configuration in your Backstage Docker image.
How we hand over
- You receive the configuration, pipeline definitions, alert rules and dashboards as files in your own repositories.
- We document the signals, expected ranges, alert ownership and steps for adding a new metric or trace source.
- Your engineers walk through a handover session and a verification checklist before the engagement closes.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after the initial handover.
- A retainer can cover Prometheus, Collector and Jaeger version updates, with changes tested before they reach production.
- Ask any provider to confirm component versions, support windows and who receives alerts before you sign.
