How to measure time saved by Backstage templates
Measuring time saved by Backstage templates means combining two numbers: a per-template estimate of how long a task takes by hand, and a count of how often the template actually ran. Multiply the two and you have a defensible figure for funding reviews. Software Templates are one of the core built-in features of Backstage, so every portal that scaffolds services already has the raw data needed, but the time-saved figure is an estimate, not a measured outcome. Say so in the report, or a sceptical reviewer will discount the whole number. Teams usually need this when they must justify the platform budget, when they choose which templates to build or improve next, and when they want evidence before retiring a manual provisioning process.
Record the estimate on each Template entity using the backstage.io/time-saved annotation, and agree on how it was derived before you publish it. Ask the platform owner and two or three teams that use the template to walk through the manual steps it replaces, including ticket handoffs, repository setup, pipeline configuration and catalog registration, and record the typical hours saved per run. Review the estimate when the template's steps change, and keep a dated note of the basis so the figure can be explained later. Confirm with your Backstage version that the annotation is read by the reporting you plan to use, since presentation of annotations in the UI and in any export depends on your installed plugins and configuration.
Count runs from the scaffolder task list in the Backstage UI, and check whether your deployment exposes scaffolder metrics through your telemetry stack, since the names and labels available depend on the version you run. Filter counts by template and by period, and separate successful runs from failed or cancelled ones, because a failed run saved no time. Report the result as runs multiplied by the agreed estimate, with a range rather than a single figure, and state the period, the templates included and the assumptions behind each estimate. Ask your finance or delivery lead which unit of effort they accept, such as engineer hours or cost per hour, before the first review so the number is not reworked later.
Ways to count template runs and time saved
This table compares four ways to count template runs and estimate time saved: the backstage.io/time-saved annotation, counting completed runs in the task list, scaffolder metrics exported through your observability stack, and an external analytics store. Read each row against your team size and staffing, because the cheapest option gives a rough estimate and the most accurate option needs ongoing engineering time.
| Approach | What it measures and who maintains it | Accuracy limits | Fits small or large platforms |
|---|---|---|---|
| Time-saved annotation | A value you set in the template metadata to estimate manual effort avoided per run. The platform team writes and reviews it, and it changes only when someone edits the template. | It is an estimate, not an observation. It does not reflect actual run outcomes, failed attempts or whether teams used the output. | Small platforms with a handful of templates. Easy to start, but hard to defend to funders without other evidence. |
| Completed runs from the task list | Counts of scaffolder tasks by template and final status, visible to users in the Backstage UI. Maintained by the platform team through normal portal operation. | Shows runs, not value. Repeated retries, dry runs and test runs can inflate counts unless you filter them. Task history depends on your retention settings. | Small and mid-size platforms that need a quick count for a monthly review. Confirm retention and how you export or query task data. |
| Scaffolder metrics through the observability stack | Metrics emitted by the scaffolder backend, such as task counts and durations, sent to your monitoring tools. Maintained by the team that runs the Backstage deployment and its monitoring. | Measures activity and timing, not business benefit. Labels and metric names depend on the Backstage version you run, so confirm them against your release before building dashboards. | Larger platforms that already run metrics and traces for Backstage. Useful for trends, alerting on failures and capacity planning. |
| External analytics store | Exported run records and template metadata loaded into a warehouse or analytics tool, joined with repository, ticket or delivery data. Maintained by a data or platform analytics owner with its own pipeline. | Accuracy depends on the export job, identity mapping and how you define a completed or valuable run. Joined data can drift from the source when schemas change during upgrades. | Large platforms with several business units that must justify funding. Requires a named owner for the pipeline and agreed definitions of time saved. |
Setting up measurement for your templates
This sequence turns template usage into a number you can defend: each template gets an agreed manual baseline, a documented time-saved value, verified run counts and a reporting query that funders can rerun. Work through the steps in order, because each one depends on the output of the step before it.
Agree the manual baseline for each template
For each template, the platform owner meets the team that currently performs the same work by hand and writes down the steps, the people involved and the typical elapsed time before the template existed. Keep one short baseline note per template, stored with the template's source so changes are tracked in Git. Done means every template in scope has a baseline note signed off by its owning team, and any template without a clear manual equivalent is flagged for exclusion rather than given a guessed value.
Add the time-saved annotation with its assumption
Add the backstage.io/time-saved annotation to each template's metadata, using the value agreed in the baseline note. Record the assumption next to the value in the template description or the baseline note, including what the estimate covers and when it was agreed. Done means each template renders the annotation in its metadata, and a reviewer can trace every value to a written assumption. Confirm the annotation key and value format against the current Backstage documentation for your release before rolling it out.
Confirm completed runs appear in the task list
Run each template once in a non-production environment and open the scaffolder task list in Backstage. Check that each run shows the template name, the status and the user who started it. Done means the task list shows the expected number of completed runs per template for your test runs. If a run is missing or shows the wrong template, fix the template reference or the permissions before going further, because every later count depends on this list.
Enable and verify scaffolder metrics
Ask the platform team to confirm that scaffolder metrics are enabled in the deployed Backstage backend and that they reach the metrics backend you use for dashboards. Check which metric names and labels your release exposes, and record them in the reporting notes. Done means a set of test runs increases the scaffolder counters by the same number you counted in the task list, and the metric labels identify the template. Where the counts differ, investigate retries and dry runs first, since they can inflate one source and not the other.
Reconcile dry-run and test counts
Compare the count of test runs from step three with the metric values from step four and with any dry-run records. Dry runs should be excluded from the value calculation, so confirm they are labeled separately in your data before you rely on the totals. Done means you can explain every difference between the task list and the metrics for the test period, and the reconciled figures are written down with the date of the test. If the numbers cannot be reconciled, do not publish a value yet.
Build the reporting query
Write a query, or a saved report in your metrics or analytics tool, that multiplies the run count per template by its time-saved annotation value and totals the result for a period. Store the query in version control alongside the baseline notes so it can be rerun and reviewed. Done means the query returns the same totals as your reconciled test figures when run against the test period. Keep the assumptions visible in the output so funders see what the number is built on.
Review the first report with template owners
Share the first report with each template owner and the people who fund the platform, and ask owners to confirm the run counts and baseline assumptions for their templates. Record any corrections and rerun the query after changes are applied. Done means each template owner has confirmed their line, the corrections are in version control, and the funding group has a report that states its assumptions and the period it covers.
Reporting results to different funders
Executive sponsors usually fund the platform as a portfolio and want to know whether it is returning its cost in time for the teams that use it. They need a short, stable view: total runs of the templates that matter most, the estimated hours those runs represent, and how that compares with the platform's running cost. Keep the report to one page per quarter so trends are visible without drowning the reader in per-template detail.
- Headline figure: a single estimated time-saved total for the period, shown with the number of template runs it is based on.
- Trend over time: the same measure for the last several periods, so a flat or falling line is visible early.
- Coverage: the share of templates that carry a time-saved estimate, so the total is not read as complete when it is not.
- Cost context: the platform's running cost set beside the estimate, with the people and infrastructure costs named.
- Caveats to state: the figure is an estimate built from an assumed manual baseline, not a measured productivity gain.
Before the first report, confirm with the sponsor which decisions the number will inform, such as renewal, staffing or scope, because that choice sets how conservative the headline should be.
Platform team leads run the templates and need to know which ones are used, which ones stall, and where the assumed baseline is out of date. Their report is per template and per period, with enough detail to decide what to fix, retire or promote. They also need to see failed and abandoned runs, because a template with high volume and high failure rates saves less time than its raw count suggests.
- Runs per template: counts from the task list for each template, split by successful, failed and cancelled runs.
- Estimate per template: the
backstage.io/time-savedvalue for each template, shown with the date it was last reviewed. - Trend lines: run counts and failure rates over several periods, so a change after a template edit is visible.
- Adoption gaps: teams that still create repositories by hand, where a template exists but is not used.
- Caveats to state: run counts show activity, not completed value, and a template that is rerun after failure can inflate counts.
Ask the platform team which template changes and scaffolder releases fall inside each reporting window, so trend changes can be tied to a known cause.
Finance and procurement reviewers do not need more dashboards. They need the method behind the number so they can test its assumptions, rerun the arithmetic and decide how much weight it deserves. Their report documents each input, its source, who approved it and how it changes over time. A reviewer who can trace every figure back to a run count and an assumed baseline will trust the result more than one who receives a polished total.
- Method statement: how the estimate is calculated, including how runs are counted and which runs are excluded.
- Assumption register: each baseline value, the person who set it, the date, and the evidence behind it.
- Data sources: where the run counts come from, and how they were exported and checked.
- Sensitivity view: how the total moves if the baseline values are lower or higher, so the range is visible.
- Caveats to state: the estimate excludes review queues, rework and quality effects unless those were measured, and it is not an audited figure.
Ask finance which cost lines they want in the comparison, and whether they expect the estimate to be reviewed by internal audit, since that determines how the method and register should be documented.
Security and data considerations for run metrics
Run metrics are built from task logs, parameters and catalog annotations, so the data you collect to prove template value can also expose secrets and personal details. The cards below cover the main risks and the control for each, so the measurement program does not create a new data store that is weaker than the portal it measures.
Sensitive values in task logs
Scaffolder task logs record each step, its input and its output, and parameters entered in a template form can include tokens, hostnames, customer identifiers or cloud account numbers. Before you count or export runs, check which parameters are echoed into logs and which action outputs print credentials. Mark sensitive parameters as secret in the template where your version supports it, and test a sample run to see what the log actually shows. Redact at the action level, not in the dashboard.
Personal data in metric labels
Metrics systems such as Prometheus keep label values as searchable time series, so a label built from a requester's name or email address stores personal data in a system that often has longer retention and wider access than the portal. Limit labels to low-cardinality values such as template name, owning team and outcome. Put per-user detail in the task record, where your access controls already apply, and never use secret values or raw parameter values as labels.
Dashboard access matching template access
A dashboard that shows which teams run which templates can reveal the existence of sensitive templates, such as those that create cloud accounts or repositories, to people who are not allowed to run them. Apply the same permission model to dashboards and to the template list, so a viewer sees run counts only for templates they could already open. Confirm this with a test user from a team that has no access to the restricted templates before you publish a report.
Retention matched to data policy
Run records are kept only as long as the scaffolder database and your log pipeline retain them, and those defaults are set by your deployment rather than by the template. Decide how long run records and exported metrics may exist under your data protection and records policies, then configure each store to that period. Ask legal or privacy staff whether the run history counts as personal data for your staff, and document the deletion schedule alongside the backup schedule.
Access to the metrics backend
Scaffolder metrics and logs are usually sent to a monitoring stack with its own accounts, so an operator who can query that stack may see parameters or outcomes that the portal would hide. Restrict query access to the platform team and named auditors, use separate service credentials for the exporter, and keep those credentials out of template parameters. Review who holds these roles each quarter as part of the platform's access review.
Annotations that hold no secrets
The backstage.io/time-saved annotation is stored in the catalog entity and is readable by anyone who can view the component, so it must hold only a planning estimate such as a number of minutes. Do not place internal cost rates, customer names, incident references or credentials in the annotation or in custom fields near it. Validate annotation values during template review, and state in the template README that the value is an estimate that anyone with catalog access can read.
Questions teams ask before measuring template value
Who sets the baseline for time saved by a template?
The platform team and each template owner usually set it together, with input from the engineers who used to do the work by hand. Someone has to record how long the manual task took, which steps it included and how often it happened, because the template's savings are only as credible as that starting figure. Ask whether the baseline comes from ticket history, time sampling or structured interviews, and who signs off on it. Teams often skip this step and later argue about the numbers. A partner should propose a baseline method for each template in the written scope, and your side should name an owner who can confirm the figures.
How many templates do you need before measuring is worth it?
Measurement effort grows with the number of templates, so start with the few that run most often. A single template used by one team can be tracked with the run history and a simple spreadsheet. A larger portfolio needs a consistent annotation convention, a shared baseline format and a report that rolls results up by template, team and period. Ask a partner to quote per template and to separate the first template, which sets the pattern, from the rest. Each added template also needs its baseline reviewed when the manual process it replaces changes.
How much instrumentation does template time tracking require?
The minimum is a time-saved annotation on each Template entity and access to the scaffolder run history that Backstage keeps for template executions. More instrumentation means exporting scaffolder metrics to your monitoring stack, joining run data with ticket or delivery data, and keeping annotations accurate as templates change. Confirm with your own Backstage version and deployment which metrics are exposed and where they are collected, because this varies with how the portal is run. Plan for each template owner to review their annotation before the first report is produced.
How long does it take to produce a first credible report?
No researched duration is available for this, so treat any figure you hear as something to test, not a commitment. The first report depends on how many templates are in scope, whether baselines already exist and how much run history the portal has accumulated. A report built on actual template runs is more credible than one built on projections. Ask a partner to show a schedule covering baselines, annotation, first data collection and the review meeting, and to mark which steps depend on your staff's availability.
What does the work need from platform and template owners?
Three groups carry most of the work:
- Platform team: access to the Backstage deployment and scaffolder task list, plus permission to add annotations through template changes.
- Template owners: agreement on the baseline steps, review of their annotations and sign-off on each figure reported.
- Finance or program sponsor: a definition of what counts as value and a named recipient for the report.
Ask for a responsibility matrix in the written scope so that each group knows which tasks and approvals are theirs before work starts.
Can our own team measure template value without a partner?
Yes, if you have engineers who can read Backstage templates and the run history, and someone who can own the baseline method. Most of the work is data collection, annotation edits and reporting, so a team already running the portal can produce a first pass. A partner adds the most value when baselines need interviews across several teams, when the report must hold up to finance or audit scrutiny, or when metrics need export and dashboards built. List which of these your team can staff, then ask for a quote covering only the gaps.
What recurs after the first report is published?
The one-time work is the baseline study, the initial annotation pass and the first report or dashboard. Recurring work covers keeping annotations accurate when templates change, reviewing baselines when the manual process changes and checking metrics after portal upgrades. Backstage ships a main release line monthly and a next release line weekly, so metric names and template behavior should be rechecked on upgrade. Ask whether the quote includes a retainer for this upkeep, since DeployOpen offers ongoing support as a separate arrangement.
How DeployOpen scopes and delivers template measurement
DeployOpen scopes time-saved measurement as a fixed-scope engagement with the deliverables written down before any work starts. The written scope should name each template in the measurement set, the baseline you will compare against, the annotation values agreed for each template, the metrics to be collected, the reporting format and the audience for each report. Ask for the scope to state who approves the baseline estimates, because those estimates are the part of the number a finance or platform sponsor is most likely to challenge. The scope should also list what is excluded, such as templates you have not yet published or Git platforms outside the current engagement.
Delivery happens inside your infrastructure. DeployOpen works with your team to add the backstage.io/time-saved annotation to each template, confirm that the value is read correctly in the catalog, and set up collection of run counts from the scaffolder task list and the scaffolder metrics that your Backstage deployment exposes. Confirm with your platform team where those metrics are sent, who can read them, and how long they are retained, since reporting depends on that data staying available. The reporting layer, whether a dashboard or a scheduled export, runs on systems you control, and access follows the same roles you use for the portal.
Measurement also needs upkeep after the first report. Template owners change, baselines go stale, and new templates get published without annotations. Ongoing reporting upkeep, including refreshing baselines, checking that annotations stay accurate and producing periodic reports, can be covered by a retainer rather than re-scoped each quarter. DeployOpen does not publish prices. A quote follows from the written scope, so bring your template count, the number of Git platforms involved and the reporting cadence your sponsors expect when you ask for one.
