Backstage Tech Insights scorecards setup

How Backstage Tech Insights scores services against your standards

Most organizations that adopt Backstage end up with a catalog full of services that have no owner, thin documentation or undeclared dependencies, and leadership asks for a number that shows how far the estate has to go. Tech Insights is a Backstage plugin, so the scoring itself needs no separate external connector. Fact retrievers read data from catalog entities and from the sources they are configured to query, checks evaluate those facts against your standard, and scorecards display the results on entity pages where teams already work. The design work is deciding which facts matter, who owns each standard and how a failing check turns into a catalog fix.

Before you plan a rollout, confirm the exact package names, configuration keys and retriever APIs against the current backstage/backstage repository and the Tech Insights documentation for the release line you run. No researched facts for this plugin were supplied for this page, so treat any names you see in older blog posts or community examples as unverified until they match your version. Check whether the plugin's retrievers can read the fields your standard depends on, such as an owner reference, a docs annotation or a relation to another component, and whether any check needs data from outside the catalog, which would require a retriever you build or a source you connect.

Start with a short list of standards that each map to a catalog field, for example an owner set on every component, a TechDocs site present and a declared dependency on each API consumed. Agree on who can change a check, because a check that moves with each team's preference stops being a standard. Run the checks on a small group of entities first, review the false positives with service owners, and only then publish scores to leadership. Expect the first run to surface gaps in catalog ingestion as well as in services themselves, since a missing entity cannot pass a check.

What a working Tech Insights setup needs before you start

A Tech Insights scorecard is only as good as the foundation under it. The cards below cover the platform, database, standards, catalog data and source access that have to be in place before any fact retriever or check runs.

A current Backstage release line

Pick the main release line or the next line deliberately, then pin every @backstage package to the same version. The main line ships monthly and the next line ships weekly, so a scorecard built on one line will need a planned bump as the plugin APIs move. Confirm which release line your team will follow and who owns the monthly bump. Check that your Node.js version matches the pair the versioning policy supports for your Backstage release before you install the Tech Insights backend module.

Supported Node.js runtime

Backstage supports exactly two adjacent even-numbered Node.js releases, and the supported pair changes between Backstage releases. Scorecard checks run inside the backend, so a runtime outside the supported pair will fail in ways that look like plugin bugs. Confirm the Node.js versions listed for your pinned Backstage release, and make the same versions available to developers, CI and the production image.

PostgreSQL for production storage

Tech Insights stores its fact and check results in the backend database, and production deployments should run on PostgreSQL rather than SQLite. Each backend plugin receives its own logical database through Knex, so the scorecard data is separated from the catalog and search data. Confirm that a PostgreSQL instance with backups, a credential for the Backstage backend and room for historical fact rows is available in your environment before the first retriever is enabled.

An agreed list of standards and owners

Each check needs a named standard, such as a service must declare an owner or must link to documentation, and a named group that decides when the standard changes. Without that agreement, a failing check becomes a debate instead of a backlog item. Write each standard as a sentence a team lead can verify, record who approves changes, and decide which entity kinds the standard applies to. Confirm this list with engineering leadership before you write any check code.

Catalog entities with ownership and system data

Scorecards measure what the catalog already says about a component, so entities need an owner reference and a system reference before checks can pass or fail meaningfully. Many checks depend on the same fields that drive catalog completeness. Run a query against your current catalog to count entities missing an owner, a system or a lifecycle value, and confirm which team will fix those gaps before scoring begins.

Read-only credentials for external sources

Fact retrievers that query Git hosts, CI systems, or cloud and Kubernetes APIs need credentials that can read metadata and nothing else. Create a dedicated account or token for each source, scope it to the repositories or projects the portal covers, and store it in your secrets manager rather than in plain configuration. Confirm with each system owner that the token cannot write, merge, deploy or change settings, and set an expiry and rotation owner.

Configuring Tech Insights in the order that avoids empty scorecards

This sequence sets up standards, data collection and checks in the order Tech Insights needs them, so each scorecard shows meaningful results from the first load. The result is a tested scorecard on entity pages that a team can explain and maintain.

Agree standards and their owners
Before any code changes, the platform team and the service owners agree on a short list of standards, such as a named owner, a published docs location and a declared dependency list. Each standard gets one accountable owner who approves its pass criteria and changes to them. Write the list in a shared document and keep it in version control next to the catalog rules. This step is done when every standard has a name, a definition in plain language and an owner who has accepted it, so no check is written without someone answerable for it.
Add the Tech Insights plugins to the app
A platform engineer adds the Tech Insights backend plugin to the Backstage backend and the frontend plugin to the Backstage frontend in the existing app, then runs the install and build commands that match the Backstage version in use. Tech Insights stores its data in the backend database, so confirm that the backend database connection works before starting the backend. This step is done when the app builds, starts cleanly and the Tech Insights API responds to a request from an authenticated user, showing an empty list of checks rather than an error.
Register fact retrievers for ownership, docs and dependencies
Register one fact retriever for each standard, starting with ownership, documentation and dependency data. Each retriever reads from the catalog and related sources and writes a defined set of facts for each entity, and each fact needs a stable name and a schema. Retrievers that have never run leave their facts empty, which is why this step comes before checks are defined. Confirm the exact retriever API and fact schema against the version of Tech Insights you install. This step is done when a retriever run completes for one test entity and the facts appear in the stored data.
Run retrievers once and confirm stored facts
Trigger a single retrieval run for a test entity, then query the stored facts for that entity by name. Compare each returned value with what the catalog shows for the same entity, such as the owner reference, the presence of a docs directory and the listed dependencies. Mismatches usually mean the retriever reads the wrong field or the entity lacks the annotation it depends on. This step is done when every fact for the test entity matches the source data, which proves the scorecard will have something to score.
Define checks with explicit pass criteria
Define one check per standard, written as a rule that reads stored facts and returns pass or fail with a message a service owner can act on. A good ownership check states which fact must exist and what counts as valid, not just that a field is set. Store the check definitions in version control so changes go through review. Avoid checks that depend on facts no retriever writes, because they will show as failing for every service. This step is done when each check returns a result for the test entity and its failure message names the missing item.
Set the retrieval schedule
Configure how often each fact retriever runs, balancing freshness against load on the catalog and source systems. Ownership and dependency data changes less often than build or test data, so these retrievers can usually run less frequently. Confirm the schedule options and any timeout settings for the version you run, and check the backend logs after the first scheduled run. This step is done when a scheduled run completes without errors and the stored facts update after a change made to the test entity.
Add the scorecard component to entity pages
Add the Tech Insights scorecard component to the entity page layout, and place it on the overview tab for component entities so owners see results when they open their service. Limit the component to the entity kinds your checks cover, so unrelated pages do not show empty panels. Confirm the component renders with a logged-in user who has read access to the entity. This step is done when the scorecard appears on the page of the test entity and lists each check with its current result.
Verify with a known-good and a known-bad service
Pick two test entities: one that meets every standard, and one deliberately built to fail, such as an entity with no owner and no docs. Confirm the scorecard shows all passes for the first and the expected failures with clear messages for the second. Then fix the failing entity and confirm its result changes after the next retrieval run. Ask the owning team of a real service to review the messages before rollout. This step is done when both test cases behave as described, which shows the checks are correct before real teams see them.

Which facts to retrieve first, by standard type

Ownership facts answer a simple question: does every component have a named team that answers for it? This fact is usually read from catalog fields, specifically the spec.owner value on each entity, which should point to a Group or User entity. The check should require that the owner reference resolves to an existing entity, not merely that the field is filled in, because a free-text or misspelled owner looks complete on a dashboard while nobody is accountable. Confirm with your release notes which ownership retrievers ship out of the box and which you must write yourself.

Owner present: the entity has a non-empty spec.owner field.
Owner resolves: the referenced Group or User exists in the catalog, which catches renamed or deleted teams.
Owner active: an illustrative policy might treat a group with no members as a failed check, since members often leave before the group is cleaned up.
Common breakage: teams are renamed in the identity directory but not in catalog descriptors, so ownership quietly drifts out of date.
Common breakage: components created by templates default to the template author, not the owning team.

Before you rely on ownership scores, check which fields your retrievers read and whether the owner reference is validated against the catalog.

Documentation facts measure whether a component has the material its consumers need, and whether that material is findable from the catalog entry. In a Backstage deployment, the most common signal is the presence of the backstage.io/techdocs-ref annotation and a successful TechDocs build, so the fact depends on both catalog metadata and the output of a build pipeline. A check that only tests the annotation can pass while the rendered site is empty or stale, so the check should confirm content exists and was built recently. Confirm which TechDocs-related retrievers your release provides before you design the rule.

Annotation present: the entity declares a TechDocs reference in its annotations.
Build succeeded: the documentation site was generated from the current source, not left from an old build.
Minimum content: an illustrative rule might require an index page and a short description, rather than a single placeholder file.
Common breakage: the documentation lives in a repository that was moved or archived, so the reference points to nothing.
Common breakage: teams write docs once and never update them, which a presence-only check cannot detect.

Decide early whether freshness is measured by build time or by source commit date, because the two give different results.

Dependency facts describe how a component relates to others: what it calls, what calls it, and whether those links are declared. Most of this data comes from the catalog's relationship model, such as dependsOn and dependencyOf references, which teams write by hand or which an external source generates. Because it is partly declared and partly discovered, the check must distinguish between a component with no declared dependencies and one whose dependencies have never been recorded. Confirm with your release notes which dependency retrievers ship with your release and which require custom code that reads from a service registry, CMDB or deployment tool.

Relations declared: the entity lists at least one dependsOn reference, or an explicit statement that it has none.
References resolve: every referenced component exists in the catalog, so stale links are caught.
External source agreement: an illustrative check could compare declared dependencies with runtime traffic or deployment data from a source you control.
Common breakage: dependencies are declared on the consumer side only, leaving the provider with an incomplete picture.
Common breakage: renamed components break every reference to them, and the errors are easy to miss in large catalogs.

Whatever source you choose, record which system is authoritative for each relation so scores can be explained to auditors.

How catalog fields and standards map to facts and checks

This table traces each catalog attribute a scorecard might test to the fact a retriever produces, the check that evaluates it, and where the underlying data lives. Read the last two columns first: they show which gaps are closed by editing the descriptor and which need an external tool or a team to fix. Field names and retriever choices should be confirmed against the Tech Insights version your team installs.

Catalog attribute	Fact produced and check applied	Source of truth and maintainer
Owner (spec.owner)	Fact records the owner reference; check passes when the field is set to a known group or user	catalog-info.yaml in the service repo; owning team maintains it
System (spec.system)	Fact records the system the component belongs to; check confirms the referenced System entity exists	Descriptor for the component and the System entity; platform team or owning team maintains
Lifecycle (spec.lifecycle)	Fact records production, experimental or deprecated; check flags values outside the allowed set for production services	Descriptor in the repo; owning team maintains, reviewed at release
Documentation presence	Fact records whether TechDocs source exists in the repo; check passes when an mkdocs.yml or docs folder is present	Docs in the service repo; owning team maintains the content
Declared dependencies (dependsOn)	Fact lists declared relationships; check passes when at least one dependency or explicit none is recorded	Descriptor relations; owning team maintains. Runtime traffic is not captured by this fact
Build and CI annotation	Fact records whether the pipeline annotation is present; check passes when it references a live pipeline	Annotation in the descriptor; the CI tool owns the pipeline data
Security findings	Fact is pulled from the scanner's API; check passes below a threshold the team sets	Scanner (for example, a vulnerability tool); security team maintains the thresholds and data
Team process standards (for example, on-call rota)	Fact is pulled from the incident or on-call tool; check confirms a rota is linked	On-call tool; the incident team maintains the schedule, not the catalog

Securing the data behind scorecards

A scorecard is only as trustworthy as the data it reads and the people who can see it. The cards below cover the credentials, configuration, access boundaries, code review and field exposure that decide whether scorecard results stay inside the right audience.

Read-only credentials per source

Each fact retriever calls a source system such as a Git host, CI server or ticketing tool, so each source needs its own credential. Issue read-only tokens scoped to the repositories and fields the checks need, and avoid reusing an administrator token across retrievers. Ask each source owner which permissions the token actually holds and document the rotation owner for every credential.

Tokens in backend configuration

Retriever credentials belong in the backend configuration, read from environment variables or a secrets store at runtime, and never in frontend config or in the scorecard UI. Check that the frontend bundle contains no token values after each build. Treat the backend config file as sensitive even when the secrets are injected, because it shows which sources the portal connects to.

Team-scoped scorecard access

A scorecard for one team can reveal gaps in another team's services, so decide whether results should be limited to the owning team and its managers. Confirm whether your Backstage authorization setup can filter Tech Insights results by entity ownership. If it cannot, decide whether the broader view is acceptable for your organization and record that decision.

No guest access in production

The authentication documentation advises against offering guest access in production. A guest session could read scorecard results without an identity to audit, so turn guest sign-in off in production configuration and confirm it in a test sign-in before go-live. Every viewer should resolve to a real identity from your identity provider.

Review custom retriever code

A custom fact retriever runs inside the Backstage backend with the credentials it is given, so it deserves the same review as any other plugin code. Check what hosts it calls, how it handles errors and whether it logs response bodies that may contain sensitive data. Require a second reviewer and pin dependency versions before the retriever reaches production.

Fields exposed to viewers

Scorecard results can include check names, failing values and source details that viewers may not need to see. List every field each check writes and confirm which ones appear in the UI and the API for each role. Remove raw values such as internal hostnames, vulnerability details or access-related data from results before publishing them to a broad audience.

How scorecards drive catalog completeness over time

The core feedback loop is simple: a failing check tells the owning team exactly which descriptor field or document is missing, the team fixes it in the catalog entry or the repository, and the next fact retrieval run updates the result. The check itself is the instruction, so a well-written check names the missing field, such as a missing spec.owner value or an absent TechDocs mkdocs.yml, rather than reporting a vague failure. Confirm with your Tech Insights setup how often retrievers run in your environment, because that interval sets how long a fix takes to appear on the scorecard. Owning teams only act on what they can see, so make the failing entity easy to find from its own page.

Keep standards realistic by phasing checks in. Start with a small set that reflects what the catalog already holds, such as ownership and a documented description, and add stricter checks only after most entities pass the earlier ones. Agree the phasing schedule with platform owners and team leads before any check becomes visible, and record which checks are informational and which are expected to pass. Measure the pass rate per check, the number of entities with no owner, and how long entities stay in a failing state. These measures show whether the standard is achievable or whether a check needs redesign, and they should be reviewed on a fixed cadence rather than after complaints arrive.

Avoid turning checks into gatekeeping by agreeing in writing what a failing score does and does not trigger. A scorecard should inform decisions, not block deployments or template use, unless the owning organization has deliberately chosen that rule for a specific control. Scorecard data can also inform software templates: if a check shows that new services repeatedly lack a documented owner or a catalog-info.yaml field, the template skeleton can include that field by default, so new services start complete. Track whether newly created entities pass their checks on the first retrieval, since that measure shows whether templates are doing their part.

Why scorecards show empty, stale or wrong results

This table lists the failures teams hit most often with Tech Insights scorecards, the cause behind each one and the fix to apply. Read it when a scorecard shows blanks, old values or failures across every entity. Most problems trace to registration, annotations, copied thresholds or credentials rather than to the plugin itself.

Symptom	Likely cause	Fix
Facts are empty for every entity	Fact retrievers are not registered in the backend, so no data is collected	Register each retriever in the Tech Insights backend setup, restart the backend, then confirm the fact appears for a sample entity
Facts are missing for some entities only	The catalog entity lacks the annotation or field the retriever reads, such as an owner or docs setting	Check the entity descriptor for the required fields, add them through the catalog, and re-run the retriever for that entity
Results never change after an edit	The retriever runs on a schedule and the last run has not happened yet, or the schedule was changed without a restart	Review the schedule configuration, confirm the scheduler is running in the backend logs, and trigger a run to compare
Stale values after a schedule change	Old facts remain in storage until the next successful collection cycle	Run collection once after the change and verify the timestamp on a few entities before trusting the trend
Checks fail for nearly every entity	Thresholds were copied from another organization or example without review against your own catalog	Review each check threshold with the owners of the standard, pilot on a small group of teams, then roll out in stages
Errors for external sources such as source control or CI	Tokens are expired, scoped too narrowly, or stored in a place the backend cannot read	Confirm the credential's scope with the source system, store it in backend configuration rather than the frontend, and rotate it on a planned schedule
Scorecard plugin breaks after a versions:bump	The Tech Insights or community plugin lags the core @backstage release line	Use the Backstage Upgrade Helper to compare template changes, check the plugin's release against the core line, and pin or update before deploying

Questions buyers ask before setting up Tech Insights

How much effort does a first Tech Insights scorecard take?

Effort depends mainly on how many standards you score, whether each check can read data the catalog already holds, and whether you need custom fact retrievers that pull from outside systems. A small first set of checks on owners, docs and dependencies is a contained build when the catalog is clean. Each added standard means one more check to define, test and explain to service owners. Custom retrievers that call CI systems, cloud accounts or ticketing tools add authentication, rate limits and error handling. Ask for the standards, retrievers and external sources listed in writing, with effort shown for each.

How long until a scorecard gives useful results?

Results appear once the fact retrievers have run against the catalog and the checks have evaluated, but useful results depend on catalog quality. Your team must supply the standards and the pass criteria for each, an owner who signs off on them, access to each source system, and a service account for every external read. Expect a first run to expose many missing owners and missing docs. That is the purpose of the scorecard, but teams should hear about it before launch so early numbers do not cause alarm. The duration belongs in the fixed-scope agreement, which is built from these inputs.

Can an internal platform team set up Tech Insights without a partner?

Yes, if the team already runs Backstage in production and has engineers comfortable with TypeScript and the plugin model. Backstage is licensed under Apache-2.0, and the project itself offers no paid edition, so the software cost is zero and the real cost is staff time. The hardest parts are designing fact retrievers, agreeing pass criteria with service owners and keeping checks working across Backstage upgrades. Teams new to Backstage should first stabilize the catalog, since scorecards depend on entity data. A partner adds value mainly when the team lacks capacity or wants a fixed delivery date.

What does keeping Tech Insights running cost after launch?

The recurring cost is people time. Checks need updating when standards change, retrievers need updating when a source API changes, and the Backstage instance itself needs upgrades. Backstage ships a main release line monthly and a next release line weekly, and the versioning policy says high and critical vulnerabilities will be backported to releases from the last 6 months if feasible. Upgrades are typically made with yarn backstage-cli versions:bump, with the Upgrade Helper showing template changes. Ask any quote which upgrade work is included, who performs it and how often it is reviewed.

Do Tech Insights scorecards replace audit evidence?

No. A scorecard shows how entities measure against standards your organization chose, and it can help show that owners and documentation exist. It does not make a control effective on its own. Auditors decide whether evidence is sufficient, and they typically test the process, the scoring logic, the data sources and whether teams acted on failing results. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies make those determinations, so involve your assessor when you design the checks.

What should a fixed-scope Tech Insights quote include?

Ask for a written list of standards with pass criteria, each fact retriever and the source system it reads, the catalog entities in scope and the environments covered. The quote should also state what you receive at handover, the acceptance criteria for each deliverable, and who provides service accounts and access. Confirm how changes to a source API are handled, how checks are rolled out to teams and whether Backstage upgrade support is included. Separate one-time build work from any ongoing retainer. DeployOpen agrees scope, deliverables and timelines in writing before work starts, and a quote follows from that written scope.

How DeployOpen scopes and delivers a Tech Insights setup

DeployOpen plans, builds, documents and supports Backstage Tech Insights for scorecards and maturity checks inside your own infrastructure. We start from the standards your organization wants to measure, agree on the checks and data sources in writing, and hand over a scorecard setup your own team can maintain as the catalog grows.

How we scope

Agree which maturity standards matter, such as owners, documentation and dependency data, before any check is written
Inventory the catalog descriptors, annotations and data sources each check depends on for your entities
Record the written scope, deliverables and timeline in advance, since most engagements are fixed-scope

How we build

Configure the Tech Insights backend, fact retrievers and checks against your Backstage version and database
Test each check against sample catalog entities, including ones that should fail, before wider rollout
Document scorecard logic, retriever ownership and the catalog fixes teams need to make each check pass

How we hand over

Deliver the configuration, check definitions and runbooks in your own repositories under your control
Walk your platform and catalog owners through adding a new check and reading scorecard results
Record open gaps, such as checks that lack a reliable data source, so your team can decide next steps

How we support

Ongoing support, upgrades and continuous testing are available as a retainer alongside your Backstage upgrades
Review check results and fact retriever changes when Backstage releases or plugin versions move
Confirm with your team which owners approve changes to standards so scorecards stay aligned with policy

Backstage Tech Insights for scorecards and maturity checks