How Backstage Tech Insights replaces Compass scorecards
Teams that rely on Compass scorecards, such as production readiness, face a practical question as Atlassian moves its catalog and scorecards toward DX: can the same pass-or-fail logic run in a portal you host yourself? It can. The scorecard rules can be rebuilt in self-hosted Backstage with the Tech Insights community plugins, which let you define facts (data points collected from catalog entities) and checks (rules that evaluate those facts). The parts that need extra work are the dashboards and alerts, because the Tech Insights plugins do not give you a finished scorecard screen or notification routing out of the box, so you should plan to build or assemble them. The tech-insights-maturity plugin adds maturity rankings and categories to checks, which covers the leveled view many Compass users rely on, but you should confirm its current behavior against the Tech Insights workspace in the community plugins repository before you commit to a design.
On cost, Backstage is an Apache-2.0 framework with no paid edition from the project itself, so the license line on your quote should be zero. What you will pay for is the infrastructure to host the portal and its PostgreSQL database, plus engineering time to write the fact collectors, rules, dashboards and alerts, and then to keep them current through Backstage upgrades. Those costs recur, and they grow with the number of checks and the number of data sources you connect. Ask any implementation partner to price these as separate line items with the scope written down, so you can compare them with your current Compass or DX subscription on equal terms.
The right design depends on two things you must settle first. The first is your data sources: production readiness checks usually read from CI/CD, incident tooling, repository settings and observability systems, and each source needs an integration you own. The second is your ownership model: scorecards only work if every component has a reliable owner in the catalog, and that depends on how you map teams from your identity provider and how the owner field is maintained in each catalog-info.yaml file. Inventory both before writing a single rule, because the answers decide whether Tech Insights fits your team or whether a different scorecard approach is the better path.
Compass scorecard capabilities mapped to Backstage
This table maps each Compass scorecard capability to its Backstage counterpart, from catalog data to the maturity ranking and the portal view. The coverage note on each row shows whether the Tech Insights plugins handle it directly, need custom fact retrievers, or depend on work outside the plugins. Read it as a build plan, not as proof of feature parity with Compass.
| Capability | Backstage counterpart | Coverage note |
|---|---|---|
| Component ownership and metadata | catalog-info.yaml descriptors with Component, System, Group and API kinds, replacing compass.yml and UI-created components | Covered by the Software Catalog. Confirm that every component you track has an owner and lifecycle value in its descriptor before you write any check against them. |
| Data points that feed scorecards | Tech Insights facts that collect data about each catalog entity | Built-in facts cover only generic data. Expect custom fact retrievers for signals such as on-call setup, alert configuration or SLO presence, which depend on where your data lives. |
| Rule-based pass or fail criteria | Tech Insights checks that evaluate facts against rules for each entity | Checks are configured in code. Test each rule against a sample of real entities, since a rule that reads a missing fact may silently fail to match what your teams expect. |
| Maturity categories and rankings | tech-insights-maturity plugin, which adds maturity rankings and categories to checks | Confirm the level names and thresholds in your own environment, because they must match the production readiness levels your organization already uses in Compass. |
| Scorecard view for teams | Scorecard presentation in the portal UI, backed by the checks and maturity results | Verify the view meets your needs for filtering by team, domain or tier. Expect UI work if leaders need trend lines or exports. |
| Dashboards across all services | Portal views and the data behind them, not a cross-service analytics dashboard by default | Needs extra work outside the plugins, such as exporting results to Grafana or a data warehouse and building the panels your leaders read. |
| Alerts when a service drops below target | No built-in alerting from checks in the plugins described here | Needs extra work, typically a scheduled job that reads check results and notifies owners through your chat or incident tools. Confirm the trigger and routing rules yourself. |
| Access and team permissions on scorecards | Backstage authentication with sign-in resolvers that map each provider to one identity, plus the permission framework | Confirm in your environment how group membership from your identity provider controls who sees which scores. Guest access should be off in production. |
Which open-source options rebuild Compass scorecards
Compass scorecards are rule sets that score each component against a standard, such as production readiness. The options below range from a Backstage plugin path to a licensed commercial self-managed edition, and each one trades setup effort, vendor dependence and out-of-the-box coverage differently. Pick the path that matches how much rule authoring your platform team can own.
Backstage Tech Insights with maturity rankings
The Tech Insights community plugins let you define facts (data points collected from catalog entities) and checks (rules evaluated against those facts). The tech-insights-maturity plugin adds rankings and categories to checks, which is the closest match to Compass scorecard levels. This fits teams that want rules in version-controlled code. Confirm which fact retrievers cover your repositories, CI systems and on-call tooling, and check the plugin's compatibility with your Backstage release before you commit.
Backstage with in-house fact retrievers
When the data you need for readiness lives in internal systems, such as a deployment ledger, incident tool or service register, your engineers write custom fact retrievers that pull it into Tech Insights. This gives full control over what counts as production ready. The trade-off is ongoing ownership: your team maintains the retrievers, handles API changes and tests them through Backstage upgrades. Confirm who on the platform team will own each retriever and how failures are surfaced.
Self-hosted Cortex or OpsLevel edition
Cortex and OpsLevel both ship scorecard and maturity rubric models out of the box, which suits teams that want a defined model quickly and accept a vendor product. Their self-managed editions still need a vendor license key or license file, so self-hosting does not remove the subscription. Cortex requires a JWT license key, an access token and PostgreSQL. OpsLevel installs through Replicated and needs a license file. Confirm the license terms, data flows and what the vendor can see before choosing.
Port with self-hosted Ocean integrations
Port is a commercial developer portal, but its data integrations, most built on the Ocean framework, can run in your own infrastructure with Helm or Docker rather than on Port's hosted resync. This path keeps data collection in your environment while the scorecard logic stays in Port's product. It is a non-Backstage route, so you still depend on a commercial portal. Confirm whether Port's scorecard features cover your rules and what licensing applies to the self-hosted integrations.
Catalog plus scheduled scripts
For a small estate, a set of catalog-info.yaml files and scheduled scripts can evaluate readiness, such as checking for an owner, a runbook link, a health endpoint or a CI pipeline, and write results to a shared report. This avoids a plugin stack entirely. The risk is that scripts drift from the catalog and become hard to audit as the number of services grows. Confirm how results are stored, who reviews failing services and how the script set is tested before anyone relies on it for release decisions.
How facts and checks score catalog entities in Tech Insights
Tech Insights gives you a scorecard model you can run inside Backstage without a vendor console. Teams define facts, which are data points collected about a catalog entity, and checks, which are rules that evaluate those facts and return pass or fail. The community tech-insights-maturity plugin adds maturity rankings and categories to checks, so a single failing rule can be grouped and ranked instead of just counted. For a Compass-style production readiness scorecard, this is the mapping you need to rebuild: each readiness criterion becomes a check, the data it needs (an on-call owner set in catalog-info.yaml, a runbook link, a health endpoint, a passing build) becomes a fact, and the maturity category expresses whether the criterion is a baseline requirement or an improvement goal.
Illustrative example: a check named production readiness might require that a Component has an owner, a documented runbook and a recent successful pipeline. The owner comes from the descriptor, the runbook from annotations, and the pipeline status from a fact source your CI system exposes. Before you write any checks, confirm which fact sources your Backstage backend can reach from your own network, such as your Git host, CI server, Kubernetes clusters or internal APIs, and whether credentials for each source are stored securely. A check that depends on an unreachable source will silently stop producing useful results. Tech Insights stores fact results in the database. Backstage uses PostgreSQL in production and SQLite for development, and it gives each backend plugin its own logical database through Knex. Plan for a dedicated PostgreSQL instance or database with backups, sizing for the history you want to keep, and an owner for upgrades. Confirm the plugin's current storage schema and retention behavior in the plugin documentation before you commit to a retention policy.
What dashboards and alerts need beyond the scorecard
The platform team owns the fact retrievers, the check definitions and the upgrade path. The Tech Insights community plugins let you define facts (data points) and checks (rules) against catalog entities, and the tech-insights-maturity plugin adds maturity rankings and categories to checks. Everything that turns those results into something people act on sits outside the plugins, and that is where most of the platform team's extra work lands.
- Fact retrievers: Each fact needs a source. Decide which systems feed it, such as CI, the Kubernetes API or a repository, and who maintains the retriever when those systems change.
- Check design: Write checks that map to your production readiness policy, and document which rule belongs to which maturity category so ranking changes are reviewed, not silent.
- Dashboard layout: The plugins provide the data model, but the views, filters and team breakdowns are your own frontend or Grafana work.
- Upgrades: Backstage releases monthly with a weekly next line, so plan a regular upgrade cycle that re-tests your custom fact retrievers.
Confirm which telemetry store and alerting tool your organization already runs and who is on call for the portal itself.
Service owners need to see their own score, understand why a check failed and know the fastest fix. The plugins show results against catalog entities, but the explanation and remediation guidance for each failing check is content you have to write. Without it, a red check becomes a ticket with no owner and no next step.
- Reading scores: Agree with teams what a rank means in practice, so a maturity category is not read as a judgment on the team.
- Fixing failing checks: For each check, link to the runbook, the template or the config key that resolves it. A short note per check is usually enough.
- Ownership mapping: A score is only useful when the entity has an accurate owner. Check that each catalog-info.yaml names a group that exists in your identity provider.
- Exceptions: Decide how temporary waivers are requested and expire, since the plugins evaluate rules but do not decide who may skip one.
- Notifications: Routing a failing check to the owning team's chat channel or ticket queue is custom integration work.
Confirm with a few pilot teams which checks they can fix within their own sprint and which need platform help.
Leadership wants a rollup: how many services meet the bar, where the gaps cluster and whether things are getting better or worse. The Tech Insights plugins evaluate entities at a point in time. Trend views, rollups across teams and regression alerts are separate work, and they depend on how long you keep historical results and where they are stored.
- Rollup dashboards: Aggregate by domain, system or group in your dashboarding tool, using the same ownership data the catalog holds.
- Historical trend views: Confirm how check results are persisted and for how long, since a trend needs stored history, not only the latest run.
- Regression alerting: Define what counts as a regression, such as a previously passing check now failing, and route it to a named owner rather than a shared inbox.
- Reporting cadence: Agree which numbers go to a monthly review and who signs off on the definitions behind them.
- Data quality: A rollup is only as good as the entity inventory, so track services missing from the catalog separately.
Confirm which dashboard and alerting tools leadership already uses and whether they can read from your portal's data store.
Running scorecards alongside the rest of your Atlassian and CI stack
A Backstage scorecard is only as good as the facts feeding it, and those facts come from identity, source control, CI and the Atlassian tools your catalog still references. The cards below cover what to connect, what to confirm before Compass access ends, and where the portal runs while both systems overlap.
Jira and Bitbucket references
Compass components often point to Jira projects, Bitbucket repositories and team channels. Before you rebuild scorecards, list every Atlassian reference your catalog depends on and confirm with Atlassian which of that data you can still export. Anything that does not export cleanly should be recorded as a manual catalog-info.yaml field, so a scorecard check does not silently fail when the link is gone.
Identity through built-in providers
Backstage ships 19 built-in authentication providers, including GitHub, GitLab, Okta and generic OIDC, so sign-in can follow the identity provider you already run. The authentication documentation advises configuring sign-in resolvers so every provider resolves to the same identity, and not offering guest access in production. Check that a person's scorecard owner resolves to one user record, whichever provider they sign in with.
CI and source systems feeding facts
Tech Insights facts are only as current as the systems that produce them. Map which CI pipelines, repository settings and deployment records supply each fact, such as build status, test coverage or on-call metadata, and confirm each source can be read by the portal with a token scoped to read-only access. A check that reads stale CI data will report a passing scorecard that no longer reflects production.
Parallel running during phase-out
Running Compass and Backstage side by side lets teams compare scorecard results before they switch decisions to the new portal. Agree which system is authoritative for each scorecard, record the date you stop editing in Compass, and keep both sets of results for a period long enough to explain differences. Ask Atlassian for the exact date Compass access ends, since that date sets the end of the overlap.
Where the portal is hosted
Backstage is an Apache-2.0 framework you deploy yourself, so the portal, its PostgreSQL database and its cache run in the infrastructure you choose. Decide whether the instance sits in your cloud account or data center, who can reach it, and how its database backups are handled. Hosting in your own network also means your team owns upgrades, following the release line and security backport policy.
Rebuilding Compass scorecards before Compass access ends
This sequence moves each Compass scorecard into Backstage as checks and facts with maturity categories, so you can run the old and new scorecards side by side and confirm they agree before you rely on the new one. It also sets up the ongoing upgrade work that keeps the portal current.
Inventory current Compass scorecards and rules
Export the definition of every scorecard in Compass, including each criterion, its threshold, the components it applies to, and the team that owns it. Your platform team or the Compass administrators can produce this list, and a spreadsheet with one row per rule is enough. The step is done when every production readiness rule has a named owner and a plain-language description of what it tests.
Split rules into checks and facts
Decide which rules are pass or fail checks, such as whether a service has an on-call owner or a runbook link, and which are data points that other checks or dashboards need, such as deployment frequency or last scan date. Facts are collected by fact retrievers, and checks are evaluated against those facts using the Tech Insights plugins. The output is a mapping table from each Compass rule to a check, a fact, or a combination of both.
Set up Backstage on a supported Node.js release
Install Backstage on a Node.js release that the versioning policy supports. As of Backstage 1.46.0, that means Node.js 22 or 24. Your platform team starts from the Backstage getting started guidance, pins the Backstage version it will run, and records which release line it follows. The step is done when the portal starts locally and the team can build it with a repeatable command.
Configure PostgreSQL and sign-in resolvers
Point Backstage at a PostgreSQL database, which the architecture overview describes as the production choice, and confirm each plugin receives its own logical database. Configure your identity provider so every sign-in method resolves to the same user identity, and turn off guest access before any production use. The step is done when a test user signs in through one provider, sees the same identity as through another, and guest access is refused.
Load catalog-info.yaml entities
Add a catalog-info.yaml descriptor to each repository that Compass covers, using kinds such as Component, API, System, Group and User. Register them through a Location entry or your chosen discovery method. Check that the component count and ownership match your inventory from the first step. The step is done when every in-scope component appears in the catalog with an owner, and the mismatches are listed and resolved.
Implement fact retrievers and define maturity categories
Write fact retrievers that read the data each fact needs from your source systems, such as the repository, CI system or incident tool, and store the results. Then define maturity categories and ranks in the tech-insights-maturity plugin so that each check result maps to a level your teams recognize. The step is done when each fact returns a value for a sample of components, and the maturity levels reproduce the agreed rankings for those components.
Run both scorecards in parallel
Keep Compass scorecards active while Backstage evaluates the same components, then compare the results component by component. Investigate every difference and classify it as a rule mapping error, a data gap or a real change in service health. Do not switch teams to the new scorecard until the differences are explained and the owners have signed off. The step is done when the two sets of results agree or each remaining gap has a documented decision.
Rebuild dashboards, alerts and ongoing upgrades
Recreate any Compass dashboards and alert routes that depended on scorecard results, using the Backstage portal views and your existing alerting system. Note that these parts often need extra work because the Tech Insights plugins provide checks and facts, not a full dashboard or alert product. Then schedule recurring upgrade work: Backstage's main release line ships monthly, the next release line ships weekly on Tuesdays, and high and critical vulnerabilities are backported to releases from the last 6 months where feasible. Confirm your upgrade cadence and staffing before Compass access ends.
What it costs to run scorecards yourself
The biggest cost in running Compass scorecards yourself is engineering time, not licensing. Backstage is Apache-2.0 licensed and the project itself charges no license fee, so the money goes to hosting, PostgreSQL, and the people who build and keep the scorecard checks working. Your Compass subscription is the baseline you already pay today, and it is the number every self-hosted estimate should be compared against. Ask any implementation partner to price these components separately: the initial build of fact retrievers and checks, the migration of existing scorecard definitions, and the recurring platform work after go-live.
Some costs are one-time and some recur. The one-time work is building the Tech Insights facts and checks, mapping your Compass scorecard levels onto the maturity categories from the tech-insights-maturity plugin, and building the dashboards and alerts that Compass gave you out of the box. The recurring work is hosting and the PostgreSQL database that Backstage uses in production, plus upgrades. Backstage's main release line ships monthly and its next line ships weekly, so someone must test each upgrade against your custom checks. High and critical vulnerabilities are backported to releases from the last 6 months if feasible, which means you must stay within that window to receive them. Confirm which Node.js versions the release you pick supports, since the versioning policy ties support to specific even-numbered releases.
What drives cost up or down is mostly the number of checks, how many data sources each check reads, and how often your standards change. Fewer, well-owned checks are cheaper to maintain than a long list nobody reviews. Ask for a quote that lists each component, who performs it, whether it is fixed-scope or retained, and what support runs after handover. Note that Cortex and OpsLevel self-managed editions still need a vendor license key or license file, so self-hosting either of them does not remove the subscription. Confirm that point in writing before comparing totals.
Questions buyers ask about replacing Compass scorecards
How much effort does it take to rebuild a production readiness scorecard in Backstage?
The effort depends mostly on how many rules your scorecard has and how many of them can be read from data you already hold. Each rule in a Tech Insights setup is a fact (a data point collected from a catalog entity) plus a check (a rule evaluated against that fact). Rules that read catalog fields, such as an owner, a lifecycle tag or a linked runbook, are usually the quickest to define. Rules that need data from CI, monitoring or incident tools need a fact retriever or custom collector, which is where most of the work sits. Ask for a rule-by-rule inventory of your current Compass scorecard before anyone estimates effort, because that list is the real input to any quote.
Can our own team rebuild the scorecards without outside help?
A team with Backstage experience can build the first version alone, provided it has someone who can write TypeScript plugins, manage a PostgreSQL database and maintain a Backstage instance. The harder question is ongoing ownership. Backstage releases a main line monthly and a next line weekly, so the people who maintain the portal must schedule upgrades and check plugin compatibility. Confirm internally whether you have that capacity. If you do not, a fixed-scope engagement can cover the initial build and hand over documentation, or a retainer can cover upgrades and rule maintenance after go-live.
How long does a Backstage scorecard migration take?
Duration depends on scope and on where your data sources live. A scorecard that reads only catalog metadata can move faster than one that pulls from several CI systems, incident tools and cloud accounts, because each source needs access, testing and a mapping to your rules. Team structure matters too: the number of teams whose services must be checked, and how many of them keep their ownership data in consistent form, changes the timeline. DeployOpen does not publish a fixed duration. Ask for a written timeline tied to the scope, and confirm your data sources and access approvals first.
What do the platform team and service owners need to contribute?
The platform team owns the Backstage instance, the Tech Insights configuration, the database and the access model. Service owners need to keep their catalog-info.yaml files accurate, because scorecard results are only as good as the entity data behind them. In practice, that means confirming ownership, lifecycle and documentation links for every component in scope. Plan for review time from both groups: someone must agree which rules are blocking and which are advisory, and service owners must be able to act on failing checks. Ask your sponsor to name these people before the work starts.
Do the Tech Insights plugins include dashboards and alerts?
The Tech Insights community plugins let you define facts and checks and evaluate catalog entities, and the tech-insights-maturity plugin adds maturity rankings and categories to checks. Dashboards and alerting are not a complete out-of-the-box replacement for every Compass view, so confirm this against the plugin version you intend to run. Typically you should expect to build or adapt frontend cards for scorecard views and connect results to your existing alerting, such as your monitoring or chat tools. Treat dashboard and alert work as a separate line item in the scope.
How do we export our Compass scorecard and component data?
Start by listing what Compass holds for you: components defined in compass.yml files, components created only in the UI, scorecard definitions with their thresholds, and team ownership. Compass files can be converted to catalog-info.yaml entities, while UI-only components usually need an export or manual capture. Scorecard rules rarely map one to one, so document each rule's logic in plain language before rebuilding it as a check. Confirm the export options and any data retention terms with Atlassian and your DX transition contact, since the phase-out timeline affects how long you have to extract the data.
What does a fixed-scope engagement cover for scorecards?
A fixed-scope engagement covers what is written into the agreed scope before work starts. For a scorecard rebuild, that typically includes the rule inventory, the Tech Insights configuration, the fact sources to connect, the entity data cleanup agreed with you, testing against your real entities, documentation and a handover to your team. Items such as custom dashboards, alert routing, additional integrations or ongoing rule changes are included only if the written scope names them. Ask for a scope document that lists each rule, each data source and each deliverable, and state who signs off on acceptance.
How DeployOpen scopes and delivers scorecard replacement
DeployOpen plans and implements self-hosted Backstage inside your own infrastructure, including the Tech Insights setup that replaces Compass scorecards. We agree scope, deliverables and timelines in writing before work starts, so the scorecard rules, data sources and handover items are settled before any build begins.
How we scope
- Inventory the current Compass scorecards, their criteria and the components each one applies to
- Agree in writing which facts, checks and maturity levels the Backstage replacement will carry
- Confirm the data sources for each check, such as CI results, ownership fields and documentation links
- Record which dashboards and alerts are in scope and which will be rebuilt outside Backstage
How we build
- Define Tech Insights facts and checks in code, alongside the catalog-info.yaml entities they evaluate
- Configure the tech-insights-maturity plugin so checks roll up into the rankings your teams use
- Test each check against sample entities and document expected pass and fail results
- Set up the hosting, PostgreSQL database and sign-in resolvers in your own environment, with guest access disabled
How we hand over
- Deliver the scorecard definitions, check configuration and runbooks as versioned files in your repositories
- Document how to add a new fact or check and how to review results with team leads
- Hand over the plugin upgrade procedure, including how monthly and weekly Backstage releases are tested first
- Walk your platform team through the alert and dashboard work that needs separate tooling
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover
- Retainers can cover Backstage version upgrades and checks against new security backports
- Scope changes are priced from a written change request, so your budget stays visible
- Ask for a quote based on the written scope and the number of components and teams involved
