Open-source alternative to Cortex Scorecards: the short answer
If your engineering leadership uses Cortex scorecards to set engineering standards and inform renewal and investment decisions, you need those rules to keep running inside your own infrastructure. The main open-source route is Backstage with the Tech Insights plugins, where facts define the data collected from catalog entities, checks evaluate rules against those facts, and the tech-insights-maturity plugin adds maturity rankings and categories to checks. The community plugins also include the tech-insights-backend-module-jsonfc module, which calculates boolean facts from JSON rules, so many standards can be expressed without custom code. What Backstage does not give you out of the box is packaged scorecard reporting and dashboards comparable to Cortex's, so you will plan and build that layer yourself.
Be clear about the licensing before you commit. Self-managed Cortex still requires a JWT license key and access token from the Cortex account team, so running it in your own Kubernetes cluster does not remove the subscription. Backstage is licensed under Apache-2.0 and the project itself offers no paid edition, so the software cost is zero, but your costs move to engineering time for building checks, maintaining plugins and operating the platform. Ask your team to inventory each Cortex scorecard rule and mark whether it maps to a fact or check, needs a custom module, or requires reporting you must build.
Which Cortex scorecard capabilities map to Backstage
This table compares the main Cortex scorecard capabilities with the Backstage Tech Insights options that can rebuild them. Read each row for what maps directly, what needs custom code, and what the community plugins may not cover. Confirm current plugin behavior in the community-plugins repository before you commit to a design.
| Cortex capability | Open-source option | Coverage notes |
|---|---|---|
| Scorecard rule definitions | Tech Insights jsonfc module with JSON rules that compute boolean facts | Good fit for pass or fail checks on catalog fields. Complex conditions across several sources may need custom TypeScript checks. |
| Data sourcing from catalog entities | Tech Insights facts read from Software Catalog entities | Covers entity metadata and annotations. Data from outside the catalog, such as CI results or incident tools, needs custom fact retrievers. |
| Maturity levels or ranks | tech-insights-maturity plugin, which adds rankings and categories to checks | Closest match to Cortex levels. Confirm how the plugin defines levels and whether it fits your rubric without code changes. |
| Scorecard grouping by team or domain | Catalog ownership fields with Tech Insights filtering in the portal UI | Grouping follows the owner and system fields in catalog-info.yaml. Cross-cutting groups that Cortex builds in its UI may need custom views. |
| Historical trends and reporting | Custom backend code storing check results, with Grafana or another reporting tool | Gap to plan for. Tech Insights evaluates current state, so trend history needs a store, a schedule and a dashboard you build and operate. |
| Owner notifications | Custom backend code or a notification plugin connected to check results | Gap to plan for. Confirm which notification channels and triggers current plugins support before you assume alerts work out of the box. |
Which Backstage approach fits which scorecard need
The tech-insights-backend-module-jsonfc module is the lightest route. You describe each fact as a JSON rule that evaluates catalog entity data and returns a boolean, such as whether a Component has an owner, a runbook link or a non-empty lifecycle field. Rules live in configuration rather than in application code, so platform staff who do not write TypeScript can review and change them.
- Best for: yes or no standards, such as ownership set, on-call contact present, or a TechDocs site published.
- Trade-off: rules can only express what the catalog data can express. Cross-referencing a CI system or a vulnerability database needs another fact source.
- Who maintains it: the module is a community plugin in the Backstage community plugins repository, so your team owns the configuration and tracks upgrades against the Backstage release line.
- Operational note: every rule change should be reviewed like code, because a single misplaced key can make a whole check silently pass or fail.
- Confirm: the exact rule syntax and supported operators for your Backstage version in the Tech Insights documentation.
Before committing to JSON rules, list each scorecard criterion and mark whether the catalog already holds the data it needs.
The tech-insights-maturity plugin adds rankings and categories on top of checks. Instead of a pass or fail per rule, an entity receives a maturity level based on the checks it passes, grouped into categories such as reliability, security or documentation. This is the closest Backstage equivalent to a scorecard with levels, which is what many teams buy Cortex for.
- Best for: tiered standards where teams progress from basic to advanced, such as Bronze, Silver and Gold levels defined by your engineering council.
- Trade-off: the ranking is only as good as the underlying checks. Weak or inconsistent checks produce rankings that teams will challenge.
- Who maintains it: this is also a community plugin. Your platform team owns the level definitions and the check set, and must test them after each Backstage upgrade.
- Operational note: agree on level thresholds and category weights with application owners before publishing a ranking, since changes later shift every team's score.
- Confirm: that the plugin's ranking model matches your level names, and whether it supports the category grouping your reporting requires, in the Tech Insights documentation.
Use this tab once the checks themselves are stable, because rankings amplify whatever the checks measure.
Custom Tech Insights checks are TypeScript code added to a backend plugin or module. They handle logic that JSON rules cannot express, such as calling an internal API to confirm a service is registered in a change management system, comparing a deployed version against a policy, or applying conditions across several entity relationships. Your engineers write, test and deploy these checks like any other backend code.
- Best for: rules that depend on external systems or multi-step logic that a declarative rule cannot represent.
- Trade-off: each custom check is code your organization must version, test and secure. Every Backstage upgrade may require changes.
- Who maintains it: your own engineering team owns the code, which means the cost of ownership sits with you, not with a vendor or the community.
- Operational note: keep custom checks narrow and documented, and reuse the JSON or maturity layer for the rest so the custom surface stays small.
- Confirm: the current Tech Insights extension points and the Backstage release your portal runs, using the Backstage repository and the versioning policy to plan upgrades.
Ask your team how many custom checks they expect to maintain for the next two years before choosing this route as the main approach.
How Tech Insights facts and checks evaluate catalog entities
Scorecards in Backstage are only as good as the catalog behind them, so the first decision is the descriptor standard, not the plugin. Each service, library, API or system is described by a catalog-info.yaml file in its repository, using kinds such as Component, API, Resource, System, Domain, Group and User. The Tech Insights plugins read entities from the catalog and evaluate them. A fact is a data point collected for an entity, such as whether a spec.owner field is set, whether a lifecycle value is present, or whether an API entity carries a definition. A check is a rule that evaluates one or more facts and returns a pass or fail result for each entity. The maturity plugin then turns the set of passed checks into rankings and categories, which is what a scorecard view displays.
This flow means a scorecard can only measure what the descriptors contain. If metadata.tags, spec.owner, spec.system or spec.lifecycle are missing or inconsistent across repositories, the checks will fail for reasons unrelated to engineering quality, and teams will stop trusting the results. Before writing any check, agree on a minimum descriptor standard: which kinds are mandatory for which repositories, which owner references must resolve to a Group or User entity, and which lifecycle values are allowed. Then run a catalog audit to count how many entities lack each field. The gap between your standard and the current catalog is the real migration backlog, and it usually costs more time than the plugin configuration itself.
Confirm three details against your installed plugin versions before you rely on the results. First, check the Tech Insights workspace in the Backstage community plugins repository for the exact modules you need, including the JSON rule calculator and the maturity plugin, since their configuration keys and package names can change. Second, confirm how often fact collection runs and where fact data is stored in your deployment, because scorecards are only as current as the last collection. Third, decide who owns the standard and who fixes failing descriptors, since a check that nobody acts on becomes a report rather than a control. Document these decisions with the descriptor standard so the scorecard definitions can be reviewed like any other engineering policy.
Running Backstage scorecards beside Cortex during the transition
A parallel run keeps Cortex as the system of record for scorecard reporting while Backstage checks prove they reach the same answers. The cards below cover the parts that tend to break: entity matching, ownership data, sign-in identity and the decision to switch off Cortex.
Keep Cortex as the reporting source
Until the Backstage results are trusted, leave Cortex as the source for leadership and audit reporting, and treat Backstage output as a shadow report. Changing the reporting source early makes any mismatch hard to explain. Name the owner of the cutover decision in writing before the parallel run starts, and confirm that your Cortex contract or self-managed license still covers the reporting period you plan to run.
Compare results for the same entities
Pick a fixed set of components, such as a representative sample across teams and languages, and record each rule result from both systems on the same day. Most disagreements come from different definitions of a passing rule, stale data or a fact that reads a field Cortex populates differently. Log every mismatch with its root cause so the rule, not the reviewer, is corrected.
Share ownership data between systems
Scorecards depend on owners, teams and system membership, so an ownership drift shows up as a false failure. Decide which system owns each field, then generate catalog-info.yaml files from that source and check them in Git. Run a periodic diff of owner fields between Cortex and the Backstage catalog so that a team reassignment does not change one report and not the other.
Align identity with one sign-in resolver
Backstage authentication documentation advises configuring sign-in resolvers so every provider resolves to the same identity, and not offering guest access in production. Without that, the same engineer can appear as two users, which splits ownership and approvals. Test the resolver with a user who signs in through each provider you enable before you rely on scorecard results.
Retire Cortex scorecards once results match
Define the exit criteria before the run ends, such as an agreed period with no unexplained mismatches across the sample and sign-off from the teams that consume the reports. Export the final Cortex scorecard history for records first, since that data may not be available after the subscription ends. Then remove the Cortex scorecards, keep the Backstage checks under version control, and confirm the reporting dashboards still work.
Rebuilding scorecards in Backstage step by step
This sequence moves each Cortex scorecard rule into a Backstage Tech Insights check that runs against catalog entities, with maturity levels reported alongside. The output is a validated scorecard that owners recognize and reporting that no longer depends on Cortex.
Inventory Cortex scorecard rules and owners
Export the list of scorecards, their rules, thresholds and levels from the Cortex account, then record who owns each rule and who acts on its results. Include the teams that read the scorecards in reviews and the initiatives tied to them. The step is done when every rule has a written definition, a named owner and a note on the entities it applies to.
Standardize catalog-info.yaml fields for checks
Identify which metadata each rule reads, such as owner, lifecycle, system, tags or API definitions, and make sure those fields are present in every catalog-info.yaml descriptor. Descriptors use kinds such as Component, API and System, and missing fields are the most common reason a check returns no result. The step is done when a catalog query shows the required fields populated for the in-scope entities.
Install and configure Tech Insights and maturity plugins
A platform engineer adds the Tech Insights backend and the JSON facts module from the Backstage community plugins repository, then the maturity plugin for ranking categories. Confirm the plugin versions match the Backstage release line your portal runs, since plugin compatibility follows the core release. The step is done when the Tech Insights backend starts and the fact retriever runs without errors on a test entity.
Translate each rule into a JSON check
Write one fact per data point, such as whether an entity has an owner or a documented runbook, using the JSON facts module, then define a check that evaluates the fact against the rule's threshold. Where a Cortex rule cannot be expressed as a JSON fact, plan a custom check and record it as extra development work. The step is done when every rule in the inventory maps to a named fact and check, or to a documented custom check with an owner.
Map checks to maturity levels
Group the checks into the categories and levels that the scorecard should display, so each entity receives a ranking rather than only a pass or fail result. Agree the level thresholds with the same owners from the inventory step, since the rankings change what teams see. The step is done when each check carries a category and a level, and a sample entity shows its ranking in the portal.
Validate results against Cortex for sample services
Choose a set of services that represent different languages, teams and lifecycle stages, then compare each entity's pass or fail results and level in Backstage against the Cortex scorecard for the same entity. Investigate every mismatch and classify it as a rule translation error, a missing descriptor field, or an intended change agreed with the owner. The step is done when mismatches are resolved or documented and the owners sign off on the sample.
Cut over reporting and communicate changes
Point dashboards, reports and review processes at the Backstage results, then notify each team of the new rule definitions, rankings and where to find them in the portal. Keep the Cortex scorecards read-only for a period agreed with the owners, and confirm that any exported history needed for audits is stored in your own infrastructure. The step is done when the reporting consumers read from Backstage and the teams have acknowledged the change.
What reporting needs extra work in Backstage
The Tech Insights plugins evaluate facts and checks on each catalog entity, which covers current pass and fail status well. Historical trend views, executive rollups, initiative tracking and notifications are a different matter. Before you commit to a rebuild, confirm with the plugin maintainers which of these the versions you install actually provide, because community plugins change between releases and the answer may differ by version. The Tech Insights source in the Backstage community plugins repository is the place to start that check.
Expect to build or assemble the missing views yourself. A common pattern is to keep the portal's check results in its PostgreSQL database and point a separate analytics layer, such as Grafana, at that data for trend charts and portfolio-level rollups. Initiative tracking, such as a campaign to raise a maturity level across every team, usually needs custom catalog metadata and a reporting query, not a plugin toggle. Notifications, such as alerting a team lead when a service drops below its target level, need an integration with your chat or ticketing system. Ask each vendor or internal engineer proposing this work to state which database tables and schema they depend on, since a Backstage upgrade can change them.
Budget for this custom work as a separate line item from the portal itself. The cost drivers are the number of reports, how often the schema changes with each Backstage release, and whether the analytics layer is new to your team. Ask for a written list of the reports in scope, the data source for each, and who maintains them after handover. Without that list, the Cortex features you relied on for executive reporting may return as a long backlog after cutover.
Questions buyers ask before rebuilding Cortex scorecards
What drives the cost of rebuilding Cortex scorecards in Backstage?
The cost is mostly people time, not license fees, because Backstage is Apache-2.0 licensed and has no paid edition from the project itself. The main cost components are the engineers who define facts and checks, the platform team that runs the Backstage instance, the infrastructure it runs on (Kubernetes and PostgreSQL in most setups), and any outside help you buy. Costs rise with the number of scorecard rules, the number of catalog entity kinds you evaluate, the amount of data you must pull from source systems, and how many teams must agree on definitions. Ask a consultant to price each component separately in the written scope so you can see what is one-time and what recurs.
How long does it take to rebuild scorecards with Tech Insights?
Duration depends on how many scorecard criteria you already have, how clean your catalog data is, and how many systems hold the facts you need. Teams usually start with a small set of checks on services they already track, then widen coverage. Before you commit to a date, ask for a timeline tied to specific deliverables: the list of facts, the list of checks, the maturity categories, and the reporting view. No duration is given here because the research supplied does not establish one for this work. Your own catalog audit is the best input to any estimate.
What do we need from our engineers and platform team?
You need engineers who own services and can state what good looks like for each standard, a platform team that can run and upgrade Backstage, and someone who understands your identity provider and sign-in setup. Backstage supports 19 built-in authentication providers, and its documentation advises configuring sign-in resolvers so every provider resolves to the same identity and not offering guest access in production, so identity work belongs in the plan. Your catalog-info.yaml files must be accurate, because scorecards only evaluate what the catalog contains. Plan time for owners to review and correct their entries.
Can our own team rebuild Cortex scorecards without outside help?
Yes, if you have engineers comfortable with Backstage plugins, YAML and TypeScript, and a platform owner who can keep the instance running. The Tech Insights community plugins, including the jsonfc module that calculates boolean facts from JSON rules and the maturity plugin that adds rankings and categories, give you a starting point. Be realistic about gaps: these are community plugins, so you own their maintenance and any custom data source you need. Outside help is most useful for scoping the rules, wiring data sources and handing over documentation. Ask any provider to confirm which parts they will build and which your team keeps.
Is Backstage free to use, and what recurring work does it create?
Backstage is free to use under the Apache License 2.0, and there is no paid edition from the project itself. The recurring work is what matters. The project ships a main release line monthly and a next release line weekly, so someone must plan regular upgrades. The versioning policy says high and critical vulnerabilities are backported to releases from the last 6 months where feasible, which means you should stay within that window. Node.js support is also limited to specific versions, so runtime upgrades follow the Backstage version. Plugin upkeep, infrastructure, monitoring and on-call support recur for as long as the portal runs.
What should we ask a consultant to include in a written scope?
A written scope should name the deliverables, the owners and the acceptance criteria. Ask for the full list of scorecard criteria with the source system for each fact, the checks and their rules, the maturity levels and categories, the reporting view and who can read it, and the ownership data each check depends on. Ask how identity and sign-in will be configured, how upgrades and plugin updates are handled, what documentation and handover are included, and what is not covered. Fixed-scope engagements agree scope, deliverables and timelines in writing before work starts, and DeployOpen works that way. Request a quote built from that scope.
Do Tech Insights checks cover reporting out of the box?
Not fully. Tech Insights gives you facts, checks and maturity rankings on catalog entities, which covers evaluation. Executive dashboards, historical trends, and reporting across initiatives usually need extra work, such as a data store, a reporting layer and access controls for who sees which team's results. Confirm the reporting requirements before you design checks, because the data model affects what you can report later. Ask any provider to show how scorecard history is retained, how it is exported, and how reporting access follows your existing roles.
Risks to plan for when replacing scorecards
Rebuilding Cortex scorecards in Backstage moves the work of keeping checks current onto your team. The risks below connect because each one changes how often you upgrade, which plugins you can trust and how clean the data behind every score must be.
Plugin maturity and maintenance
The Tech Insights and maturity plugins are community plugins, so their maintenance is not guaranteed by the Backstage project itself. Before you commit, check each plugin's recent commit history, open issues and compatibility notes against the Backstage version you run. Record who owns each fact and check in your team, because a stalled plugin leaves scorecards silently out of date. Confirm current status in the community plugins repository.
Upgrade effort on two release lines
The main release line ships monthly and the next line ships weekly, so a portal that tracks either line needs a standing upgrade routine. Plugins that rely on internal APIs can break between versions, which means every upgrade may also require plugin updates and test runs. Decide which line you follow, assign an owner for upgrades and schedule them as recurring work in your budget, not as a one-time project.
Security backport window
The versioning policy says high and critical vulnerabilities are backported to releases from the last six months where feasible, while other security fixes may be backported. That means a release that falls outside the window may not receive fixes for serious issues, and you must upgrade to stay covered. Map your running version against the policy at each review and set a maximum allowed age for production. Ask your security team to include Backstage in its vulnerability scanning.
Node.js version support
Backstage supports exactly two adjacent even-numbered Node.js releases, and the supported pair changes as the framework moves forward. A Node.js upgrade can therefore force a Backstage upgrade, and the reverse is also true. Confirm the pair for your target Backstage version before you build your container image, and include runtime patching in your platform calendar. Older base images in your registry are a common source of drift.
Authentication and resolver configuration
Backstage ships with 19 built-in authentication providers, and each must resolve a user to the same identity, or ownership and scorecard results will attach to duplicate or wrong users. Configure sign-in resolvers deliberately and test them with real team accounts before rollout. Do not enable guest access in production, because it weakens who can see and change catalog data. Review the authentication documentation with your identity team before go-live.
Descriptor data quality
Scorecards are only as accurate as the catalog-info.yaml files they evaluate. Missing owners, stale lifecycle values or components without descriptors will produce misleading scores, and Cortex's ownership data will not migrate automatically. Run a catalog audit before you define checks, count entities without owners and agree on required fields. Assign each team a remediation deadline and make descriptor validation part of the code review for new services.
How DeployOpen scopes and delivers scorecard replacement
DeployOpen is a Chainscore Labs company that plans, implements, customizes, integrates and supports self-hosted open-source platforms inside the client's own cloud accounts, data centers or air-gapped networks. For a scorecard replacement, that means the Backstage deployment, the catalog descriptors it reads, the Tech Insights fact and check definitions, and the maturity rankings sit in your infrastructure, under your identity provider and your change control. Most engagements are fixed-scope, so the work starts only after scope, deliverables and timelines are agreed in writing.
The scope document is where scorecard logic gets decided. Before any rule is written, the agreement should list each scorecard, every fact it depends on (for example, whether a catalog-info.yaml carries an owner, a lifecycle value and a documentation link), each check and its pass condition, and which maturity levels the tech-insights-maturity plugin will assign. It should also name who signs off on the rule set, because a check that a team disputes after go-live will reopen the scope. Ask for the rule set as a written appendix to the agreement so that later changes go through the same process.
Ongoing support, upgrades and continuous testing are available as retainers, which matters because Backstage releases on a fixed monthly and weekly cadence and plugin code changes between versions. DeployOpen also prepares systems, controls and evidence for compliance assessments, such as the scorecard history and access records an auditor may ask for, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies do that work, so confirm with your assessor which evidence they will accept before the scope is signed.
