Testing Backstage templates with dry run

Testing Backstage Software Templates with dry run before developers see them

A template that creates real repositories, webhooks and catalog entries cannot be tried casually in production, so the platform team needs a way to test it before any developer runs it. Dry-run testing lets you exercise a Software Template without creating the external resources it would normally produce. It matters most when a template calls actions that write to source control, register webhooks or add entities to the catalog, because a mistake there leaves artifacts that someone must clean up by hand. Software Templates are one of the core built-in features listed in the Backstage architecture overview, so the same review discipline applies to every template your organization publishes.

Dry-run support depends on how each action is written. Your custom actions should check ctx.isDryRun before any side effect and return fake or placeholder outputs in dry-run mode, so downstream steps still have values to read. Confirm in the Backstage scaffolder documentation and in your Backstage version exactly how dry runs are invoked, what the task log shows, and which built-in actions honor the flag. Do not assume that every third-party action behaves the same way. Where an action does not support dry run, the template should be structured so the risky step is isolated and can be reviewed separately.

Dry run is one layer, not the whole test. Handler unit tests catch logic errors in the action code with mocked clients, and a review gate ensures a second engineer reads the template and its actions before they are registered for all developers. Plan who signs off, which test template or sandbox namespace is used for end-to-end runs, and who is responsible for cleaning up any artifacts that a live test still creates. Ask your Backstage implementation partner or internal team to document these steps alongside the template so that every change follows the same path.

Which test approach fits which template

Some templates only fetch a skeleton directory, replace placeholders with the parameter values and write files to the workspace. They create no repository and call no external API, so most of their risk sits in the template file itself: a wrong variable name, a missing if guard or a broken path. A dry run is usually enough for these, because the scaffolder runs the steps and shows the rendered output without publishing anything.

Dry run first: Run the template with representative parameters and inspect the rendered files in the output view before any publish step.
Edge parameters: Test optional fields left blank, values containing spaces or special characters, and the largest set of conditional branches.
Rendered content review: Read the generated files for leftover placeholders, wrong casing and incorrect license or owner fields.
Path and name checks: Confirm generated file and directory names are valid on the target Git platform.
What it cannot prove: Dry run does not show how the skeleton behaves once it is committed, built or deployed, such as whether the generated project compiles or passes CI.

Confirm that the skeleton itself builds in a throwaway checkout before you rely on the template for new services.

Templates that call publish or provisioning actions create repositories, webhooks, branch protection rules or cloud resources. Dry run prevents these side effects by skipping the external calls, so it is the first gate. But it cannot confirm that credentials, permissions or API behavior match production. For those checks, use a sandbox organization or a staging instance of the Git platform or cloud account, with credentials scoped to that environment.

Skip side effects in dry run: Confirm each action checks ctx.isDryRun and logs the call it would have made, so reviewers see the intended change.
Sandbox organization: Run one full, non-dry execution against a test organization that mirrors your naming, group and branch protection settings.
Credential scope: Verify the token or app installation has only the permissions the action needs, because dry run never exercises them.
Cleanup path: Delete the sandbox repositories and webhooks after each run, and record which template version created them.
What it cannot prove: A sandbox does not reproduce your production policies, org rulesets or rate limits, so repeat the final run in a staging instance before wide release.

Check that the sandbox settings match the production settings you will rely on, and write those comparisons into the review record.

Custom actions written with createTemplateAction contain your own logic: input validation with zod schemas, transformations, calls to internal APIs and file writes. Dry run only tells you the handler was invoked and skipped its side effects, not whether the logic is correct. Unit tests call the handler directly with mocked context objects and assert the outputs, the errors and the calls made, so they catch logic regressions quickly and without network access.

Handler unit tests: Test each branch of the handler with valid, invalid and boundary inputs, and assert that schema validation rejects bad values.
Dry run branch checks: Test that every path guarded by ctx.isDryRun produces the intended log output and makes no external call.
Mocked clients: Replace API clients with stubs that record calls, so tests fail when a handler sends an unexpected request.
Error handling: Assert that failures return actionable messages and do not leave partial output that a retry would duplicate.
What it cannot prove: Mocks do not match the real API's responses, auth flows or timing, so run an integration test against a sandbox before publishing the action.

Review the mocks against the real API documentation at each upgrade of the platform client.

Testing options and when each fits

This table compares four ways to test a scaffolder template before developers use it: dry-run execution, ctx.isDryRun branches, handler unit tests with mocked clients, and a sandbox Git organization or staging portal. Read each row for effort, what it catches, what it misses and who should own it. The main takeaway is that no single option covers both logic errors and live platform behavior, so most teams combine a cheap check on every change with a live run before release.

Option	Effort to set up	What it catches and misses	Who should own it
Dry-run execution	Low: run the template in dry-run mode and read the logged steps	Catches parameter flow, step order and skeleton rendering. Misses real API responses, permissions and repository settings	Template authors, during development
ctx.isDryRun branches	Moderate: add checks in custom actions so side effects are skipped during dry runs	Catches side effects that would run in production, such as repository or webhook creation. Misses bugs in the code path that only runs when the flag is off	Action authors who maintain shared modules
Handler unit tests with mocked clients	Moderate to high: write mocks for each Git platform client and keep them in step with the SDK	Catches branching logic, input validation and error handling in custom handlers. Misses wrong endpoints, token scopes and platform-side rejections	Engineers who write custom actions, reviewed in the same pull request
Sandbox Git organization or staging portal	High: a separate organization, credentials, catalog and cleanup routine that someone must keep running	Catches real platform behavior, branch protection, permissions and catalog registration. Misses differences from production policy unless the sandbox mirrors it	Platform team, with a named owner for sandbox credentials and cleanup

Setting up dry-run tests for a template

This sequence takes a template from code change to org-wide availability, with a check at each step so that dry runs prove the template's logic without creating repositories, webhooks or other external resources. Each stage produces evidence the next stage depends on.

Make each custom action dry-run aware
The platform team that owns the custom scaffolder actions adds dry-run handling to each one, so the action reads the dry-run flag from its context and returns early before any write to Git, a registry or a cloud API. Built-in actions should also be checked against your own templates, because not every action behaves the same way in dry-run mode. Confirm the current dry-run behavior of each built-in action you use in the Backstage documentation for your release. This step is done when every action in the template's path has a documented dry-run branch and the team can point to the line where it returns.
Add isDryRun guards around external writes
Within each handler, the developer wraps every call that changes external state, such as creating a repository, setting branch protection, adding a webhook or registering a catalog entity, in a check on ctx.isDryRun. Read-only calls, like fetching a skeleton or looking up a group ID, can run in dry-run mode so the rendered output reflects real inputs. Log the skipped operations with their intended target so reviewers can see what a live run would do. The check for this step is a code review where every write call is either guarded or explicitly listed as safe to run in dry-run mode.
Write handler unit tests with mocked clients
The developer writes unit tests for each handler that replace the Git, cloud and catalog clients with mocks, then asserts which methods were called and with what arguments. Tests cover both the live path and the dry-run path, and they check that the dry-run path makes zero write calls. Use a fixed set of input parameters that match the template's form fields, including invalid values, so validation failures are caught before anyone sees them in the portal. The step is complete when the test suite passes in continuous integration and the write-call assertions fail if a guard is removed.
Run the template in dry-run from the portal
A template author opens the template in the Backstage scaffolder, enters representative parameters and starts a dry run from the portal interface. The task log should show each step completing, the rendered skeleton files and the list of skipped external operations. Compare the output against the expected layout and naming rules, and verify the template's output links point to placeholders rather than real repositories. The check is a task log with no failed steps and no entries claiming a repository or webhook was created.
Confirm no external resources were created
After the dry run, a reviewer checks the sandbox organization on the Git platform and the cloud or catalog accounts the template touches, looking for any new repository, webhook, team, secret or catalog entity created by the run. Compare the audit or event logs for the time window of the test against the list of expected operations from the dry run. Any unexpected change means a guard is missing, so the handler returns to the code step before the template moves forward. The check is an empty diff between the pre-run and post-run state of the sandbox.
Run a live test in a sandbox organization
The template is run for real in a dedicated sandbox organization or account, with a non-production identity and credentials scoped to that sandbox. The run creates real resources, which the team then inspects for correct naming, branch protection, catalog registration and the time-saved annotation if one is used. Test cleanup and the Start Over path as well, so the team knows how to remove sandbox resources. The check is a sandbox result that matches the design document line by line.
Promote the template through peer review
A second engineer, who did not write the template, reviews the code, the test results, the dry-run log and the sandbox outcome before the template change merges. Approval should be recorded in the repository history, and the template should be published to the catalog only after the merge. Use the access controls for who may register Template entities so that the change reaches developers through a single controlled path. The step is complete when the merged commit, the review approval and the catalog entry share the same version, so an auditor or on-call engineer can trace what is live.

Security risks in test runs and how to contain them

Dry-run and test runs execute the same handler code as production runs, so they can reach the same credentials, systems and people. The cards below cover the main ways a test can cause harm and the controls that keep test runs contained. Each control is something to verify in your own environment before templates reach developers.

Dry-run tests holding production tokens

A test run that loads the same integration credentials as production can create repositories, write files or call APIs with full rights, even when the handler is meant to skip side effects. Check which secrets the test environment can read, not only which secrets the template declares. Use a separate, scoped token for tests, with permissions limited to the sandbox organization or project. Confirm that a missing or wrong token fails the test rather than falling back to a broader credential.

Sandbox runs leaking into real organizations

A test that targets a live GitHub, GitLab or Bitbucket organization can publish repositories, register components or send notifications that other teams see. Point test runs at a dedicated sandbox organization, group or project, and make the target path a parameter that the test configuration sets explicitly. Ask who owns the sandbox, how it is cleaned up, and whether its catalog entries are kept out of the production catalog.

Test data containing secrets

Test parameters, fixture files and captured task logs often end up holding real credentials, personal data or internal hostnames. Keep secrets out of template parameters, skeleton files and test fixtures, and read them from the backend's secret store at run time. Review fixture files in the same pull request as the template, and confirm that log output from test runs is redacted and retained for a defined period.

Templates reaching developers without review

A template that was edited directly on the main branch, or registered by someone outside the review group, can skip the dry-run checks entirely. Require that every change to template.yaml, actions and fixtures passes through a pull request with the test suite and a named reviewer. Record who can register Template entities in the catalog, and audit that list on a regular schedule so that new registrations stay visible.

Scaffolder runtime exposure

Test runs execute the scaffolder on the same backend, so a crafted parameter or custom action can reach internal network addresses, the file system or other plugins. Limit which hosts the test backend can reach and run it in an isolated network segment where possible. The scaffolder hardening topic covers these controls in depth, including URL reader restrictions and action permissions, so review it before allowing custom actions to run in test.

Questions teams ask before testing Backstage templates

How much effort does a template test suite take to build?

The effort depends mainly on how many templates and custom actions you have, how many branches each template takes, and how many Git platforms and external systems each action touches. A template with a few conditional steps and only built-in actions needs far less test work than a set of custom actions that call your ticketing, cloud or repository APIs. Handler unit tests with mocked clients are usually the largest share of the work, because every action needs cases for valid input, failure responses and dry-run behavior. Ask any partner to break the estimate into templates, custom actions, test harness setup and a review cycle, and to state which items are one-time and which recur when templates change.

Can our own platform engineers run dry-run tests without outside help?

Yes, if they already know TypeScript, the Backstage backend plugin model and how the scaffolder runs steps. The dry-run pattern itself is simple: each action checks whether the context reports a dry run, skips external writes, and returns the values a real run would produce. The harder skills are writing reliable mocks for Git and API clients and deciding which behavior a dry run must still prove. Teams new to Backstage often underestimate how much of the scaffolder model they need to understand before tests are trustworthy. A short review by someone who has written custom actions before can prevent months of tests that check the wrong thing.

What do we need to provide for a dry-run testing engagement?

Plan for four inputs before work starts. First, a sandbox Git organization or project group where test repositories can be created and deleted without touching production namespaces. Second, test credentials scoped to that sandbox, with the smallest permissions each action requires. Third, a non-production Backstage instance connected to the same catalog model and authentication setup as production, so template parameters and ownership behave the same way. Fourth, named reviewers from the platform and security teams who will approve templates. Ask for a written list of these inputs in the scope, including who in your organization grants access and how long that takes.

How long does a template review gate add to publishing?

The gate adds whatever time your reviewers need to read the template, run the test suite, and run at least one end-to-end test in the sandbox, plus any rework cycles. Most of that time depends on reviewer availability and how many changes a template receives, not on the test run itself. You can shorten the gate by automating checks for schema validity, required owner and lifecycle fields, and action naming, so reviewers focus on logic and security. Agree on a service expectation for reviews within your change process, and confirm how urgent fixes bypass or shorten the gate.

How do dry-run results differ from a real template run?

A dry run shows that the template parses, that parameters validate, that steps execute in order and that actions return the expected outputs without calling external writes. It does not prove that a repository was created, that branch protection applied, that a webhook fired, that a catalog entity was ingested, or that the credentials have the permissions the real call needs. Those behaviors need a sandbox end-to-end run. Treat dry-run results as the fast check on logic and treat sandbox runs as the check on integration and permissions. Document which of each your review process requires before a template reaches developers.

Which parts of a dry-run test suite need the most maintenance?

Maintenance concentrates on the parts that talk to external systems. When a Git platform changes its API, when a plugin upgrade changes scaffolder types, or when a Backstage release alters how the context exposes dry-run status, mocks and assertions may need updates. Templates that change often also need their tests updated with each parameter or step change. Ask for an explicit plan covering who updates tests after Backstage version bumps, how often the sandbox run is repeated, and how failing tests are triaged before a release. These costs recur for as long as the templates stay in use.

Do dry-run tests change how we handle security for templates?

They help, but they do not replace a security review. A dry run can show that an action refuses unsafe input, such as a repository name outside an approved pattern, and that secrets are not written to logs or outputs. It cannot show whether the template grants too much access once it runs for real. Include security reviewers in the gate for templates that create cloud resources or repositories, and confirm that test credentials never have production scope. Restricting who may run or register templates is a separate control that should be designed alongside testing.

Limits to confirm in your environment

A dry run shows how a template behaves in the scaffolder without creating anything, but it only covers part of the path a real run takes. The cards below list what to confirm with the Backstage docs, your Git platform and your own environment before you treat a passing dry run as release approval.

Dry-run behavior per action

Not every action treats dry run the same way. Some actions skip their side effects, some log what they would have done, and custom actions only behave correctly if their authors checked ctx.isDryRun. For each action your template calls, confirm in the Backstage docs for your release line what a dry run does, and read the action source if the docs are thin. A silent side effect in a custom action is the most common gap.

Scaffolder behavior by release line

Scaffolder behavior and the Node.js runtime can change between Backstage releases. The main line ships monthly and the next line ships weekly, so a template tested on one line may behave differently after an upgrade. Record the exact release you tested against, and retest dry runs after each version bump, including the Node.js versions your release supports.

Git platform sandbox options

A dry run does not contact your Git platform in the same way a live run does, so it cannot confirm that a token can create repositories, that the target organization or project exists, or that branch protection rules apply. Ask your platform team whether a sandbox organization, group or project exists for end-to-end testing, and who may create test repositories in it. Clean up test repositories on a fixed schedule.

Permissions and token scopes

A dry run runs under the identity you test with, which may hold broader or narrower rights than the production run will use. Confirm which credential the scaffolder uses for each action, whether it is a GitHub App, a personal token or a service account, and which scopes it holds. Check the permissions for creating repositories, teams and registering catalog entities separately, because a dry run may pass while the live step fails.

Rate limits and quotas

Git platforms and cloud APIs enforce rate limits and quotas that a dry run seldom reaches. A template that makes many calls in a live run can hit limits that never appeared in testing, especially when many developers run it at once. Ask each platform owner for the limits that apply to your credential type and whether your organization has a higher allowance, and test the largest parameter set you expect under load.

Webhook and catalog delivery

A dry run does not create webhooks, and it does not prove that events from a new repository reach Backstage, that the catalog picks up the new Component or that a later registration step succeeds. Confirm how your catalog discovers new repositories, the delay between creation and registration, and what happens when a webhook delivery fails. Test these in a sandbox after the dry run passes.

A review process that keeps untested templates out of the catalog

A dry run proves that a template's steps execute, but it does not decide whether a template should reach developers. Put a review gate in front of every change to a template.yaml or to a custom action it calls. Name one platform owner who approves template changes and a security reviewer who approves any change that touches credentials or creates resources. Keep the two roles separate where your size allows, so the person who wrote a change is never the only approver. Record the approvals in the same Git pull request that carries the change, so the history of what was approved travels with the code.

The review checklist should cover four things. First, the parameters: every field has a validation rule, a sensible default and a description a developer can understand without asking. Second, the actions used: each action id is one your team has reviewed, and any custom action has its own tests passing in the same change. Third, credential scope: the token or GitHub App installation behind each publish step grants only the access that template needs, and you have confirmed that scope in your own environment rather than assuming the defaults. Fourth, output links: the success page points to repositories and catalog entries that exist and that a new user can open. Version templates through Git tags or a metadata.version-style convention your team chooses, so a rollback is a single revert rather than a rebuild.

Stage every approved change to a pilot group before the whole organization sees it. Register the new version under a separate template entity or a restricted audience, have a small set of teams run it for real work, and collect failed task logs and feedback for a fixed period you set in advance. Promote the version only when the pilot runs complete cleanly. Limiting who can register Template entities and who can run them is covered on the sibling topic on restricting template access, so this section assumes that control exists. Confirm with your platform team which Backstage release line you run, since the Backstage release and versioning policy sets how often scaffolder behavior can change under you.

How DeployOpen scopes and delivers template testing

DeployOpen plans and delivers dry run testing for Backstage Software Templates inside your own infrastructure, so template changes are exercised before any repository or webhook is created. Engagements are fixed-scope, with the test harness, review process and handover agreed in writing before work starts.

How we scope

Inventory the templates, custom actions and Git platforms that need dry run coverage before writing any code.
Agree in writing which actions must honor ctx.isDryRun and which handlers need unit tests.
Confirm who approves template changes, how reviews are recorded, and which teams receive templates first.

How we build

Add dry run support to custom actions so side effects such as repository creation are skipped during tests.
Write unit tests for action handlers using mocked Git platform clients and captured context values.
Set up a review workflow where template changes pass dry run tests and a named reviewer before publication.

How we hand over

Document how to run dry run tests locally and in your continuous integration pipeline, with the steps written out.
Deliver the test suite, review checklist and template authoring notes as files in your own repository.
Walk your platform team through the review process so they can add new templates without outside help.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Retainer work can cover updates to dry run tests when Backstage versions or Git platform modules change.
Ask for the support scope, response arrangements and upgrade handling in the written proposal before signing.

Testing Backstage Software Templates with dry run