Why Backstage scaffolder actions must be safe to retry
A template run fails halfway through. The repository was created in Git, but the run stopped before the component was registered in the catalog. The developer clicks retry, the scaffolder runs every step again, and now there is a second repository with a similar name, a duplicate webhook, or a half-configured project that someone has to clean up by hand. Scaffolder actions often have external side effects: they create repositories, set branch protection, provision cloud resources and call APIs that do not roll back when Backstage stops. A retry is only safe if each action can tell whether its work already happened and skip or reuse it. This page covers how to design custom actions that behave this way, using checkpoints, cancellation through abort signals and Start Over runs that reopen the form with the earlier parameters filled in.
This matters most to platform teams that publish custom actions for many product teams. A single action that is not retry-safe will produce duplicates for every team that uses it, and the failures show up as support tickets rather than in one place. Before you write or reuse an action, list every external call it makes, note which of those calls create something that a second attempt would duplicate, and decide what the action should check first, such as whether a repository with the target name already exists and whether it was created by the same task. Confirm with the owners of each target system which lookups and idempotency features their APIs provide, because the answer changes the design.
The decisions here also affect how you operate the portal. Someone must decide who may cancel a running task, what happens to resources created before cancellation, and whether a Start Over run should reuse values from the failed attempt or let the user change them. Those choices should be written down in the action's documentation and tested with the same dry run and review process you use for templates, so the first real failure is not the first test of the retry path. Ask your Backstage team to show you the current task list behavior in your installed version before you commit to a retry model.
Which retry-safety technique fits each action
This table compares five ways to keep a scaffolder action safe to retry, with what each protects against, what it misses and when it fits. Read each row against the actions in your template, since most templates need more than one technique. Confirm the exact checkpoint and abort-signal APIs in the scaffolder documentation for the Backstage release line you run before you build on them.
| Technique | Protects against | Does not protect against | Best fit |
|---|---|---|---|
| Checkpoints | Re-running completed steps after a later step fails, by recording a stored result and returning it on retry | Side effects that happened before the checkpoint was saved, such as a repository created just before a crash | Multi-step actions where each step creates a remote resource and returns an identifier |
| Existence check against a natural key | Duplicates when the same repository, project or entity name already exists on the Git platform | Two concurrent runs using the same name before either creates it, and name changes after creation | Repository creation where the name or path is unique within an organization or group |
| Abort-signal handling | Work continuing after a user cancels the task or the run times out, so partial resources are not left running | Failures that never send a signal, such as a crashed worker or a network error mid-call | Long actions that poll, wait or upload, where stopping early avoids orphaned work |
| Start Over with pre-filled parameters | Re-entering every field after a failed run and drifting from the original input | Duplicates from the failed run itself, because Start Over starts a new run with the same values | Failures caused by a bad input or an external outage, where the team should correct the form and rerun |
| Dry-run exclusion | Side effects during template testing, because actions skip remote writes when the run is a dry run | Real-run duplicates, since dry-run checks do not execute remote calls | Template review and unit testing before publishing to developers |
How to make a custom action retry-safe
This sequence turns a custom scaffolder action into one that can be retried after a partial failure without creating duplicate repositories, projects, or registrations, and that can be restarted from the same parameters. Each step ends with a check you can run against your own environment.
Inventory every side effect and its owner
List each call the action makes that changes something outside the scaffolder run, such as creating a repository, setting branch protection, opening a merge request, writing a catalog entity or publishing a container image. For each one, record the system that owns the resource and the credential the action uses. Keep the list in the action's source or a design note next to it. You are done when every write in the handler appears on the list and a reviewer can point to the owning system for each row.
Choose a natural key for each resource
For every side effect, pick an identifier the target system already enforces as unique, such as a repository name within an owner or group, a project path, or an entity reference in the catalog. Store the key in the task context as soon as it is computed so later steps and Start Over runs read the same value. You are done when each row in the inventory has a key, and you can explain in one sentence why two runs with the same parameters produce the same key.
Add an existence check before each create call
Before each create request, query the owning system with the natural key. If the resource exists and matches the parameters the template expects, record its identifier and skip the create call. If it exists but differs, fail with a message that names the conflict instead of overwriting it. Confirm the exact lookup endpoints and their behavior with your Git platform's API documentation, since the read and write permissions differ by platform and credential type. You are done when a unit test with a mocked lookup returning an existing resource shows no create call was made.
Place checkpoints after each external side effect
Write each successful side effect's output into the task checkpoint state before starting the next one. A retry then resumes after the last recorded checkpoint instead of repeating earlier writes. Keep checkpoint values small and free of secrets, and make sure the step name and key are stored together so a renamed step does not silently reuse stale data. You are done when a test that fails the action after the second of four writes, then retries, shows that the first two writes are not repeated.
Wire the abort signal into requests and cleanup
Pass the signal provided to the action context into every long-running HTTP request, polling loop and wait, and check it between steps. When the run is cancelled, stop waiting, and run only the cleanup you have designed for partial state, such as removing a repository created moments earlier when the next step has not begun. Be explicit that cleanup which deletes data is risky, so limit it to resources your action created in this run and verify ownership by key. You are done when cancelling mid-request stops the request promptly and a test confirms that no further create calls run after cancellation.
Enable Start Over with parameters carried forward
Configure the template so that a failed or cancelled task can be restarted from the task view with the original parameter values pre-filled. Keep the parameters that define identity, such as the component name and owner, unchanged on restart so the natural keys match the resources already created. Check the Backstage version your portal runs for the current Start Over behavior, and confirm it in your own instance. You are done when a failed run restarted with Start Over reuses the existing resources and reports no new ones.
Test by injecting failures between steps
Run the action in a test environment with a fault injected after each side effect, one at a time, then retry it. After each retry, count the new resources in the target system and compare them with the expected total. Include a case where the failure happens after the create call returns but before the checkpoint is written, since that is the most common source of duplicates. You are done when every injected failure ends with exactly one of each resource after a successful retry.
Confirm zero new resources on re-run
After the tests pass, run the template a second time with identical parameters against a completed environment. Check the Git platform, the catalog and any other owning system and confirm that the run creates zero new resources and reports the existing identifiers. Keep this check in your pipeline or review checklist for every change to the action, so a later edit that breaks idempotency is caught before developers see it. You are done when the re-run completes with no new repositories, projects, or catalog entities.
Retry design differs for create, update and notify actions
Create actions make new external objects such as repositories, projects, groups, container registries or cloud resources. The duplicate risk is that a first attempt succeeds on the remote system but the scaffolder never records the result, for example because the process stops or the response is lost. A retry then asks the platform to create the same name again, which either fails with a conflict or, on some platforms, creates a second object under a slightly different name. The preferred guard is an existence check before the create call, keyed on the name and location the template computed. Store the created identifier in the task checkpoint as soon as the call returns, so a retry reads it back instead of creating again.
- Existence check first: look up the repository or project by its full path before calling the create API, and branch to reuse when it already matches the expected owner and settings.
- Checkpoint the identifier: write the remote ID to the step output immediately after creation so later steps and retries reuse it.
- Deterministic names: derive names from parameters only, never from timestamps or random suffixes, so the existence check can find the earlier object.
- Reject mismatches: if an object exists but was created with different settings or by a different owner, fail with a clear message instead of adopting it silently.
- Cloud resources: confirm whether the provider offers idempotency tokens or client request IDs for the specific API you call.
Before shipping, run the action twice against a sandbox group with the same parameters and confirm the second run reuses the object, then kill the process between the create call and the checkpoint write and confirm the retry does not duplicate it.
Update actions change objects that already exist, such as committing files, opening merge requests, or changing repository settings and branch protection. Retrying these is usually safer than retrying creates, but they carry their own risk: a commit may land twice, a merge request may be opened once per attempt, and a settings change may overwrite a manual edit made in the meantime. The preferred guard depends on the operation. File writes should use upsert semantics, where the desired content is written to a fixed path and an unchanged file produces no new commit. Merge requests should be found by source branch before creation, and settings should be set to declared values rather than toggled.
- Upsert files: write the full desired content to a fixed path so repeating the step converges to the same state.
- Find before opening: query for an open merge request from the same source branch and reuse it instead of creating another.
- Declared settings: apply explicit values for each setting so a retry sets the same value rather than flipping a boolean.
- Branch naming: derive branch names from the run's stable parameters so retries target the same branch.
- Manual edits: decide whether a retry may overwrite changes a person made after the first attempt, and document the answer.
Before shipping, confirm that a retry after a successful commit produces no second commit, that a second run finds the existing merge request, and that the action's behavior is stated for each platform you publish to.
Notify actions send messages, call webhooks or trigger downstream systems such as chat posts, ticket creation, or external pipelines. These cannot be rolled back, so a retry that repeats a successful call produces a visible duplicate to a person or a system that may act on it. The receiving system often has no way to know the call is a repeat unless the sender provides a key. The preferred guard is a deduplication key: a stable identifier derived from the task and step, sent with the request, so the receiver or a local record can ignore repeats. Where the receiver offers no key support, accept duplicates but make them visible, for example by adding the task ID to the message so recipients can recognize a repeat.
- Deduplication key: build a key from the task ID and step name and include it in the request header or body when the receiver supports it.
- Local sent record: store a marker in the checkpoint after a successful send and skip the call on retry.
- Visible marker: when duplicates cannot be prevented, include the run ID in the message text so readers can recognize repeats.
- Ordering: send notifications after the create and update steps succeed, so a failed template does not announce a finished service.
- Failure messages: avoid sending a failure notice on every retry; send it once when the run finally fails.
Before shipping, confirm with each receiving system whether it honors a deduplication key, then test a retry after a successful send and check what the recipient sees.
Failure modes that create duplicates or orphans on retry
Each of these failures starts the same way: a scaffolder run stops partway through, and the next attempt repeats work that already happened. The cards below cover the side effects most likely to duplicate or leave debris, and the design fix for each one.
Repository created, registration failed
The action creates a Git repository, then the run fails before the catalog registration step. A retry creates a second repository under a new name, and the first one is never registered, so no owner sees it. Check whether the create step records the repository location in the task output and whether the register step reads that stored value instead of rebuilding the name from parameters. Make the create step look up an existing repository before creating one.
Retry generates a new resource name
A cloud resource action builds its name with a random suffix or timestamp. When the run fails after the resource exists and the step is retried, the action provisions a second resource with a different name, and billing continues on the first. Derive names deterministically from run-stable inputs such as the task ID or a normalized parameter, and check for an existing resource with that name before creating it. Ask the platform team which resources are tagged to the template so orphans can be found.
Aborted run leaves a partial branch or live token
Cancelling a run through its abort signal can stop an action between pushing a branch and cleaning up. The partial branch remains, and a credential created for that run may still be valid. Confirm that each action handles the abort signal at safe points, removes or reports partial state, and revokes short-lived tokens in a finally block. Treat a stale branch as a known outcome that the next run can detect and reuse.
Webhook or notification sent twice
A step that posts to a chat channel, calls an incident tool, or fires a webhook is often the last action before the run is marked complete. If the run fails afterward and restarts from an earlier step, the message goes out again and downstream systems may create duplicate tickets. Store a record of each outbound notification in the run's checkpoint data and skip the call when the record exists. Ask receiving systems whether they accept an idempotency key.
Parameters lost on restart
When a user clicks Start Over, the form can return empty, which pushes people to re-enter values they had already chosen and sometimes choose different ones. The new run then creates a second component that differs from the first in small ways that are hard to trace. Pre-fill the form from the previous run's recorded parameters, and show the user which values came from that run so they can change them deliberately.
Checkpoints placed before the side effect
A checkpoint that records a step as done before the step's side effect runs will skip that side effect on retry even though it never happened. The result is a repository or resource that was never created but is marked complete. Place each checkpoint after the side effect succeeds, and record the external identifier it produced, such as a repository URL or resource ID. On retry, verify that identifier still exists before skipping the step.
How retries interact with credentials and audit trails
A retry does not get fresh authority. When a run is resumed, each action runs again with the credentials configured for it, so the token scope and expiry decide whether a retry hours later can still act at all. A token that was valid when the run started may have expired, been rotated, or lost a permission in the meantime, and the failure then looks like an ordinary step error. Before you rely on checkpoints, confirm the lifetime of every token or GitHub App installation the template uses, confirm that the scopes cover the resumed step and not only the first attempt, and decide whether a retry should fail fast when credentials are stale rather than retrying blindly. Keep the token used by each action in the platform's secret store, referenced by name in the configuration, and never in the template's parameters.
Start Over should carry only non-secret parameters. When a user restarts a run with pre-filled values, the form is rebuilt from the earlier task's input, so anything typed into a parameter field, including a token pasted into a free text box by mistake, is copied forward and stored again in the task record. Design the parameter schema so that secrets never appear in it, validate that no field is marked as a password or accepts token-shaped strings, and review the stored task input for each template as part of testing. Credentials should always be resolved from configuration at execution time, which means a restarted run uses the same credential path as the original and a rotated secret takes effect without editing the template.
Each attempt should leave a traceable record in the task log, and the retry should not erase who asked for the resource. Record the attempt, the action that ran, the outcome and the identity that started the run, so a reviewer can see that a repository was created on the second attempt at the request of a named user. Confirm with your platform team how the scaffolder task log is retained and who can read it. Aborted runs also need a documented cleanup path: a partially created repository, a dangling branch or a half-registered catalog entity should have an owner, a runbook step and a check that identifies it. Finally, decide whether a Start Over run keeps the original requester or records the person who clicked the button, and write that rule into the template's ownership and audit guidance so the catalog and the Git platform agree on who requested the resource.
What retry-safe actions cost to build and who should build them
How much effort does making existing scaffolder actions retry-safe take?
Effort depends mostly on how many side effects each action performs and how many external platforms it touches. An action that only reads data or writes one repository is a small change, while an action that creates a repository, adds a webhook, registers a catalog entity and posts a notification has several points where a retry can duplicate work. Test depth matters too: proving that a rerun after a failure at each step produces no duplicates needs a test account on every platform involved. Ask any partner to size the work per action, not per template, because the same template can contain actions of very different risk.
Can an in-house platform team make scaffolder actions retry-safe on its own?
Yes, if the team already writes custom scaffolder actions and has someone who understands the Git platforms and external APIs the actions call. The work is mostly careful design: deciding what each step must check before it acts, storing checkpoints, and handling cancellation through abort signals. The harder part is testing against real platform behavior without touching production organizations. Teams with a small platform group often hand the test and review work to a partner so that engineers keep building the portal while the retry behavior is verified independently.
What does the work need from our team?
Expect to provide a list of the templates and custom actions in scope, the Git platforms and any cloud or ticketing systems they call, and the current template.yaml files. You will also need a sandbox Git organization or project where test runs can create and delete repositories, plus test accounts or service identities with scopes limited to that sandbox. A named reviewer who can approve changes to templates that developers use every day shortens the review cycle. Confirm these items in the written scope before work starts.
How long does retry-safe action work usually take?
Duration follows the same drivers as cost: the number of actions, the number of side effects per action, the number of platforms, and how many templates depend on each action. Once the action list is known, the partner can plan the work in phases, starting with the actions that create the most external resources. Ask for a timeline tied to the written scope and check whether it includes review by your engineers and a period of trial runs. No general duration is given here because it depends on your action inventory.
Does the Start Over feature require a Backstage upgrade?
Confirm this against your installed version before planning. Start Over depends on how the scaffolder stores task state and passes parameters to a new run, so the answer can differ between Backstage releases. Backstage ships a main release line monthly and a next line weekly, and its upgrade guide shows template changes between versions through the Upgrade Helper. If your instance is behind, include the upgrade in the scope so the checkpoint and Start Over behavior is tested on the version you will run.
How do we ask for a fixed-scope quote for retry-safe actions?
Most DeployOpen engagements are fixed-scope, so the quote follows from a written scope agreed before work starts. Prepare an inventory with one row per action: its inputs, the external calls it makes, the resources it creates, the templates that use it, and whether a dry run already exists. Add the test environments you can provide and the acceptance criteria, such as a rerun after each failure point producing no duplicate repository, webhook or catalog entry. With that inventory, a quote can show which work is one-time, such as the action changes and tests, and which work recurs, such as retaining tests as templates change.
What recurs after the retry-safe work is delivered?
The one-time work covers the action changes, checkpoint design, cancellation handling and the test suite. Recurring costs come from keeping that protection in place as Backstage, the Git platform APIs and your templates change. Each Backstage upgrade may change scaffolder internals, and each new template or action needs the same review for side effects. Ask any partner whether the retry tests will run in your pipeline, who maintains them, and whether ongoing support is handled through a retainer or by your own team.
What to confirm before relying on these patterns
The checkpoint, cancellation and Start Over behavior described on this page depends on your Backstage release, the action modules you install and the database settings you keep. Each card below names one thing to verify before you rely on a retry design in production.
Checkpoint and abort behavior by release line
Checkpoint handling and abort signal behavior are internal scaffolder details that can change between releases. Backstage ships a main release line monthly and a next release line weekly, so the behavior you test must come from the line you run. Confirm the exact behavior against the release notes and documentation for your version, then rerun your retry tests after every version bump. Do not assume a checkpoint written under one release will be read the same way under another.
Cancellation in third-party action modules
The scaffolder cancels a run through an abort signal, but each action module must read that signal and stop its outbound calls. A module that ignores the signal can keep creating a repository or sending a webhook after the run has been cancelled. Review every community or vendor action you install, check how it handles the signal in its source, and test cancellation in the middle of each step that writes to an external system.
Duplicate name handling on your Git platform
A retried step that creates a repository, project or branch can collide with a resource that already exists. Each Git platform responds differently: some reject a duplicate name, some return the existing object, and some allow a rename under a similar slug. Find out what your platform does for each create call the template makes, and decide whether the action should look up the resource by name first, reuse it, or fail with a clear message.
Start Over parameter restoration
Start Over is only useful if it restores every input the user entered. Test each parameter type in your template, including single-line text, enums, booleans, nested objects and any file or list inputs. Confirm that the pre-filled form shows the saved values and that hidden or conditional fields return to the state the user had before the failure. Missing or blank fields after a restart mean users retype parameters, which raises the chance of a different result on the second run.
Task store and database retention
A resumed run needs the checkpoint history and the task record to still exist. The scaffolder keeps its task state in the database that each backend plugin uses, so the retention and cleanup settings in your deployment decide how long a failed run can be resumed. Check how long your PostgreSQL backups and task records are kept, and confirm that a run that failed a day or a week earlier still has the state it needs. Test a resume after the cleanup job has run.
Authorization on retried runs
A retry or Start Over should run under the same identity and permissions as the original run, and it should not let a user reach an action they could not start the first time. Confirm which user or token the resumed run uses for each action, and check that changes to template permissions take effect before a resume. Also confirm that the sign-in resolver maps each user to the same catalog identity, so the audit trail for the retry matches the original attempt.
How DeployOpen scopes and delivers retry-safe scaffolder actions
DeployOpen treats retry safety as a design requirement for each custom action, not a patch added after duplicates appear. We scope the actions a template depends on, build them with checkpoints, cancellation and Start Over behavior, and hand over tests and documentation your platform team can maintain inside infrastructure you control.
How we scope
- We list every custom action in your templates and mark which ones create repositories, webhooks, or cloud resources.
- The written scope names the side effects each action produces and the external system that holds each one.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts, and a quote follows from that scope.
How we build
- We design each action with checkpoints that record completed side effects so a retry skips work already done.
- We wire abort signals into long-running steps so cancellation leaves no half-created repository or stray resource.
- We build Start Over runs with pre-filled parameters and test each action with dry runs and retries before release.
How we hand over
- You receive action source, tests and a retry matrix showing the expected result after each failure point.
- Documentation covers checkpoint storage, naming conventions and how to add a new side effect safely.
- Your platform engineers walk through the run history and failure cases so they can maintain the actions themselves.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, covering Backstage version bumps that touch scaffolder code.
- We review new or changed actions against the retry design before they reach developers.
- Confirm with your team which support hours, response expectations and escalation paths belong in the retainer scope.
