Why Backstage token signing keys reset on restart and when to pin them
If your production Backstage signs users out, or plugin-to-plugin calls start failing, after every restart or redeploy, the likely cause is that the token signing keys were generated automatically and never persisted. An auto-generated key lives in memory or local state. When a new process starts, it creates a new key pair, and any token signed by the previous key no longer verifies. The fix is to configure key stores that stay the same across restarts, so every process loads the same keys. Two places in Backstage use these keys: the user key store, which signs the tokens issued to people after sign-in, and the plugin key store, which signs the tokens backend plugins use when calling each other.
You need pinned keys in three situations: when you run more than one backend replica, because each replica would otherwise generate its own keys and reject tokens issued by its peers; during rolling deploys, when old and new pods overlap and must accept each other's tokens; and whenever sessions must survive a restart at all. If you run a single instance with short sessions and can accept a forced sign-in after each deploy, auto-generated keys may be acceptable for a sandbox, but they are a poor fit for production. Plan the key stores before the first multi-replica rollout, not after users report failed sign-ins.
Before you change anything, confirm the exact configuration keys, the key store options and the supported signing algorithms in the Backstage authentication documentation for the release you run, since these details can differ between versions. Ask your platform team where the private keys will be stored and who may read them, because a pinned private key is a credential that can mint valid sessions. Record the owner of the key material and the rotation procedure from the start, so that the first rotation is a planned change and not an incident response.
Key store options and when each one fits
This table compares auto-generated keys, a static user key store and a static plugin key store on purpose, restart behavior, replica behavior and operating effort. Read each row against your deployment shape: how many backend replicas you run and how often they restart. The main takeaway is that anything beyond a single throwaway instance needs keys that persist outside the process.
| Option | Best fit | What survives a restart | Replica behavior | Operational effort |
|---|---|---|---|---|
| Auto-generated keys | Local development and disposable test instances | No. Keys are regenerated on restart, so existing sessions and tokens stop validating | Not suitable for multiple replicas, since each instance can hold different keys | Lowest to start, but restarts break user sessions |
| Static user key store | Persisting keys that sign and verify user-facing tokens | Yes, when the key material is supplied from a file, environment variable or secret reference | All replicas must load the same key set from the same source | Moderate: you supply, protect and rotate the key pair yourself |
| Static plugin key store | Persisting keys for plugin-to-plugin and service calls | Yes, when the key material is supplied the same way as the user key store | All replicas must load the same key set, and it is separate from user keys | Moderate: a second key set to supply, protect and rotate |
| ES256 (ECDSA P-256 with SHA-256) | Asymmetric signing where verifiers need only the public key | Depends on key storage, not the algorithm | Public key can be shared widely; private key stays with the backend | Requires generating and handling an EC key pair; confirm the supported key format in the authentication documentation |
| Other algorithms listed in the docs | Only where your security team has a stated requirement | Depends on key storage, not the algorithm | Confirm which algorithms the current release accepts before choosing | Confirm the list and configuration keys in the authentication documentation |
| Key from a file path | Mounted secrets in Kubernetes or a file managed by your secrets tooling | Yes, if the file persists across pod restarts | Mount the same secret to every replica | Rotation means updating the mounted secret and restarting pods |
| Key from an environment variable | Simple deployments where your platform injects secrets at start | Yes, if the variable is set from a persistent secret store | Set the same value on every replica | Visible to anyone who can read the process environment; confirm your platform's exposure |
| Key from a secret reference | Teams with an external secrets manager | Yes, if the reference resolves to a persistent store | Every replica resolves the same reference | Depends on your secrets manager; confirm the reference syntax in the authentication documentation |
How to configure static Backstage token signing keys
This sequence moves a Backstage instance from auto-generated signing keys to a static ES256 key pair, so sessions and issued tokens survive backend restarts and redeployments. Each step ends with a check you can run in a non-production environment before the change reaches production.
Confirm the current key behavior
Before changing anything, record how the instance behaves today. Restart the backend in a non-production environment, sign in, and note that the session is invalidated after the restart. Check the backend startup logs for a message indicating that a signing key was generated. Confirm the exact key store settings for user and plugin keys in the Backstage authentication documentation, because configuration key names change between releases and should not be copied from older examples. Done when you have a written baseline showing the session loss and the generation message.
Generate an ES256 key pair outside the cluster
Generate an ES256 (elliptic curve P-256) private key and its public key using a standard tool such as OpenSSL, run on a workstation or a controlled build host rather than inside the cluster or the application container. Keep the private key out of the Git repository and out of any shared drive. Create one pair for user keys and one for plugin keys if your key store configuration keeps them separate. Done when each pair has been created, the public key can be derived from the private key, and both files are stored only in a restricted location your team controls.
Store the private keys in the secret store
Place each private key in the secret store your organization already uses for Backstage, such as a Kubernetes Secret with encryption at rest or an external secrets manager, and mount or inject it into the backend environment as a file or variable. Restrict read access to the service account that runs the backend and to the operators who must rotate keys. Do not place private keys in app-config files. Done when the backend pod or process can read the key at startup and no other workload or user role can read it.
Add the public keys to the key store configuration
Reference the public keys in the static key store configuration for user keys and plugin keys, using the key names and paths listed in the Backstage authentication documentation for your release. Include the public key material only, so the configuration can be reviewed and stored in version control without exposing signing capability. Keep the private key path or secret reference in the deployment configuration, not the public configuration file. Done when the configuration validates and the backend starts with the static keys referenced.
Deploy to a non-production environment
Roll the change out to a non-production Backstage instance that matches production in authentication provider, database and deployment method. Start the backend and watch the startup logs for the absence of any message that reports a newly generated signing key. Confirm that the authentication flow completes with your configured sign-in resolver. Done when the backend starts cleanly, sign-in succeeds, and the logs contain no key generation message for the static keys.
Verify that a session survives a restart
Sign in and keep the browser session or a token issued by the instance. Restart the backend without changing any key material, then call an authenticated endpoint with the same token or reload the page. The session should still validate. Repeat the restart once more and check the logs again for key regeneration. Done when the token issued before the restart still validates after each restart and the logs show no key regeneration.
Document the rotation and ownership plan
Record who owns each private key, where it is stored, who can read it, and how a new public key is added alongside the existing one during rotation so that tokens signed with the old key keep validating until they expire. Confirm the rotation mechanics against the Backstage authentication documentation for your version before you write the runbook. Done when the runbook has been reviewed by the security owner and a rotation has been rehearsed once in the non-production environment.
How to rotate Backstage signing keys by adding a new key first
Rotate a Backstage signing key by adding the new key pair alongside the old one, keeping the old public key available until every token signed with it has expired, and only then removing it. The old private key should stop signing new tokens as soon as the new key is in place, but its public half must remain in the key store for as long as tokens issued before the change are still valid. Removing the old key early causes sessions to fail verification at the moment users are least expecting it, so the overlap is the part of the procedure that needs a deliberate decision rather than a default.
The length of the overlap window depends on the token lifetime configured in your deployment, and that setting is not something to assume from defaults. Confirm for your release how long issued tokens remain valid, including any refresh behavior of your sign-in provider, and read the authentication documentation at Backstage: Authentication for the settings that govern it. Set the retirement date for the old public key at or after the longest lifetime you confirm, plus a margin agreed with the security team. Record that date in the change ticket so the retirement step is not forgotten after the rotation team has moved on.
Assign ownership before scheduling anything. The platform team usually performs the change, the security team approves the key lifecycle and the retirement date, and the change approval board records both steps as one change with a rollback plan. Rehearse the full sequence in staging first: add the new key, confirm new sign-ins and plugin calls verify, confirm tokens issued before the change still verify, then retire the old public key and repeat the checks. Schedule production rotation in a low-traffic window, keep the previous key files available for a rollback, and make sure the private key is removed from any location outside the approved secret store once the rotation is complete.
Which key setup fits each Backstage deployment
A single backend instance in a test or pilot environment is the simplest case. One process holds the signing keys, so a static key store backed by a local file is usually enough. The main risk is forgetting that automatically generated keys change on every restart, which signs out every user and makes sessions look unreliable. Keep the setup simple, but do not let pilot keys or pilot file locations carry into production unchanged.
- Key store: Use the static key store with an ES256 key pair you generate yourself, so a restart does not invalidate existing sessions.
- Private key location: Keep the private key file on the host or volume that runs the backend, readable only by the account that runs Backstage.
- Delivery: Mount the file from a local path or a container volume. Do not copy it into the image or commit it to the application repository.
- Backups: Confirm that backups of the host or volume are protected, because a copy of the private key can sign tokens.
Before go-live, confirm that a restart keeps existing users signed in and that the key file is excluded from version control. Review the Backstage authentication documentation for the current key store configuration keys, since these settings can change between releases.
With several replicas, every pod must accept tokens signed by the same keys. Each replica that holds a different key set will reject sessions issued by another, so the key material must be identical across the fleet and changes must roll out in a controlled order. Generated keys that differ per pod are not workable at this scale, and a rolling update can expose that problem quickly.
- Key store: Use the static key store with the same public keys and key IDs in the configuration for every replica.
- Private key location: Store the private key in an external secrets manager or in a Kubernetes Secret protected by encryption at rest. It should never be part of the container image.
- Delivery: Inject the secret into a file path that the container reads at startup, using the same source for all replicas, so a new pod receives the same keys as the old ones.
- Rotation: Add the new public key to the configuration, roll out all replicas, and only then switch signing to the new key. The rotation section covers this sequence in detail.
Before go-live, test sign-in through each replica and confirm that restarting a single pod does not break sessions handled by another. Confirm with your platform team who can read the secret and how access to it is logged.
Some environments have no external secrets manager, or their network rules block outbound calls to one. In these settings the static key store still works because it reads local files, so the deciding questions are how the private key is generated, moved and stored without a managed service. The team needs a documented custody process, because nothing else will track who holds the key or when it was last replaced.
- Key generation: Generate the ES256 key pair on a trusted workstation or in an approved offline process, not on a shared host.
- Transfer: Move the private key to the Backstage host through a channel your security team approves, and delete any temporary copies after the transfer.
- Storage: Place the file in a root-owned directory with restricted permissions, on an encrypted volume where the platform supports it.
- Custody record: Keep a written record of who generated the key, who can read it, and where it is backed up.
Before go-live, confirm that Backstage starts and signs users in with no outbound calls to a secrets service, that the key does not appear in logs or configuration dumps, and that your restart and rotation runbook has been tested on the restricted network. Ask your security team to confirm the custody and backup rules in writing.
Protecting Backstage token signing keys in production
Once keys persist across restarts, the private key becomes a credential that can mint sessions for your whole portal. The cards below assign each protection to the team that acts on it, from file access and secret storage to change control and incident response.
Restrict private key reads to the backend
Only the Backstage backend service identity should be able to read the private key file or secret. Platform engineering sets this up through file permissions or Kubernetes RBAC, scoping read access to the workload's service account and nothing else. Check that developers, CI jobs and other namespaces cannot list or mount the secret. The public key is safe to share, so keep the two artifacts separate.
Keep keys out of config and source
Private keys must never appear in app-config.yaml, environment files committed to Git or container images. Application teams own the config files, so they should reference the key by path or secret name instead of embedding its contents. Add secret scanning to the app repository so an accidental commit is caught before it is pushed.
Encrypt keys at rest
Store the private key in a secret store that encrypts data at rest, such as a Kubernetes Secret with encryption at rest enabled, or an external secrets manager synced into the cluster. The platform team owns the secret store and should confirm its encryption and backup settings with the provider. Avoid copies in ticket attachments, chat history or shared drives.
Limit who changes key configuration
Changes to key store settings, signing algorithm or key file location alter how every session is validated. Restrict write access to the deployment repository and the secret store to a named group, and require review for changes to these values. Security owners should approve any change that adds or removes a key, since a mistake here can break sign-in for all users.
Audit every key change
Record who created, read, replaced or deleted a signing key, and when. Use the secret store's audit log together with Git history for configuration changes, and forward both to the security team's log platform. Compare these records during reviews so an unexpected key addition shows up as an event, not a surprise during an incident.
Treat a suspected leak as an incident
If a private key may have been exposed, treat it as a security incident. The security owner should add a new key, remove the compromised one from the key store, and invalidate existing sessions so tokens signed with the old key stop working. Confirm the exact session invalidation steps with your Backstage version before an incident occurs. Also review sign-in resolvers and guest access settings against the Backstage authentication documentation while you are in the middle of the response.
What persisting Backstage signing keys costs and what it asks of your team
What drives the effort of pinning Backstage signing keys?
The effort depends on four things: whether you already run a secrets manager that your platform team operates, how many environments need keys (development, staging and production each need their own), how many backend replicas must read the same key material, and what your audit requirements ask you to record. A team with a working secrets manager, a deployment pipeline and a staging environment has a short path. A team that must first build secret delivery into Kubernetes or a cloud account has more work before the key change itself. Ask a partner to quote each of these as a separate line so you can see which items you would absorb internally.
Can our internal team pin Backstage keys without a partner?
Yes, if your engineers can edit the app configuration, generate ES256 key pairs, store the private keys in your secrets manager and roll the change through your pipeline. The work is configuration and operational discipline rather than custom code. Most internal teams benefit from a written checklist covering who holds the private keys, how the public keys are published to the backend, and how a restart is tested in staging before production. A partner is useful when you lack that experience, need the change done within a fixed window, or want an independent review of how keys are stored and accessed.
How long does the initial key change take, and what sets the timeline?
The timeline depends on the same drivers as cost: the state of your secrets manager, the number of environments, the number of replicas and the change control your organization requires. We do not publish a duration for this work because it varies with those inputs. The steps are generating key pairs, adding the static key stores, testing a restart and a rotation in staging, and then promoting the change. Ask any provider, including us, for a schedule in the written scope that lists each step, the environments it covers and the approvals it needs.
What must our team provide before a key change can start?
You need to provide three things. First, access to the secret store where private keys will live, with rights to create and read the entries that the Backstage backend uses. Second, access to the deployment pipeline that rolls out configuration changes to each environment. Third, a staging environment that matches production closely enough to test a backend restart and a key addition without affecting real users. Name the owners of each item in the scope, because delays in these areas usually cost more time than the key work itself.
What recurs after go-live, and who pays for it?
Pinned keys are a one-time build, but they create recurring work. Your team must rotate keys by adding a new public key, deploying it, switching signing to the new key and then retiring the old one once tokens signed with it have expired. You must also review who can read the private keys at a set interval and record those reviews. If you pay a partner for ongoing support, the recurring cost covers rotation runs, access reviews and upgrade testing. If your team does this internally, the cost is staff time, which should be planned as a standing operational task.
What evidence should a compliance reviewer expect for key handling?
Reviewers usually look for a record of where private keys are stored, who can access them, when keys were created and rotated, and how access was reviewed. Your secrets manager's access logs and change history can supply much of this. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports or certifications. Independent auditors and certification bodies make those determinations. Confirm with your auditor which evidence format they accept before the change is scoped.
What should we ask for in a written quote for this work?
Ask the provider to state the scope in writing: the Backstage version and deployment method, the environments covered, the secrets manager used, the number of backend replicas, the rotation procedure to be documented, and the evidence to be delivered. Ask which items are one-time and which are recurring, and whether rotation runs or access reviews are included or billed separately. Request the assumptions behind the schedule and the list of items you must supply. A fixed-scope quote with these items lets you compare a partner with your internal estimate on equal terms.
What to confirm about key behavior in your Backstage release
The key store behavior described on this page depends on the release you run, so treat each item below as a question to answer, not a fact to assume. Confirm them against the release your organization follows and record the answers before you change production configuration.
Supported configuration keys and formats
Confirm the exact configuration keys for the static user and plugin key stores, and which key formats the release accepts, including whether ES256 key pairs are supported as written in your version. Check the key names against the authentication configuration reference for your release line, not against examples copied from older guides. A key name that is silently ignored leaves the portal on auto-generated keys, which reset on every restart. Ask the platform owner to confirm the keys by starting a test instance and checking that the same token verifies after a restart.
Handling of removed public keys
Confirm how the release treats a public key that you remove from the listed keys. Some releases continue to accept tokens signed by a key that is no longer published, while others reject them immediately. This decides whether rotation is a two-step process (add the new key, then retire the old one after tokens expire) or a single change. Test with a token signed by the old key, remove that key from the store, and record whether verification fails, and when.
Plugin caching of keys
Confirm whether plugins outside the core authentication flow cache signing keys or fetch them on each request. A plugin that keeps a copy of the public key may keep trusting a retired key, or may fail to validate a new one until its process restarts. List each backend plugin your portal runs that verifies Backstage tokens, and ask its maintainer or check its source for key caching behavior. Record the refresh interval, if one exists, in your rotation runbook.
Session behavior during a key change
Confirm what happens to signed-in users and open sessions when the signing key changes. Some setups force users to sign in again, and others keep existing tokens valid until they expire. Find out whether the release keeps user sessions across a key change or invalidates them, and whether the answer differs for user tokens and plugin-to-plugin tokens. Tell the service desk what to expect, and schedule the change for a window your users can absorb.
Restart or rolling update requirement
Confirm whether a change to the key store takes effect only after a restart, or whether the backend reloads configuration without one. Also confirm whether replicas of the backend that run in parallel must all receive the same key store before any of them sign tokens. If the backend runs more than one replica, a rolling update can leave old and new replicas issuing tokens that the other side does not yet trust. Test the change in a staging environment with two replicas before you schedule production work.
Differences across your release line
Confirm how key handling differs between the release line your organization follows and the newer line. Backstage ships a main release line monthly and a next release line weekly, so a key behavior change may appear in one line before the other. Review the release notes for each version between your current release and your target, and check the versioning policy for the line you follow. Record the version you tested so that the result stays tied to a specific release.
How DeployOpen scopes and delivers Backstage key hardening
DeployOpen plans, implements and hands over Backstage key hardening inside your own infrastructure, so the signing keys, their storage and their rotation procedure are documented and owned by your team. Work is agreed as a fixed-scope engagement with written deliverables, and ongoing support is available as a retainer.
How we scope
- Inventory where your Backstage instance currently generates keys and whether restarts invalidate existing sessions.
- Agree in writing which key stores, key formats and secret locations are in scope for your environment.
- Confirm with your team who approves rotation, who holds private key access and which systems must be updated first.
How we build
- Configure a static key store for user and plugin keys, with ES256 key pairs generated under your own key management process.
- Test that tokens issued before a restart remain valid after the instance restarts, in a non-production environment first.
- Rehearse a rotation by adding a new key, confirming both keys validate, then retiring the old key in a controlled window.
How we hand over
- Document the key locations, file permissions, environment variables and the rotation runbook your operators will follow.
- Provide verification steps so your team can confirm the active key set after every deployment or configuration change.
- Walk your engineers through the procedure and record the deliverables that were agreed in the written scope.
How we support
- Offer ongoing support as a retainer for key rotation reviews, Backstage version upgrades and configuration changes.
- Review key handling again when Backstage releases change authentication or plugin token behavior, as your scope allows.
- Confirm the support terms, response expectations and included activities in writing before the retainer starts.
