Backstage high availability

Running several Backstage backend replicas only improves availability if every replica sees the same state and no replica does work that another one already did. The main risks are local in-memory caches that drift apart, scheduled tasks that run twice or not at all, and rollouts that leave replicas on different plugin versions. Each of these is a decision you make in configuration and operations, not something the platform handles automatically, so the section below should be read as a checklist for your own deployment.

In-memory caches diverge. Backstage supports Redis, Memcache, Infinispan or in-memory caching, according to the architecture overview. In-memory caching is fine for a single process, but with two or more replicas each process holds its own copy of cached catalog responses, plugin data and session-related values. A user who refreshes an entity may see different results depending on which replica answers, and an invalidation on one replica does not reach the others. Choose a shared cache backend such as Redis for any multi-replica deployment, and confirm in your release's configuration reference which plugins read through the cache so you know what must be shared and what can safely stay local.

Scheduled tasks run in duplicate or not at all. Catalog ingestion, search indexing and many plugin tasks run on a schedule inside the backend. If every replica runs its own timer, a catalog refresh can execute once per replica, multiplying load on source systems such as GitHub or GitLab and risking rate limits. Coordination can also fail in the other direction: if a replica holding a lock crashes mid-run and the lock is never released, the task may be skipped until its lease expires. The safe pattern is a single shared PostgreSQL instance per environment that all replicas use, so task coordination is recorded in one place. Do not assume this behavior from memory. Read the scheduling and task documentation for your Backstage release, check which tasks use distributed locking, and confirm the lock timeout values in configuration.

Rollouts leave replicas on different versions. During a rolling update, old and new replicas serve traffic at the same time. If the database schema or a plugin's API changed between versions, a request routed to an old replica can fail or write data the new replica does not expect. Keep all @backstage packages on the same release line, as the upgrade guide describes, and confirm in the release notes whether a version requires a database migration that must finish before old replicas are drained. Make the migration step an explicit gate in your deployment pipeline rather than a side effect of the first pod to start.

Verify before you trust it. Before going live, check scheduling behavior in two places: the documentation for your exact release, and the backend logs during a test run with at least two replicas. Confirm that each scheduled task logs one start and one completion per interval, and that a deliberately stopped replica does not stop the task. Then run a rolling restore of the backend under representative load and watch for errors, duplicate ingestion events and cache inconsistency. Record what you observed so the result can be repeated after every upgrade.

A Backstage high availability design is only proven when a test shows it keeps serving users through a failure and recovers to known data. Run each test below against a staging copy of the production topology before go-live, repeat it after material changes such as a new replica count, a database engine change or a cache swap, and schedule the restore test on a fixed cadence. Each test should have a named owner, a written pass condition agreed before the run, and an evidence folder that an auditor or incident reviewer could read later without asking the platform team questions.

Start with the replica and node tests. While a user session is active and someone is browsing catalog pages, terminate one backend replica. The pass condition is that the session either continues or re-authenticates cleanly, catalog pages keep loading from the surviving replicas, and no request returns a server error for longer than your agreed threshold. Keep the load balancer access logs, the replica termination timestamp, and the error rate chart for that window. Then cordon and drain a node that hosts a replica and confirm the scheduler reschedules it, the readiness probe passes before traffic returns, and scheduled work does not stall. Record the drain command output, pod or container events with timestamps, and the time to return to full replica count. Confirm the exact Kubernetes or container platform commands with your platform team, since they depend on your deployment.

The cache test checks degradation, not just uptime. Take the shared cache offline (for example, stop the Redis instance or block its network path) and confirm the portal keeps serving catalog data, with slower responses rather than data loss. Verify that the catalog and TechDocs data in PostgreSQL are unchanged by running a row count and checksum comparison before and after. Keep the cache outage start and end times, application logs showing cache connection errors, and the comparison output. A failed test here should trigger a review of cache client timeouts and fallback behavior, because a cache outage that turns into a request storm against PostgreSQL is a capacity risk, not only a performance blip.

The restore test is the most important one for data. Restore the most recent PostgreSQL backup into a separate environment, point a non-production Backstage instance at it, and compare catalog entity counts, software template entries, owner references and location records against the source. Use comparison queries that run on both sides and save the restore tool logs, the backup identifier and timestamp, and the diff output. Time the restore and record it, since that figure is what you will quote to leadership as recovery time. Also confirm that TechDocs can be rebuilt from object storage on a fresh replica, and that scheduled tasks do not run twice by checking that each scheduled job writes one execution record per interval after the new replica joins. A failed test should block the change that caused it, open an incident record, and trigger a backup or restore fix before the next release cycle.

Agree in advance what a failed test triggers. Treat a failed failover or rescheduling test as a blocker for production rollout, a failed restore comparison as a data recovery incident with a root-cause review, and a duplicate scheduled task as a defect in coordination configuration to fix before any scaling change. Ask your implementation partner to deliver the test scripts, the evidence template and the pass criteria in writing as part of the fixed scope, and confirm with your security and compliance leads which evidence they need to retain and for how long.

After go-live, the operating routine should watch the four parts of the high availability design that can fail independently: the Backstage backend replicas, the PostgreSQL databases, the shared cache and the scheduled tasks that coordinate work across replicas. Alert on the backend health endpoints for each replica, not only on the load balancer address, because a load balancer can report healthy while one replica is failing. Track the number of ready replicas against the count you deployed, and alert when the gap lasts longer than your agreed tolerance. Watch database connections per replica so that a pool which grows after a deployment is caught before it exhausts the server's connection limit. Confirm the exact health endpoint paths and metric names for your Backstage version and plugin set, since these vary with what you install.

Backups need a schedule and retention period that your organization sets from its recovery point and recovery time requirements, and these should be written down before go-live. Cover every PostgreSQL logical database that Backstage creates for its backend plugins, because catalog, scaffolder and other plugin data live there. Confirm that the TechDocs storage in object storage has its own versioning or replication setting, since it is stored separately from the database. Treat a successful backup job as unproven until someone restores it. Run restore tests into a separate environment on a fixed cadence, record the time taken and the data checked, and keep those results as evidence. Treat the cache as rebuildable, and have the restore plan describe how the cache is warmed and what slower responses users should expect in the meantime.

Backstage publishes a main release line monthly, on the Tuesday before the third Wednesday of each month, and a next release line weekly on Tuesdays. Use the main line as the production baseline and run the next line in staging to catch breaking changes early. Place a recurring test cycle on the calendar after each monthly release: upgrade staging, run the replica failover and restore checks, then promote to production within a window your team defines. Assign one person to review the weekly next line release notes so that changes are understood before the monthly release arrives. Backstage supports two adjacent even-numbered Node.js releases, which were Node.js 22 and 24 as of Backstage 1.46.0, so confirm the supported pair for your version and schedule any Node.js change in the same cycle.

The Backstage versioning policy says high and critical severity vulnerabilities will always be backported to releases from the last 6 months if feasible, and other security fixes may be backported. Your patch window therefore depends on staying inside that six-month band. A deployment that falls behind the monthly line may find that a fix exists only in a newer release, which turns a routine bump into a larger upgrade. Write down who monitors the Backstage security advisories, who approves an emergency version bump, and the target time for applying a high or critical fix to staging and then production. Confirm those targets with your risk owner, because the project's policy describes backport intent and does not set your internal deadlines.

Keep runbooks for three procedures: failover of a backend replica or database, restore from backup into a clean environment, and cache rebuild after a flush or outage. Each runbook should name its owner, the commands or console steps, the checks that confirm success and the rollback step. At close-out, ask for a handover package that includes an architecture diagram showing replicas, database, cache, object storage and the load balancer; the configuration repository with environment overlays and references to secrets; the current runbooks; the evidence from the latest restore test; and an owner list naming on-call contacts for the platform, database, identity provider and each plugin. Ask for these items in writing so the operating team can run the platform without depending on the implementation team.