Why Backstage health checks and Kubernetes probes need deliberate setup
A platform team runs a Backstage backend on Kubernetes, and the pod restarts during every deploy and again whenever the catalog processing load climbs. The usual cause is not Backstage itself but probes that were copied from a generic template and never matched how the backend starts and responds. Backstage exposes a health check endpoint that Kubernetes probes can call, and that endpoint is the starting point for a deliberate setup. Before you write any probe, confirm the exact health check route, the plugin or configuration needed to enable it, and how it behaves in your Backstage version by checking the current documentation for that release, since the defaults and route names are version-specific.
The two probes answer different questions. A readiness probe decides whether the pod should receive traffic right now, so a backend that is still loading the catalog or cannot reach its database should be taken out of rotation without being killed. A liveness probe decides whether the process should be restarted, so it should fail only when the process is wedged. If both probes point at the same endpoint with the same thresholds, a slow start during a deploy can look like a dead process, and Kubernetes restarts the pod before it finishes initializing. That restart loop is the failure most teams hit first.
You need deliberate probe settings whenever startup time is long or variable, which is common once the catalog holds many entities, plugins perform initial ingestion, or the database is shared across environments. Plan the probes together with the deployment: decide how long a cold start may take in your environment, measure it in a staging cluster, and set a startup probe or initial delay that covers that window. Ask your platform team to confirm resource requests and limits for the backend, because a pod that is throttled during ingestion can fail its liveness check even when the code is healthy.
Probe types and settings and when each fits
This table compares the three Kubernetes probe types and their timing settings for a Backstage backend. Read each row as a decision: which probe to use, what it should check, and how the timing settings change for a large catalog. The main takeaway is that only readiness should depend on the database and catalog state, while liveness should only confirm the process is responsive.
| Probe or setting | What it answers | When it fits a Backstage backend |
|---|---|---|
| Startup probe | Has the process finished booting yet? | Use it when catalog ingestion and plugin startup take a long time, so liveness does not kill the pod during boot |
| Readiness probe | Can this instance serve traffic right now? | Point it at the health endpoint that checks the database and dependencies, so unhealthy pods leave the service without restarting |
| Liveness probe | Is the process stuck and in need of a restart? | Keep it shallow, confirming only that the process responds, so a slow PostgreSQL or upstream source does not trigger restart loops |
| Initial delay | How long to wait before the first check | Set it from a measured cold start on your largest catalog, and confirm the value with your own test environment rather than copying defaults |
| Period and timeout | How often to check and how long each check may take | Allow enough timeout for the health endpoint under load; short timeouts on a busy catalog can mark healthy pods as failed |
| Failure threshold | How many consecutive failures count as a failure | Use a higher threshold for liveness than readiness, so transient database latency does not cause a restart |
| Dependency checks | Whether probes call PostgreSQL, caches or external sources | Keep them in readiness only, and confirm which checks the health endpoint runs in your Backstage version |
Enabling the health check and wiring the probes
This sequence confirms the health endpoint your Backstage backend exposes, enables it in configuration, and points Kubernetes readiness and liveness probes at it. It ends with a verified pod that reaches Ready and stays up through a rollout. Each step closes with a check you can run before moving on.
Confirm the health check route for your version
The platform engineer checks the Backstage version in the repository's package manifests, then confirms in the documentation for that release which health endpoint the backend exposes and on which port. Do not assume a path from an older guide or blog post, because the endpoint may differ between releases and a wrong path is a common reason probes fail on first deploy. Record the path and port in your deployment notes and confirm the backend includes the health module that version expects. This step is done when you can name the exact path and port in writing.
Enable the health check in backend configuration
Add or confirm the health check settings in the app configuration that the backend reads, placing shared settings in the base configuration so every environment behaves the same. Start the backend locally with that configuration and send a request to the confirmed path. This step is done when the request returns a success status code and the backend starts without configuration errors.
Check the backend responds on the container port
Confirm the container image exposes the same port the backend listens on, and that the backend binds to an address reachable from outside the container. A listener bound only to localhost inside the container makes probes fail even when the application itself runs. Run the container locally with the same port mapping you plan to use in the cluster, then request the path from the host. This step is done when the request from outside the container returns the same result as the request made inside it.
Configure readiness and liveness probes in the manifest
In your Helm values or Kubernetes Deployment manifest, add readiness and liveness probes that use an HTTP GET against the confirmed path and port. Confirm the port and path in your own chart or manifest, because value names and defaults differ between chart versions and forks. Use readiness to decide when the pod receives traffic and liveness to restart a process that has stopped responding, and keep the two checks distinct where the chart allows it. This step is done when rendering the manifest with helm template or a dry run shows both probe fields with the expected values.
Allow time for startup before liveness checks
Add a startup probe or an initial delay so the first liveness check does not run while the backend is still starting. Large catalogs and first-run database migrations take longer to start, so the right allowance depends on your own catalog size and the startup time you measure in staging, not on a published default. Measure the time from container start to first successful readiness, then set the thresholds with margin above that value. This step is done when a cold start in staging reaches Ready without a restart.
Verify with a direct request and pod events
Deploy to a non-production namespace, then send a request to the health path through a port-forward to the pod and compare the response with your probe settings. Run kubectl describe on the pod and read the Events section for probe failure messages, and check the restart count with kubectl get pods. A readiness failure with no restarts usually points to traffic gating, while a liveness failure with a rising restart count usually points to the path or the threshold. This step is done when the pod shows Ready, the restart count stays stable, and the event log shows no probe failures across a full rollout.
How startup time for large catalogs affects probe timing
A startup probe is the setting that protects a slow Backstage backend from being killed before it finishes starting. On a large installation, the backend does more work before it can answer requests: it connects to PostgreSQL, runs database migrations for each plugin, initializes plugins and begins the first processing pass over catalog locations. A liveness probe that starts too early will see a process that is still working as unhealthy, restart it, and begin the cycle again. A startup probe holds liveness checks back until the first successful startup response, so the failure threshold can be set for the worst expected boot time rather than the typical one. Readiness is a separate question: it controls whether the pod receives traffic, and it should not be satisfied until the backend can serve the catalog and authentication requests that users make.
Before choosing thresholds, measure how long your own instance takes to become ready. Start from a cold pod with the same image, configuration and database you use in production, and record the time from container start until the health endpoint first returns success. Repeat the measurement with a catalog at its current size and again with a realistic growth margin, because initial processing time grows with the number of entities, locations and providers you register. Also record the time spent on database migrations on the first boot after an upgrade, since that step runs only once but can be the longest. Ask your platform team which probe settings your cluster already uses and whether the database is in the same region, since network latency to PostgreSQL shows up directly in startup time. No startup duration is given here, so treat these figures as the inputs you need to collect, not values to copy.
Avoiding restart loops: common causes and fixes
Why does my Backstage pod keep restarting after I added probes?
Most restart loops come from probes that fail for reasons unrelated to whether the process can serve requests, so Kubernetes kills a pod that is actually working. Check the pod events with kubectl describe pod first, because the event message names the failing probe and the HTTP status or timeout it saw. The causes below are the ones teams hit most often, and each has a specific fix. Change one thing at a time and watch the restart count after each change, so you can tell which change stopped the loop.
Can a liveness probe that checks the database cause restarts?
Yes. A liveness probe that calls the database or another external dependency fails whenever that dependency is slow or unreachable, and Kubernetes then restarts Backstage even though the process is healthy. The restart does not fix the database, so the loop continues and every replica can fail together. The fix is to keep the liveness probe on a check that only proves the Node.js process is responding, and to leave dependency checks to the readiness probe. Readiness removes the pod from the service until the dependency returns, without killing it. Confirm which checks the /healthcheck endpoint runs in your version before you point each probe at it.
How do I set a probe timeout that fits Backstage response times?
Set timeoutSeconds and periodSeconds from measured response times under load, not from defaults. A timeout shorter than the normal response time for /healthcheck makes a busy but healthy pod fail, and repeated failures trigger a restart. Measure latency during a catalog refresh or a large search reindex, because those periods are when the backend is busiest. Give the probe a timeout with headroom above that measured value, and set failureThreshold so one slow response does not count as a failure. Load testing your own environment is the reliable way to pick these values.
What causes database connection exhaustion in Backstage and how do I fix it?
Backstage uses PostgreSQL in production and gives each backend plugin its own logical database through Knex, so a single pod holds connections for several plugins at once. When replicas scale up or the pool is sized too large, the database runs out of connections, requests stall, and probes time out. Set the connection pool size explicitly in your configuration, multiply it by replica count and plugin count, and compare the total with your PostgreSQL max_connections limit. Where the total is too high, lower the per-pod pool, reduce replicas, or add a connection pooler in front of the database. Watch active connections during a restart cycle to confirm the cause.
Why does Backstage fail its probes right after a deploy?
A probe that starts before database migrations finish sees a pod that cannot serve requests yet, so it fails and the pod restarts mid-migration. Repeated restarts can interrupt migrations again, which leaves the schema in a partial state. Use a startup probe so Kubernetes waits for the first successful start before applying liveness checks, and set its failureThreshold times periodSeconds above the measured migration and startup time. Run migrations as a separate step, such as an init container or a pre-deploy job, so the application pod does not race them. Confirm the migration order in your deployment pipeline before relying on it.
How long should a startup probe allow for a large software catalog?
The right window depends on how long your backend takes to load the catalog, process locations and finish plugin initialization, and no supplied figure gives a universal value. Measure startup time on a production-sized copy of your catalog, then add margin for slow nodes and cold image pulls. Startup time grows with the number of entities, processors and providers, so re-measure after large imports. If the startup probe window is too short, the pod restarts before it is ready and the loop repeats, so err toward a longer window and tune it down once you have data.
Can a team run this probe work in-house or do they need a partner?
Most platform teams can fix these causes in-house if they have access to cluster events, pod logs and database metrics, and the work is mostly measurement and configuration. The effort rises when probes depend on several services, when migrations are complex, or when the catalog is large enough that startup tuning needs repeated load tests. DeployOpen implements and supports self-hosted platforms in infrastructure you control, and a fixed-scope engagement can cover probe design, startup tuning and a documented runbook. Ask any partner to state in writing which environments, load tests and deliverables are included before work starts.
Security and operational considerations for health endpoints
A health endpoint answers questions about your Backstage instance and its dependencies, so it needs the same care as any other interface. The cards below cover where the endpoint is reachable, how much it reveals, how restarts are watched, and how probe settings are kept under change control.
Keep the endpoint inside the cluster
Kubernetes probes call the health path from the kubelet on the pod network, so the endpoint rarely needs to be reachable through an ingress. Check that the Backstage backend service and any ingress rules do not route /healthcheck to the public hostname. Confirm the path is absent from your external load balancer and web application firewall rules. Test from outside the cluster with a plain request after every ingress change, because a new route can expose the path without anyone noticing.
Limit dependency detail for unauthenticated callers
A readiness response that lists database names, cache hosts, or error text tells an attacker how the portal is built. Return a minimal status to anonymous callers and reserve detailed dependency output for authenticated operators or internal logs. Review what each health check plugin includes in its response body before you enable it. Keep error messages generic in the response and send the specific failure to your logging pipeline instead.
Restrict who can read health details
Where detailed output is needed for troubleshooting, decide who may see it and how they authenticate. Operators on the platform team usually need it, while portal users and external monitors do not. Confirm that any route exposing detail sits behind the same access control as your other operator tools, and that guest or anonymous access is not enabled for it in production. Record the decision in your runbook so the exception stays visible.
Alert on repeated restarts
A liveness failure that restarts the pod once can be a transient event, but a climbing restart count points to a loop. Alert on restart count increases over a window, not on a single restart, and link the alert to the pod's previous container logs. Check whether the restarts follow a deployment, a database failover, or a large catalog refresh. A readiness flap that never restarts the pod still deserves an alert, because the portal is unavailable to users during it.
Keep probe settings in version control
Probe paths, ports, initial delays, periods and failure thresholds should live in the same Helm chart or manifest as the Deployment, reviewed through the same merge process. Changes made with kubectl edit drift from the repository and disappear at the next sync, which hides the reason a restart loop started. Compare the live spec against the repository in your GitOps tool after each release. Record why each threshold was chosen so the next reviewer does not copy a default.
Review probes with each upgrade
Backstage upgrades can change startup time, the health route, or the set of dependencies a check touches. Read the upgrade notes for each release line before you move the image tag, and test the new version against a copy of your catalog. Measure startup time on the new version and adjust the startup probe before production rollout. Confirm that the probe still reports failure when PostgreSQL is unreachable, since a check that always passes gives no protection.
What does it cost to get Backstage probes right?
What drives the effort of getting Backstage health checks and probes right?
Effort depends mainly on catalog size, the number of backend instances, and how many environments need separate tuning. A small catalog on one backend instance is a short tuning exercise. A large catalog with several replicas, multiple clusters and separate staging and production settings means more load testing, more threshold decisions and more verification per environment. Ask the provider to state which of these factors their estimate assumes, and to list each environment that will be tuned separately.
Can we get Backstage probe settings right without outside help?
Yes, when your platform team already runs Kubernetes workloads in production and can read pod events, probe logs and resource metrics. The work is mostly configuration, measurement and repeated testing, not new code. Teams without that experience usually take longer to reach stable settings and are more likely to change thresholds under pressure. A partner is most useful when restarts keep recurring after the first round of tuning, because the cause often sits across catalog processing, database connections and resource limits at once.
What access and inputs does a Backstage probe tuning project need?
The work needs read and write access to the target clusters' Deployment manifests, read access to pod events and logs, and a way to run load against the backend. A staging environment with a catalog close in size to production is the best test bed. Without it, you will need a load test that simulates catalog refresh and scaffolder traffic. Confirm early who can change manifests in each environment and who approves a change, because probe values are often edited in more than one repository.
Who owns the Kubernetes manifests that hold Backstage probe settings?
Your organization should name one owner for the Deployment manifests, usually the team that runs the Backstage deployment. That owner decides the probe paths, timeouts, failure thresholds and startup allowances, and reviews each change. Without a named owner, probe values drift between environments and a fix in one cluster does not reach the others. A written scope should state who reviews probe changes, where the manifests live and how changes are rolled out.
Which parts of Backstage probe work are one-time and which recur?
The initial tuning is one-time work: measuring startup time, choosing thresholds, and checking behavior under load in each environment. Recurring work includes re-checking probes after Backstage version bumps, adding plugins that change startup behavior, and growing the catalog. Ask the provider whether a written scope covers only the initial tuning or also a review at each upgrade. Upgrades and continuous support are available as retainers, so recurring work can be scoped separately.
What should a written scope for Backstage probe tuning include?
A useful scope lists the environments in scope, the Backstage version and deployment method, the probe endpoints being configured, the load test or staging catalog to be used, and the acceptance checks. It should name the deliverables, such as updated manifests, a short runbook and a restart-behavior report. It should also state the owner on each side and the schedule. DeployOpen agrees scope, deliverables and timelines in writing before work starts, and most engagements are fixed-scope.
When does a partner make sense for Backstage restart loops?
A partner is worth considering when restarts recur after you have adjusted probe thresholds, because that usually points to a cause outside the probe itself. Common candidates include slow catalog processing at startup, database connection limits, memory pressure, or a plugin that blocks the event loop. Before engaging, collect pod events, restart counts, probe logs and the catalog size at each restart. Those records let an outside team start where your team stopped.
Does the Backstage version affect how much probe tuning will cost?
It can. Backstage ships a main release line monthly and a next release line weekly, and the release notes and Upgrade Helper show template changes between versions. Startup behavior and plugin requirements can shift between versions, so probe settings may need a check after a bump. Confirm which Backstage version and Node.js version your deployment runs before requesting a quote, because the supported Node.js releases are set by the versioning policy and the scope should match them.
How DeployOpen scopes Backstage health check work
DeployOpen scopes this work as a written statement agreed before any change is made to your environment. The scope names the Backstage version in use and the release line you plan to follow, since the project ships a monthly main release line and a weekly next line, and it records how the backend is deployed: which clusters, namespaces and replicas run the backend, and which PostgreSQL instances hold the per-plugin databases. Those layout facts decide which probe settings are safe, so they are gathered first. Most engagements are fixed-scope, so the deliverables, the environments covered and the acceptance checks are written down and priced from that scope rather than estimated open-ended.
Probe configuration is delivered as code in your repository and deployment pipeline, not as manual edits in a running cluster. DeployOpen reviews the backend health endpoints, the /healthcheck route and any plugin checks that depend on external systems, and sets separate readiness and liveness probe definitions, with a startup probe where the catalog is large enough to need one. The scope also covers load and startup testing in your own environment, using a catalog of realistic size, so that startup time and restart behavior are measured on your hardware and database rather than assumed. Confirm with your platform team which cluster, staging environment and test data you will provide before the work starts.
Ongoing support is optional and is offered as a retainer. A retainer can cover re-running startup tests after a Backstage upgrade, reviewing probe settings when catalog size or plugin count changes, and responding to restart incidents. Ask for the retainer terms in writing, including what is excluded, how incidents are reported and who on your side approves production changes. DeployOpen prepares the systems and evidence; it does not issue audit reports or certifications, so any compliance evidence from this work should be reviewed by your own assessor.
