Incremental ingestion for large Backstage catalogs

When full catalog refreshes stop working for incremental ingestion

Incremental ingestion lets Backstage read only the records that changed since the last run, instead of reading an entire source on every cycle. For a large identity tenant or inventory, a full refresh can run long enough to overlap the next scheduled pass, and it sends the same requests to the source again and again. That load can trigger throttling on the source side, which slows the refresh further. The problem is usually visible before anyone designs a fix: the catalog lags behind the source, owners report entities that should have appeared, or the source team asks why Backstage makes so many API calls. Start by measuring how long one complete pass takes and how many requests it makes against the source's limits, ideally against a non-production copy.

Teams need incremental ingestion when a full pass cannot finish inside the freshness the business requires and the source cannot absorb repeated full reads. Typical signs include passes that never complete before the next is scheduled, rate-limit responses in source logs, and entities whose owners depend on changes landing within a working day. A simpler entity provider is usually enough when the source is small, the full pass finishes comfortably within your window, and the source's limits are not a constraint. These thresholds are your decision to confirm from your own measurements, not a fixed rule. A full-refresh provider is easier to reason about and debug, so adding change tracking should be justified by evidence, and the size of the tenant is only one input.

Whatever you choose, treat incremental ingestion as an operational commitment. It keeps state about what was seen last, so an interrupted or failed cycle needs a clear recovery path. Burst and rest cycles also change when load lands on the source, so the source owners should approve the schedule. Record the staleness your catalog owners accept, the source's rate-limit policy, and who owns the module through Backstage upgrades. Before committing, confirm that the module supports your Backstage version using the keeping Backstage updated guide, because all @backstage packages move together during an upgrade.

Ingestion options and when each one fits

This table compares full refresh, the Backstage incremental ingestion module, and Microsoft Graph incremental ingestion across source size, freshness, source API load, deletion behavior and operational effort. Read each row against your own source size and change rate, then use the last row to decide which option to pilot first.

Factor	Full refresh	Incremental ingestion module	Microsoft Graph incremental ingestion
Best source size	Small to moderate sources that finish a full pass within your refresh window	Large sources where a full pass takes too long or hits API limits	Large Microsoft Entra ID tenants with many users and groups
Freshness	Entities reflect the source only as often as the full pass completes	New and changed records arrive in bursts between rest periods	Directory changes arrive in bursts, so freshness depends on cycle timing
Source API load	Reads the whole source on every cycle, which is the heaviest load	Reads changes in short bursts, then rests to spread requests over time	Reads directory deltas, which keeps requests low once the first pass completes
Deletion behavior	Records missing from a complete pass are removed, so a partial pass is risky	Removals depend on how the module tracks the cursor and marks entities as gone	Removals depend on the delta responses, so confirm how deleted objects are handled
Operational effort	Lowest setup effort, but scheduling and timeouts need tuning as the source grows	Moderate effort to set burst and rest intervals and to watch progress	Highest effort: tenant permissions, delta tokens, cycle tuning and monitoring
Confirm before choosing	Full pass duration in your environment	Module version and supported source type in your Backstage release	Tenant size, app registration permissions and the current Microsoft Graph documentation

How to set up incremental ingestion step by step

This sequence takes a large source from an unplanned full refresh to a scheduled incremental ingestion that keeps the Backstage catalog current without repeated full crawls. Each step closes with a check you can run before moving on.

Confirm the source and its API limits
Start by listing the source you want to ingest, the entity types it holds, and the rate limits or paging rules its API enforces for your account or tenant. Ask the source owner or platform administrator for the quota that applies to the service identity Backstage will use, and confirm whether the API offers delta or change-tracking endpoints. The step is done when you have a written note of the object counts, the expected change rate, and the limits, so the burst and rest settings in a later step have something to be based on. Check: the note names the endpoint, the quota owner, and whether a delta endpoint exists.
Install the incremental ingestion module
In the Backstage repository, the backend team adds the incremental ingestion backend module to the backend package using the Backstage CLI, so the package version matches the Backstage release the portal runs. Confirm the module's version against your current @backstage package versions before installing, and avoid mixing versions by hand. The step is done when the backend builds and starts with the module registered and no dependency warnings about mismatched @backstage packages. Check: the backend starts locally and the module appears in the startup log.
Configure the source provider and burst and rest settings
Add the provider configuration to the backend config file, using the source's own credentials stored as secrets rather than plain text. Set the burst length and the rest interval so each cycle stays within the limits you recorded in step one, and set a conservative value first, since you can tune it after observing real traffic. Keep the Microsoft Graph provider settings separate if your source is a large Microsoft tenant, because its paging and throttling behavior differs. The step is done when the config validates and the provider starts without credential errors. Check: a dry run reports the provider loaded and no authentication failures.
Register the entity mapping
Define how each record from the source becomes a Backstage entity, mapping source fields to metadata.name, spec.owner, spec.lifecycle and the entity kind. Agree the mapping with the team that owns the source so names stay unique and owner references resolve to existing Group or User entities. Store the mapping alongside the provider configuration so it is reviewed with code. The step is done when a sample of records maps to valid entities with no naming collisions. Check: the validation of a sample batch reports zero errors for the required fields.
Run an initial ingestion
Trigger the first ingestion cycle while the portal is in a quiet period, and let it run through its burst and rest cycles until the source has been read at least once. Expect the first pass to take longer than later incremental passes, because it establishes the baseline that later cycles compare against. Record the start time, the number of entities created, and any records skipped. The step is done when the cycle completes without a stopped or failed state. Check: the catalog entity count is close to the object count you recorded in step one, allowing for records you chose to exclude.
Verify ingestion progress and freshness
Confirm that the incremental cycles are running on schedule and that changes at the source appear in the catalog within the cycle you expect. Make a test change at the source, such as updating a non-critical field on one record, and watch it reach the catalog on the next cycle. Review the ingestion logs and any metrics your deployment exposes for cycle duration, errors and throttling responses. The step is done when the test change is visible and the logs show consecutive cycles completing without rate-limit errors. Check: the test record reflects the change and no throttling errors appear across at least two cycles.
Tune burst and rest against observed limits
After several cycles, compare the observed request volume and throttling responses with the limits from step one, then adjust the burst and rest settings in small increments. Record each change and its effect so the team can explain the final values during an upgrade or an incident. Confirm with the source owner that the resulting load is acceptable for the service identity. The step is done when cycles finish inside the quota with margin and the settings are documented in version control. Check: the configuration change is committed and two consecutive cycles complete without throttling.

Securing credentials and access for large source ingestion

Large ingestion runs read far more of a source than a normal catalog sync, so the identity, secrets, network path and logs behind it deserve the same care as the portal itself. The cards below cover the decisions to make before the first burst cycle runs. Check each one against your own tenant or source, since the exact permissions and settings depend on the provider you connect.

Scope the source identity to reading

Create a dedicated app registration or service identity for ingestion instead of reusing a person's account or a broad administrative principal. Grant only the read permissions the provider needs to list the object types you ingest, and confirm with the provider's documentation which permissions are application scoped and which need admin consent. Remove any write or directory management rights that were added for testing before go-live.

Store client credentials outside the config file

Keep the client secret or certificate for the ingestion identity in a secrets manager your platform team already operates, and inject it into the Backstage backend at runtime rather than committing it to app-config.yaml. Set an expiry date for every secret and record who rotates it. A rotation that breaks ingestion silently leaves the catalog stale, so alert on failed provider calls.

Limit the network path to the source

Ingestion calls the provider API from the Backstage backend, so decide which egress route it uses and allow only that route through your firewall or proxy. Confirm whether the provider endpoints must be reached through a corporate proxy, and whether any tenant or region restrictions apply. Keep the backend off the public internet where the source is private.

Keep guest access off in production

Backstage's authentication guidance advises against offering guest access in production. Ingested users and groups become visible through the portal, so anyone who can reach it without a real sign-in can read the catalog data that ingestion brought in. Configure sign-in resolvers so every identity provider maps to the same user entity, which keeps ownership and access decisions consistent.

Record what ingestion reads and writes

Confirm which audit records your provider and Backstage deployment produce for ingestion calls, including the identity used, the object types read and the time of each run. Decide where those logs are kept and how long, and who may review them. Ask the provider whether sign-in and API read events are logged at a level your security team can query during an incident.

Monitoring ingestion progress and catching stalls

Platform operators own the health of the ingestion pipeline itself. An incremental run is made of bursts that pull a slice of changes and rest periods between them, so a healthy system shows a steady rhythm of completed bursts. The failure modes that matter are a burst that starts and never finishes, repeated throttling responses from the source API, and a growing gap between the newest change at the source and the newest change the catalog has processed. Confirm with your Backstage version which metrics, logs and cursor or checkpoint values the ingestion module emits, because metric names and log fields differ between releases.

Stalled bursts: Alert when a burst exceeds the duration you measured during a healthy baseline run, and check whether the cursor or checkpoint advanced.
Throttling responses: Count HTTP 429 and similar throttling responses per burst, and watch whether retry backoff grows over time.
Growing lag: Track the time between the latest source change and the latest processed entity, and alert when it trends upward across several cycles.
Error counts per provider: Separate errors by provider instance so one tenant or source does not hide problems in another.
Log retention: Keep ingestion logs long enough to compare a failing burst with the last healthy one.

Before relying on any alert, run a controlled sync in a non-production environment and confirm that each signal changes as expected.

Limits to confirm before you rely on incremental ingestion

Incremental ingestion changes how fast the catalog reflects a large source, but it also changes what happens when records disappear, when throttling kicks in and when Backstage moves to a new release line. The cards below list the checks to run in your own tenant and Backstage version before you commit to a rollout plan.

Module support for your release line

Confirm that the incremental ingestion module and the Microsoft Graph provider you plan to use support the Backstage release line you run or intend to run. Support can differ between the main release line, which ships monthly, and the next line, which ships weekly. Ask the vendor or check the module's package metadata for the exact @backstage peer versions, and record the result in your upgrade plan.

Source API throttling rules

Read the throttling and quota rules for the source API your tenant uses, including per-app and per-tenant limits and how the service signals them. Burst and rest cycles only protect you if the burst size and rest interval fit those limits under peak load. Test with a copy of your real tenant data volume before setting the schedule, because throttling often appears only at full scale.

Behavior on tenant-wide changes

Find out how the ingestion handles bulk events such as a directory restructure, a mass rename or a change to group membership rules. A single tenant-wide change can produce a large delta, which may stretch a burst past its planned window or trigger a full re-sync. Confirm with a test tenant how the module responds, and decide whether such events should wait for a scheduled full refresh.

Deletion and orphan effects

Confirm how removed users and groups are handled in the catalog. Incremental runs may mark records as removed differently from full refreshes, and the orphan strategy determines whether entities are deleted, kept or left stale. Decide which behavior you want for owners who have left, and test it before enabling deletions on the production catalog.

Upgrade impact on ingestion state

Check whether an upgrade of Backstage or the module changes the stored cursor or delta state. An incompatible change can force a full re-ingestion, which may take far longer than a normal incremental run on a large directory. Read the upgrade notes for the module, use the Backstage Upgrade Helper to review template changes, and plan the upgrade in a non-production environment first.

Questions teams ask before ingesting a large catalog

How much does incremental ingestion cost for a large Backstage catalog?

The cost follows from the written scope, and DeployOpen does not publish a price list. Backstage is Apache-2.0 licensed and the project offers no paid edition, so the spend goes to the work and to the platforms you run. The build is one-time: discovery of each source, entity mapping, module configuration, pagination and rate-limit handling, a pilot against your tenant, monitoring and handover. Recurring costs are your operations time, upgrades and any retainer for ongoing support. Cost rises with the number of sources, the object types ingested (users, groups, memberships), the number of mapping rules and the number of environments to test. Ask any quote to list each source, the object counts you expect, who will operate the result and how upgrades are handled.

How long does the first full ingestion take?

No duration can be stated from the researched facts, so the practical answer is to measure it in a pilot. The first full ingestion is usually the slowest run because it has to load every object in the source. The source's throttling, page size, burst length and rest periods set the pace. Later cycles should be shorter because they process only what has changed, but confirm that behavior for your source and module version. Before asking for a schedule, gather the object counts per type, the API limits your tenant applies and the number of groups with deep nesting. A fixed-scope proposal should state milestones and pilot exit criteria in writing before work starts.

What does our team need to provide for the engagement?

Your team supplies access, decisions and people. Access means an app registration or service identity with the read permissions the source requires, a test tenant or sandbox, and access to the running Backstage instance, its configuration and its monitoring. Decisions cover which objects enter the catalog, how directory groups map to Group and User entities, which record wins when sources conflict, and what happens to owned entities when a person leaves. People include a directory administrator who can confirm source data, an owner of the Backstage deployment who approves changes, and a security contact for credential handling. Name these people early, because delays usually come from waiting for approvals and decisions rather than from the build.

Can our own engineers run incremental ingestion without a partner?

Yes, if your team already runs Backstage and has the skills the work needs. You need TypeScript and Node.js experience for backend module work, working knowledge of the catalog model and processing, directory administration skills for the source, and someone who can read rate-limit behavior and interpret logs during a long run. The main risks are configuration that removes entities after an incomplete read, and a missing monitoring signal that goes unnoticed for weeks. A common approach is for a partner to design the mapping and test plan, then for your engineers to operate it from a runbook. Whoever builds it should hand over the code, documentation and test cases in your own repository.

How do Backstage upgrades affect the ingestion module?

The module moves with the rest of Backstage, so plan for it as part of routine operations. Backstage uses yarn backstage-cli versions:bump to move all @backstage packages together, and the Backstage Upgrade Helper shows template changes between versions. The project ships a main release line monthly and a next line weekly. Its versioning policy says high and critical vulnerabilities are backported to releases from the last 6 months where feasible. Backstage supports Node.js 22 and 24 as of 1.46.0. Any custom ingestion code, mapping or patch needs a test run in staging after each bump, because the upgrade tooling cannot verify that logic for you. Read the keeping Backstage updated guide before scheduling your first bump.

How do processing interval and webhooks relate to incremental ingestion?

They solve different problems, so they work together. Incremental ingestion controls how a large source is read: in bursts, with rest periods between them, so a big read can finish across several cycles without flooding the source. The processingInterval setting is a minimum time between processing runs for an entity, so it controls freshness and load on the catalog and on Git provider APIs. Webhooks push a change to Backstage when something changes, which reduces polling on source control platforms. For a directory source, webhooks help only if the source emits events you can use, so confirm that during design. Tune burst length, rest period and processingInterval together against the rate limits you measure, and watch the logs for throttling.

Can a failed or partial read delete entities from the catalog?

It can, which is why this is the first risk to test. A source that returns a partial or failed read can look like a set of removals to the catalog, so entities may disappear until a complete run succeeds again. Ask how the module handles failed pages, what it does with entities missing from a run, and how orphan and delete behavior applies before the first production run. Guard against this with object count checks between runs, alerts on failed bursts, a staging environment that mirrors production settings and a documented rollback. Decide in advance whether owned entities of departed people stay visible while ownership is reassigned. Include a deliberately failing source in the pilot so the team sees the behavior before it matters.

How DeployOpen scopes and supports incremental ingestion

DeployOpen plans, implements, tests, documents and supports self-hosted Backstage inside your own infrastructure, including the catalog ingestion design for large sources. Engagements are fixed-scope, with the scope, deliverables and timelines agreed in writing before work starts.

How we scope

Inventory the sources that feed the catalog, their size, change rate and tenant or API limits.
Agree in writing which ingestion modules, providers and schedules are in scope for each source.
Define how removed records, orphaned entities and failed cycles will be handled before any build starts.

How we build

Configure incremental ingestion for the chosen source, with burst and rest cycle settings sized to your limits.
Test full refresh, delta cycles and deletion behavior in a non-production Backstage instance first.
Set up monitoring for ingestion progress, error rates and cycle duration, and confirm alerts reach your team.

How we hand over

Document the configuration, source permissions, cycle settings and the reasons behind each decision.
Provide runbooks for reviewing ingestion errors, resetting a source and handling a rate limit event.
Walk your platform and identity teams through the setup so they can operate and change it themselves.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Review ingestion metrics and catalog growth with your team as the source and organization change.
Re-check cycle settings and source permissions when Backstage packages or the source platform change.

Incremental entity ingestion for large Backstage catalogs