Why teams read catalog descriptors from S3 instead of repositories
Teams often generate catalog-info.yaml descriptors from a deployment pipeline or an inventory tool, and they do not want those generated files committed to Git. Backstage can read these descriptors from Amazon S3 through its AWS S3 integration and the catalog-backend-module-aws module, using buckets that the platform team controls. The method is documented in the Backstage integrations documentation, which lists AWS S3 as a supported source, so the reader should follow that documentation as the reference for configuration rather than adapting examples from community posts.
The S3 path complements repository discovery and does not replace it. Hand-written descriptors that describe services, components and APIs usually belong beside the code they describe, where owners review changes in their normal workflow. Descriptors that an inventory or pipeline produces, such as entries for legacy systems, infrastructure or resources that have no repository, are a better fit for a bucket. Before choosing, confirm with your platform team which entity types each source will own, because two sources that describe the same entity can produce conflicting catalog records.
Decide early who can write to each bucket and prefix, since anyone with write access can change what appears in the catalog. Scope the read permissions of the Backstage backend to the specific buckets it needs, and confirm the credential approach against the current AWS integration settings in the Backstage documentation. Ask the pipeline owners how they will remove descriptors for retired systems, because orphaned S3 objects keep producing catalog entities until someone deletes them or the source is changed.
What you need before configuring S3 discovery
Discovery from S3 depends on four things lining up: a Backstage release and Node.js line you can run, a bucket layout that separates descriptors from other data, read access the backend can actually use, and a network path from the backend to S3. Work through these cards before you edit any configuration, because most failed first runs trace back to one of them.
Backstage and Node.js versions
Start from the Backstage release line your portal already runs, then check the Node.js versions that line supports in the versioning policy. As of Backstage 1.46.0, the supported pair was Node.js 22 and 24, but confirm the pair for your release before you install dependencies. The AWS integration ships inside the core repository, so you do not add a separate package from outside the project, but it must match the @backstage packages you already pin.
AWS account and bucket layout
Decide which account owns the bucket and whether descriptors live in a dedicated bucket or a prefix inside a shared one. A dedicated prefix such as one reserved for catalog-info files keeps discovery from scanning inventory exports, logs or backups. Record the bucket name, the prefix pattern and who owns the generator that writes the files, since that owner decides when and how descriptors change.
Read-only IAM scoped to the prefix
The backend needs read access only to the descriptor prefix, not to the whole account or bucket. Draft a policy that allows listing the bucket with a prefix condition and reading objects under that prefix, then confirm the exact IAM actions and condition keys against the AWS integration documentation for your release. Avoid write permissions, because discovery never needs them, and document who approves changes to this policy.
Region and credential source
Fix the AWS region of the bucket and the credential source the backend will use, such as an instance role, a pod identity, or a static access key stored in your secrets system. Prefer short-lived credentials from the runtime platform over long-lived keys in configuration files. Confirm with the integrations documentation how the backend resolves credentials in your deployment model, and record the rotation owner.
Network path to S3 endpoints
Confirm that the backend host or pod can reach the regional S3 endpoint over the route you intend to use. In private networks, check whether a gateway or interface VPC endpoint for S3 exists, whether DNS resolves the endpoint to the private address, and whether an outbound proxy must be configured for the backend process. Test the path from the same subnet and runtime that will run Backstage, not from a laptop.
Descriptor and test data ownership
Name the team that writes descriptors into S3 and the team that reviews changes to them. Because S3 objects lack the pull request review that repository-based descriptors get, agree on how invalid files are caught before they appear in the catalog. Prepare a small test prefix with a few valid catalog-info files so you can verify discovery in a non-production portal first.
How to configure the S3 catalog provider step by step
This sequence takes a Backstage backend from no S3 discovery to a verified catalog load, with each step confirmed before the scope widens. Teams finish with a tested bucket and prefix, a refresh schedule, and a visible record of how bad descriptors are handled.
Confirm the bucket layout and descriptor contract
Before touching the Backstage code, the platform team and the pipeline or inventory owners agree on where catalog-info files will land in S3, how keys are named, and which entity kinds are allowed. Write this down as a short contract that the producing pipeline must follow. Done means a single test bucket exists, at least one sample descriptor is uploaded to a planned prefix, and both teams have signed off on the path and the file format. Ask the producer to confirm that each descriptor carries a valid kind, metadata name and owner, because those fields decide whether the catalog accepts it later.
Install the AWS catalog module in the backend
A backend engineer adds the AWS catalog module to the Backstage backend package, following the module's installation steps in the core repository's plugins directory, then registers it in the backend entry point. The module ships with the Backstage core repository, so no separate vendor product is needed. Done means the backend builds, starts without errors, and the log shows the catalog backend initializing with the AWS module loaded. If the build fails, check that the module version matches the other @backstage packages, since mixed versions are a common cause.
Define AWS integration credentials
The engineer adds the AWS integration block to the app configuration and points it at a credential source that the backend can use in your environment, such as an instance role, an assumed role, or a profile on the host. Keep access keys out of the committed configuration and load them from your secret store. The role should allow listing the test bucket and reading objects under the planned prefix only. Done means the backend can authenticate, and a manual list or read against the test prefix from the backend host succeeds while a read outside the prefix is denied.
Set the bucket and key prefix for discovery
The engineer adds a catalog location entry that names the single test bucket and the key prefix that holds the descriptors, and sets the file pattern to match your catalog-info naming. Start with one prefix so the scope is narrow and easy to audit. Done means the backend startup log reports the location being read, and the number of entities discovered matches the number of descriptor files uploaded in step one. A mismatch usually means a pattern or prefix typo, so compare the configured values against the actual object keys.
Choose a refresh schedule
The engineer sets the refresh interval for the S3 provider so the catalog re-reads the bucket on a fixed cadence. Choose an interval that matches how often the producing pipeline publishes, and make sure it is longer than the time a typical run takes to finish writing. Ask the producing team to confirm when their uploads complete, since a read during a partial upload can see an incomplete set of files. Done means the schedule appears in the backend log on each cycle, and an upload made between cycles shows up in the catalog after the next run.
Load the test bucket and check the catalog
The engineer restarts the backend and opens the catalog in the portal. Confirm that each uploaded descriptor appears as an entity with the expected kind, name, owner and relations. Then upload one deliberately malformed descriptor, such as one with a missing name or an invalid kind, and wait for the next refresh. Done means the valid entities appear and the malformed file produces a processing error visible in the catalog's error view or the backend logs, rather than being skipped silently. Record both results as the acceptance evidence for this stage.
Widen scope one bucket or prefix at a time
Only after the test load passes does the team add further prefixes or buckets, one at a time, repeating the same entity count and error checks after each addition. Keep the IAM role limited to the prefixes listed in configuration so each new source is an explicit decision. Done means every configured location has a recorded entity count, error checks pass for each, and the list of approved sources is stored with the configuration in version control so reviewers can see what the catalog reads.
How S3 entities map to Backstage kinds, owners and names
The descriptor format for files read from S3 is the same as for files kept in a repository. Each object must be a valid catalog-info.yaml style document with apiVersion, kind, and metadata fields, and the kind value (for example Component, API, or Resource) decides which catalog model rules apply. Set metadata.name to a stable identifier, leave metadata.namespace at default unless you have a reason to split namespaces, and make spec.owner reference a Group or User entity that already exists or will be ingested from your identity source. Set spec.system only to a System entity you intend to maintain. Entities that point at missing owners or systems will still load, but they become hard to govern, so check references in a staging catalog before you turn on the provider in production.
The provider builds each entity's source location from the bucket name and the object key, so a key such as teams/payments/catalog-info.yaml in a bucket named platform-inventory becomes a location that the catalog tracks by that path. Treat the key layout as part of the data model. If a pipeline regenerates files under new keys, Backstage sees a different location, and the entity can appear as a duplicate or be orphaned. Derive metadata.name from a value that does not change across runs, such as a service ID from your inventory, not from a build number, timestamp or object version. Then regeneration updates the same entity in place. Ask your team to document the naming rule and to test one regeneration cycle in a non-production bucket before rollout.
Confirm with the release you deploy how deleted or replaced objects are handled. The behavior of removing an entity when its S3 object disappears, and whether that removal happens on the next scheduled read or only after a manual step, depends on the provider configuration and the catalog version in use, so check it against the Backstage integrations documentation and your own test results. Also confirm how long an orphaned entity remains visible and who is alerted, because a generator that fails silently can make a live service disappear from the portal or leave a retired one listed as active.
Mapping S3 objects, catalog fields and owning teams
This table compares each part of a catalog-info descriptor with the S3 object or pipeline value that should populate it. Read the Source column as what your generator must supply, and the Team action column as what you must add because the descriptor does not derive it from S3 automatically. The main takeaway is that the object key gives location context only, so identity, ownership and lineage must be written into the file by your generation step.
| Descriptor field | Source in S3 or the pipeline | Team action during generation |
|---|---|---|
| Location (bucket and key) | Bucket name and object key of the catalog-info file | Keep keys stable so the entity location does not change between runs |
| kind and metadata.name | Inventory record for the service, API or resource | Generate deterministic names so repeated runs update the same entity |
| metadata.namespace | Business unit or account identifier from the inventory | Add explicitly; it is not inferred from the bucket |
| spec.owner | Owning team identifier from the inventory system | Must match a Group or User entity from your org data source; confirm the reference format |
| spec.system | Product or system grouping from the inventory | Add explicitly so the system model stays consistent |
| spec.lifecycle and spec.type | Stage and component type from the pipeline or inventory | Map to values your catalog conventions allow; validate before publishing |
| Annotations (pipeline run ID, generator name, source record ID) | Build identifier, generator version and inventory record key | Add explicitly; these trace each descriptor back to the job that produced it |
| Dependencies (dependsOn, providesApis) | Inventory relationships, if your inventory records them | Add only when the source data is reliable; stale links are hard to spot later |
Securing bucket access and generated descriptors
An S3 bucket that feeds the Backstage catalog sits between your pipelines and your portal, so it needs the same access discipline as any production data store. The cards below cover how the backend reads from the bucket, how credentials are issued, how the bucket is protected at rest, and how generated descriptors are checked before they become catalog entities.
Scope the backend to a read-only prefix
Give the Backstage backend a role that can only read objects under the catalog prefix, such as the catalog/ path you configure for discovery. Grant s3:GetObject on that prefix and the list permission needed for discovery, and nothing that writes or deletes. Check the policy by attempting a read outside the prefix and a write inside it; both should fail.
Separate pipeline write and list roles
Pipelines that publish descriptors need write access, and inventory jobs may need broad listing. Keep those permissions in separate roles that are not attached to the Backstage backend. If the backend role can also write, a compromised portal can alter the catalog it trusts, so confirm the two roles share no policy statements.
Prefer workload identity over access keys
Run the backend on infrastructure that assumes an IAM role, such as a Kubernetes service account bound to a role or an instance profile, rather than storing static access keys in configuration. Confirm with AWS documentation and your platform team which credential method your deployment supports, and document how credentials rotate if a static key is unavoidable.
Block public access on the bucket
Enable the account and bucket level block public access settings and review the bucket policy for any statement that grants access to anonymous principals or broad account lists. Descriptors often name internal hosts, owners and repositories, so a public read exposes your service map even when no write is possible.
Encrypt and version the bucket
Enable default server-side encryption with a key your security team controls, and turn on versioning so an overwritten or deleted descriptor can be recovered. Confirm who can use the key, since decrypt permission is a second read path to the catalog. Test a restore from a prior version before relying on it.
Validate generated descriptors before ingestion
Treat every descriptor generated by a pipeline as untrusted input. Validate it against the catalog schema in CI, reject unknown kinds and owners that do not exist in your identity source, and limit the fields a generator may set. A descriptor that passes the bucket policy can still declare a misleading owner or a system it does not belong to.
Common S3 discovery failures and how to fix them
This table lists the failures teams most often hit when Backstage reads catalog-info files from S3, with the likely cause and the first check for each. Read it top to bottom when entities are missing, wrong or lingering, since the first check usually rules out the most common cause before you change any policy or descriptor.
| Symptom | Likely cause | First check |
|---|---|---|
| Access denied errors in backend logs | The IAM identity running the backend lacks list or read permissions on the bucket or prefix, or the policy scopes the wrong ARN. | Confirm the identity the backend uses, then simulate ListBucket and GetObject against the exact bucket and prefix in the catalog configuration. |
| Discovery runs but reports no objects | The configured region or endpoint does not match the bucket's location, or a custom S3-compatible endpoint is set when the bucket is on AWS. | Compare the bucket's actual region with the region in the integration configuration and remove any custom endpoint not in use. |
| Zero entities despite files in the bucket | The prefix in the configuration does not match the object key path, including a leading slash, a missing trailing slash or a different folder name. | List keys with the same prefix from the AWS CLI and copy the exact path into the configuration. |
| Entities missing, errors in the processing log | The descriptor fails validation because required fields such as apiVersion, kind or metadata.name are absent or the kind is unsupported. | Validate the file against the catalog descriptor format and check the backend log for the object key named in the error. |
| Entities overwrite each other or flip between buckets | The same metadata.name and namespace appear in more than one bucket or prefix, so the catalog treats them as one entity. | Search the buckets for duplicate kind, namespace and name combinations and give each source a unique namespace or name. |
| Entities remain after files are deleted | The catalog keeps entities until the provider no longer returns them, and the deletion did not reach the object list the provider reads, such as a versioned bucket or a second copy. | Confirm the object is gone from the listed prefix, including previous versions if versioning is on, then check the refresh cycle in the backend logs. |
| Entities appear late or only after restart | The schedule for the S3 provider is longer than expected or the module was not registered in the backend, so no refresh runs. | Check the provider's schedule settings and confirm the catalog module is loaded in the backend startup log. |
| Only some files are read in a large bucket | Listing is paginated or the prefix is narrower than the folders that hold descriptors, so objects sit outside the scanned path. | Count objects under the prefix with the AWS CLI and compare with the number of entities the catalog reports for that source. |
Questions buyers ask about S3 catalog discovery
Does S3 discovery replace repository discovery for Backstage catalog files?
No, not in most organizations. S3 discovery is a second source for teams whose catalog descriptors come from pipelines or inventories instead of repositories. Repository discovery still fits services whose owners keep a catalog-info.yaml file next to their code, and many portals run both. The decision depends on where each descriptor is produced and who is accountable for keeping it correct. Before you choose, list every generator that writes descriptors and note whether a repository already holds the same entity, since two sources describing one component create duplicate entities and conflicting ownership.
What drives the effort of an S3 catalog discovery build?
Four factors drive most of the effort. The first is the number of descriptor generators, since each pipeline or inventory job has its own output format and failure modes. The second is descriptor quality: missing owner, system or lifecycle fields force manual cleanup later. The third is ownership data, meaning whether the owners named in descriptors match groups or users that already exist in your identity directory. The fourth is IAM and network review, covering the roles that read each bucket, bucket policies, encryption keys and whether Backstage can reach S3 from its network. Each factor grows with the number of buckets and teams involved.
How long does a pilot for S3 catalog discovery take?
The pilot length depends on your own environment, and no reliable figure applies across organizations. Confirm it against the specifics that matter most: how many buckets and generators are in scope, whether AWS account access and IAM roles already exist, whether the network path from the Backstage backend to S3 is open, and how many owners must review sample entities. A pilot limited to one bucket and one generator usually finishes sooner than one covering several accounts. Ask any partner to state the pilot duration in writing, with the assumptions behind it, before work starts.
Can an in-house platform team set up S3 discovery without outside help?
Often yes, if the team already runs Backstage in production and has someone comfortable with AWS IAM and Node.js configuration. The configuration itself is modest: the AWS integration, the catalog AWS module and the bucket and prefix settings. The harder work is operational: agreeing the descriptor schema with generator owners, writing validation so bad files do not silently fail, reviewing IAM and bucket policies with the security team, and deciding how orphaned entities are removed when a generator stops publishing. Teams without that cross-functional capacity often choose a fixed-scope engagement for the design and review work and keep ongoing ownership in-house.
What does the work need from our team?
Expect to provide three things. First, the bucket owners must confirm which buckets and prefixes hold descriptors and who may read them. Second, someone with administrative access to the AWS accounts must grant a read-only role or credentials scoped to those prefixes, and review the bucket policy and any KMS key use. Third, the platform owners must sign off on the descriptor schema, including required fields, allowed owner references and lifecycle values. Generator owners also need time to fix output that fails validation. Ask any partner to list these inputs, with named roles, in the written scope.
What changes when the descriptor generator changes?
A generator change can alter entity names, owners, annotations or the number of files written, and Backstage will reflect those changes on its next catalog refresh. Renamed or moved files can leave old entities behind, so the orphan handling policy matters. Schema changes need a versioned validation step so that a new required field does not remove every entity at once. Treat the generator as a versioned producer: test a change against a copy of the bucket contents, review the diff of resulting entities, and agree in advance who approves changes that affect ownership. Your upgrade path for Backstage itself should be tested separately, because its catalog module can change between releases.
What security controls should we review before enabling S3 discovery?
Review the principle of least privilege first. The Backstage backend should hold a role that can list and read only the required prefixes, not the whole account. Confirm that bucket encryption and any KMS key permissions allow that role to decrypt objects, and that bucket policies do not expose descriptors to unintended principals. Also check what descriptors contain, because annotations sometimes include internal hostnames or contact details that should not appear in a portal visible to many staff. Record these decisions so the security and compliance teams can test them later. The Backstage authentication documentation also advises against guest access in production, which applies to what users can see in the catalog.
How DeployOpen scopes and delivers S3 catalog discovery
DeployOpen plans, builds and hands over S3-based catalog discovery for Backstage inside your own AWS accounts and networks. We start from how your pipelines or inventories produce catalog-info files, then agree the bucket layout, credentials and verification steps in writing before any work begins.
How we scope
- Inventory the buckets, prefixes and file naming that hold generated catalog-info files
- Confirm which AWS accounts, regions and IAM roles the Backstage backend will read from
- Agree in writing whether discovery covers one bucket or several, and how deletions are handled
- Engagements are fixed-scope, with deliverables and timelines agreed before work starts
How we build
- Configure the AWS S3 catalog module in the Backstage backend with least-privilege read access
- Set the discovery schedule and entity validation so malformed descriptors are reported, not silently dropped
- Test sample pipeline output end to end and confirm entities appear with the expected owners and lifecycle
- Confirm the pipeline job that writes descriptors and the Backstage reader use the same schema version
How we hand over
- Deliver the configuration, IAM policy definitions and a runbook for changing buckets or prefixes
- Document how to diagnose missing entities, including the checks to run in the catalog and the bucket
- Walk your platform and security teams through the verification results and the rollback steps
- Record open decisions, such as bucket versioning or retention, that remain with your organization
How we support
- Ongoing support, upgrades and continuous testing are available as retainers
- Retainers can cover Backstage version bumps that affect the AWS catalog module
- Periodic review of IAM permissions and bucket policies against the agreed scope
- Ask any provider which upgrade and support terms apply to your environment before signing
