Custom sources in Backstage search with collators

Indexing custom sources in Backstage search with collators: when teams need it

A collator is the Backstage Search component that reads documents from a source and hands them to the search index. Backstage Search uses collators to index content from the Software Catalog, TechDocs and other plugins, so a developer who types a service name finds the catalog entry without extra work. A team needs a custom collator when the content it cares about lives outside those defaults, such as runbooks in a wiki, OpenAPI specifications in a registry, or incident procedures in a ticketing tool, and developers keep leaving the portal to find them. Writing a custom collator is a design decision, not a default. It adds code you own, a schedule you must run, and a permissions question you must answer before the first document is indexed.

Start by listing the content developers actually search for and checking where each item already lives. For each candidate source, confirm three things: whether an existing plugin already exposes it to search, whether the source has a stable identifier and a last-modified value that lets the collator skip unchanged documents, and whether a reader's access to the source can be mirrored in search results. If a source cannot meet the last point, indexing it can expose content to people who should not see it, so the collator needs an access decision first. Ask your platform team which search engine is in use before scoping the work, because the volume and freshness needs for a few thousand runbooks differ from those for a large document store, and the engine choice determines what the collator must produce.

Search engine options and when each fits

This table compares Lunr, the Postgres engine, and Elasticsearch or OpenSearch as Backstage search engines for the index size and licensing you expect. Read each row against your document count, your operating capacity and your license policy. Lunr is only a fit for small trials, the Postgres engine covers modest indexes, and larger collator output points toward a dedicated engine.

Option	Fit for index size	License and operating model	Confirm before choosing
Lunr	Small trials and test portals only	Built into Backstage search, no separate service	Lunr is highly discouraged in production by the Backstage search documentation
Postgres engine	Modest indexes, described as performing well with tens of thousands of documents	Uses your existing PostgreSQL, which must be version 12 or later	Confirm your PostgreSQL major version and its support window
OpenSearch	Larger indexes and heavier query load from many collators	Apache License 2.0 for all project software; the OpenSearch FAQ describes no paid edition	Confirm features such as security plugin options against the version you deploy
Elasticsearch	Larger indexes and heavier query load from many collators	Elastic License 2.0 is the default distribution; AGPLv3 is an OSI-approved option; ELv2 and SSPL are not OSI-approved	Confirm which license applies and which features your subscription tier includes
Index size growth	Document count rises as you add TechDocs, custom collators and entity types	Dedicated engines add a service to run, back up and upgrade	Ask your team to estimate document volume per collator before choosing

Decisions to make before adding a custom source

A custom collator adds documents to a search index that every signed-in user can query. The decisions below set what enters the index, who can see it, how fresh it is, how large it becomes and who maintains it after launch.

Which content earns an index entry

Start from the questions people actually ask the portal, not from every table your plugin can read. Each indexed document adds storage, refresh work and noise in results, so a runbook index entry must justify itself against a catalog entity that already answers the same question. Pick a small set of fields per document, such as title, owner and a short summary, and leave out raw bodies or logs unless a team has asked for them. Check that each source has a stable identifier so the collator can update or remove a document instead of duplicating it.

Visibility that matches the source

Search results must respect the same visibility that governs the catalog and TechDocs. A collator that stores only text and a location gives every signed-in user the result, including pages they cannot open. Store the entity reference or ownership metadata with each document, then confirm during testing that a user without access to the source never sees its title or snippet. Treat this as a design gate: if the source has no access model you can map to, index it later or not at all.

Refresh frequency against source load

The index is only as current as the last collator run. Frequent refreshes give fresher results but add API calls to the source system and load on the search engine. Ask the source owner what rate limits and maintenance windows apply, then choose a schedule that meets the team's tolerance for stale results. Remember that deletions need the same attention as updates, so a document removed at the source must also leave the index on the next run.

Growth in document volume

Document count drives the engine choice. The Postgres engine is described as performing well with tens of thousands of documents, so estimate counts for each source, including history, before launch. If a source such as tickets or deployment events can grow past that range, plan for a larger engine early rather than migrating the index under load later. Measure the count after the first full run and compare it with your estimate.

Ownership of collator code

A collator is custom code that depends on a plugin's API, the Backstage release line and the search engine version. Name an owner for it before launch, with a plan for reviewing it during each Backstage upgrade. Without an owner, a changed source API can silently stop indexing, and results decay while the portal still looks healthy. Ask who will write tests for the collator, who receives alerts when a run fails and who approves changes to the fields that get indexed.

Adding a custom collator and checking results

This sequence takes a custom content source from a collator written in the Backstage backend to documents that appear in search with correct titles and links. Each step ends with a check, so problems surface before the source is relied on.

Confirm the Backstage version and search setup
The platform team checks which Backstage release the portal runs and confirms that the search plugin is installed in the backend and that the frontend search page is reachable. Use the Backstage release and versioning policy to confirm the release line is still supported before adding new code. The step is done when the backend starts with search enabled and the existing Software Catalog and TechDocs results appear, which proves the baseline works before you change anything.
Choose and provision the search engine
The platform team selects the engine that fits the expected index size and provisions it in infrastructure you control. Lunr is highly discouraged in production, so for the Postgres engine confirm that the database runs PostgreSQL 12 or later with a version still under support, and confirm that the Backstage search database is separate from the catalog and other plugin databases. The step is done when a connection test from the Backstage backend succeeds and the reported server version is 12 or later.
Write the custom collator
A backend developer writes a collator that reads documents from the custom source, maps each record to a title, text, location and any fields used for filtering, and returns them in the format the search engine expects. Keep the mapping limited to fields the portal should expose, since indexed text is searchable by every user who can reach search. The step is done when the collator runs locally against a small sample and returns the expected number of documents with non-empty titles and locations.
Register the collator in the search backend
The developer registers the collator with the search indexing process in the Backstage backend, under a document type that the frontend can filter on, and restarts the backend. Confirm that the registration appears in backend startup logs without errors and that the new document type is listed as a source. The step is done when the backend starts cleanly with the collator registered and no indexing errors are logged at startup.
Set the indexing schedule
The platform team sets how often the indexing process runs for the custom collator, balancing how fresh results must be against load on the source system and the search engine. Confirm the chosen schedule in configuration and check that the custom source is queried on the interval you set, not only at startup. The step is done when the logs show the scheduled run for the new collator at the configured interval.
Run the first indexing pass and review index size
Trigger or wait for the first full indexing pass, then review the document count and index size reported by the search engine. Compare the count with the number of records in the source system, and check the index size against the storage you provisioned in the engine step. The step is done when the document count matches the source for the expected scope and the index size fits the provisioned storage with room to grow. If the count is off, check the collator filter before changing the engine.
Verify expected documents in search results
Search for known records from the custom source in the Backstage search page and confirm that each expected document appears with the correct title, description and link. Click several results to confirm that the links open the right record in the source system and that a user without access to the source cannot see restricted content. The step is done when a sample of known records returns with correct titles and working links, and the results are recorded so the same checks can run after the next release upgrade.

The PostgreSQL 12 requirement and supported versions

The PostgreSQL search engine for Backstage needs PostgreSQL 12 or later, but PostgreSQL 12 is no longer among the supported major versions. Before you build a search database, pick a major version that is still supported. The supported lines are 18, 17, 16, 15 and 14. Version 18 is supported until November 14, 2030, 17 until November 8, 2029, 16 until November 9, 2028, 15 until November 11, 2027, and 14 until November 12, 2026. Choosing 14 leaves little time before its end of support, so a new deployment should normally start on a newer line. Confirm the exact minimum version in the search engine documentation for the Backstage release you run, because the requirement is tied to that release and may change. The Backstage search engines documentation is the natural starting point.

Plan the database lifecycle as part of the search design. Minor PostgreSQL releases never change the internal storage format, so applying them means replacing executables and restarting the server, which is routine maintenance for the search database. Major upgrades are different. The supported paths are pg_upgrade, a dump with pg_dumpall followed by a restore, or logical replication, which works across major versions and can keep downtime to several seconds. Each path needs a rehearsal against a copy of the index data, a check that the Backstage search configuration still connects, and a rollback plan. Ask your database team which path they will use and who owns the window. Consider also which PostgreSQL version your Backstage backend database already runs, since both should be kept on supported lines together. The PostgreSQL versioning policy lists the current end dates you should track.

Security and access considerations for search indexes

A search index copies content out of its source systems, and the index does not inherit the access rules of those systems by default. The cards below cover the risks that connect collators, search engine credentials and audit trails, and the checks to run before any custom source goes live.

Index content can outlive source permissions

A collator reads what its source returns, and search results can show a document to someone who could not open it in the source system. Catalog and TechDocs visibility does not automatically carry over to a custom source. Before indexing, list who can read each source today, then confirm whether the search layer filters results by the user's permissions or only by the collator's read access. Test with a user who lacks access to a sample document.

Filtering results by user identity

Document-level filtering is the control that decides whether a search hit reaches the right reader. With OpenSearch, the Security plugin provides role-based access control, document-level and field-level security, and field masking, so a role can be limited to matching documents and fields. With PostgreSQL, Backstage's search layer decides what a query returns, so confirm how the engine applies per-user filters before relying on it for restricted sources.

Search engine credentials and TLS

The Backstage backend holds credentials for the search engine, and any collator that reads a private source may hold its own tokens. Keep these in your secrets store, not in app-config files committed to Git, and rotate them on a schedule you set. Encrypt traffic between the backend and the engine with TLS, and confirm which certificate authority and hostname checks your chosen engine enforces.

Auditing who queries what

Search logs show which terms users looked for, which often reveals sensitive project names or incident details. Decide whether query logging is needed for your compliance scope, where those logs are stored, and who may read them. OpenSearch offers audit logs through its Security plugin, and PostgreSQL logging depends on your server configuration. Confirm what each engine records before you promise an audit trail to anyone.

Separating database and application roles

Backstage gives each backend plugin its own logical database, so the search connection should use a role that can read and write only the search tables it needs. Avoid using the database owner or a superuser for the application. PostgreSQL supports several client authentication methods, including password, certificate and OAuth, so choose one that fits your directory and confirm it with your database team before rollout.

Running the index as the portal grows

When search runs on PostgreSQL, the work is mostly database operations, since the index lives in the same operational model as the database you already run. PostgreSQL does not provide the software to detect a failed primary or notify a standby, so failover depends on external tools that you select, configure and test. Plan these components before the portal depends on search:

Backups: Use continuous archiving and point-in-time recovery, and schedule restore tests. Confirm which databases your Backstage installation writes to on the same server, so the restore test covers all of them.
High availability: Hot standby servers serve read-only queries. Choose synchronous replication, where a commit waits for standbys, or asynchronous replication, which does not wait. Synchronous replication protects committed writes but adds latency to each commit.
Failover tooling: Select an external tool to detect primary failure, promote a standby and redirect clients. Rehearse promotion and confirm how Backstage reconnects after the switch.
Connection pooling: PgBouncer limits the number of server connections as backend instances grow. Confirm that your pool mode works with your database driver.
Major upgrades: Plan major version moves with pg_upgrade, dump and restore, or logical replication. Check the support end date for your release in the PostgreSQL versioning policy, since PostgreSQL 14 is supported only until November 12, 2026.

Before go-live, confirm that the server runs PostgreSQL 12 or later, that a named owner runs failover, and that restore tests are on a recurring schedule.

Limits to confirm in your environment

The collator design depends on details that change between Backstage releases, engine versions and license tiers. Confirm each item below against your target release and your own document volume before you commit to an engine or a schedule.

Collator API for your release

Collator class names, constructor options and the way collators are registered with the search engine have changed across Backstage releases. Confirm the exact collator API in the documentation for the Backstage version you run, and check whether your custom source plugin needs the same module wiring. Pin the release you plan to build against and test the collator on that version before you rely on it in a plan.

Engine version your release supports

The Postgres search engine needs PostgreSQL 12 or later, but the Backstage release you choose may set a higher practical floor and the Node.js version it requires is tied to the release line. Confirm the engine versions your Backstage release supports, then match them to a PostgreSQL version that is still in its support window. Record the result in your architecture decision.

Indexing performance at your volume

The Postgres engine is described as performing well with tens of thousands of documents, but your count depends on how many catalog entities, TechDocs pages and custom records your collators emit. Measure your real document count and index build time in a staging environment with production-like data. Decide in advance when a dedicated engine such as OpenSearch becomes necessary.

Elastic and OpenSearch license tiers

Security features such as SSO, field-level security and audit logging can sit in paid Elastic tiers, while OpenSearch documents them in its Apache-licensed security plugin. Confirm which features your access model needs, then check the current Elastic subscription terms or OpenSearch documentation before choosing. Ask legal to review any license terms that apply to your deployment.

End-of-life dates for engine versions

Each engine and database version has a maintenance window, and planned versions can reach end of life during your Backstage release cycle. Check the published end-of-life dates for the Elasticsearch, OpenSearch and PostgreSQL versions you intend to run, and compare them with the upgrade schedule for your Backstage line. Plan the search engine upgrade alongside the Backstage upgrade rather than after it.

Questions buyers ask about custom search sources

What does a custom collator cost in effort, and what drives it up?

The effort depends mainly on how many sources you add, how each source decides who can see what, how its records map to Backstage search documents, and how much testing the index needs. A collator for one internal API with a simple schema is a small task. A source with per-record access rules, several object types and inconsistent field names takes longer, because each rule must be checked against the permission model before results appear. Ask any partner to price each source separately, and to state what is included for access rules, schema mapping, a test index and handover. Those four items usually account for most of the work. Cost is mostly one-time build effort, plus recurring effort when a source's schema changes or the Backstage version is upgraded.

How long does a custom search source take to deliver?

Duration depends less on the collator code than on how ready your team and your data are. The work moves faster when the source has a stable API, documented fields, a named owner who can answer questions about the data, and a clear rule for who may see each record. It slows down when access rules are undocumented, when the source is only available through exports, or when approvals for a test environment take weeks. Ask for a written schedule that lists the dependencies on your side, such as API credentials, sample data and staging access, so that delays are visible early. No general duration is stated here, because each source needs to be scoped before it can be estimated.

What must we provide before a search collator project starts?

You need to provide four things. First, access to the sources, with read credentials for a non-production copy where possible. Second, a named owner for each source who can explain its fields, its freshness and which records are sensitive. Third, a written access model that states how a user's permission to see a record in the source maps to permission to see it in Backstage search. Fourth, a staging Backstage instance that matches production configuration, including the search engine and the sign-in setup, so that results can be tested with real user groups before anything reaches production. Missing any of these usually moves work into testing, where it costs more.

Can our internal team build a custom collator without a partner?

Yes, if the team already runs Backstage in production and has a developer comfortable with TypeScript and the Backstage plugin model. The collator itself is an ordinary plugin component, so the skill gap is usually in access control and testing rather than in writing the indexing code. The risk to check is who owns the result after the first version. Confirm that someone on your team will keep the collator working through each Backstage upgrade, will review changes to the source's schema, and will test that results respect permissions. If that ownership is not clear, a fixed-scope engagement with written handover can cover the build and the knowledge transfer.

Is Lunr acceptable for a small Backstage portal?

Lunr is discouraged for production use, according to the Backstage search engines documentation, even for portals that are small. It runs in memory and is intended for development and trials. If your portal is small but live, the safer choice is the Postgres engine, which requires PostgreSQL 12 or later and is described as performing well with tens of thousands of documents. Check the PostgreSQL version your Backstage deployment already uses before you enable it, because the Postgres engine has a version requirement that an older database may not meet. The engine decision should be made before the first collator is built, since it shapes how documents are stored.

Is switching search engines later expensive?

The cost depends on how many collators you have and how much logic sits in their document mapping, not on Backstage itself. Collators describe what gets indexed, so moving from one engine to another mainly means reconfiguring the engine, rebuilding the index and retesting results and permissions on the new engine. Mapping code that is written cleanly and tested against the permission model moves easily. Mapping code that depends on engine-specific behavior, or access rules that were never written down, is where the rework comes from. Ask any partner to document each collator's inputs and access rules as part of the build, so that a later change is limited to the engine configuration and a re-index.

Which engine should we pick if the index will grow large?

Choose between the Postgres engine and Elasticsearch or OpenSearch by the size of your index and the operations skills you have. The Postgres engine is described as performing well with tens of thousands of documents and runs on the PostgreSQL database you may already operate. Once the index and query load grow beyond that, or you need features such as richer ranking, plan for OpenSearch or Elasticsearch. OpenSearch is released under the Apache License, Version 2.0, and Elasticsearch's default distribution uses the Elastic License 2.0, which is not an OSI-approved open source license. Confirm the license and support terms of any engine against your own policy before you commit, and size the index with a test load in staging.

How DeployOpen scopes and delivers custom search sources

DeployOpen treats each custom search source as a scoped piece of work: which content enters the index, who may find it, and how the index is run afterward. We plan collators, engine choice and PostgreSQL versions together so the search layer fits the infrastructure you control.

How we scope

Inventory every plugin whose content should appear in results, with its owner and access rules.
Confirm the expected document count and whether the Postgres engine fits or a larger engine is needed.
Record the PostgreSQL version your environment runs, checked against the supported version list before work starts.

How we build

Write and register collators for each agreed source, keeping document fields limited to what users need.
Test that search results respect catalog permissions so restricted content never appears to unauthorized users.
Validate indexing schedules and rebuild behavior on a staging copy before changes reach production.

How we hand over

Document each collator, its source, schedule, index location and the owner responsible for changes.
Provide runbooks for reindexing, engine changes and PostgreSQL upgrades on your own infrastructure.
Walk your platform team through the test results and the open decisions recorded in the written scope.

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Confirm with your team which search-related changes need review when Backstage or PostgreSQL releases arrive.

Indexing custom sources in Backstage search with collators