OpenMetadata high availability

Treat each component in an OpenMetadata high availability design as its own failure path. Losing one server replica, failing over the database, losing a search node, restoring a search cluster and interrupting an ingestion run each leave the platform in a different state, and only some of those states recover without help. The supplied material does not describe OpenMetadata's failover internals, so the behavior below is what you must verify, not what you can assume. Before go-live, run each path in a non-production environment that mirrors your Kubernetes layout, your managed MySQL or PostgreSQL service and your Elasticsearch or OpenSearch cluster, and record what users, APIs and scheduled jobs actually experience. A server replica loss is the path most teams expect to be invisible. In a healthy multi-replica deployment, the Kubernetes Service should route new requests to surviving replicas, and in-flight browser requests may simply be retried by the user. Confirm with Collate or your own test results whether any server-side operation, such as a bulk metadata import or a long-running API call, is lost when its replica terminates. Also confirm that background tasks are not pinned to a single replica, because a scheduled job tied to a dead pod can stop without an obvious error. A database failover depends on your managed service and how OpenMetadata reconnects after the primary changes. Connections that were open at the moment of promotion may fail, and writes that were in flight may be rejected rather than retried. Ask your database provider how long promotion takes and whether the connection endpoint stays the same. Then test whether OpenMetadata reconnects on its own or needs its pods restarted, and whether any writes acknowledged just before the failover are missing afterward.

A search node loss is usually less disruptive than a database failover, because a clustered Elasticsearch or OpenSearch deployment is designed to keep serving shards from surviving nodes. Search is an index over metadata, not the system of record, so the practical risk is degraded search results, slower queries or indexing that stops until the cluster returns to a healthy state. Check cluster health thresholds, replica shard counts and the time your cluster takes to rebalance. A full search cluster restore is different: it brings back an index that may be older than the database, so search results can disagree with the metadata users see on their pages. After any search restore, plan to reindex from the database. Do not assume the restored index is current. Decide who triggers the reindex, when it runs relative to business hours, and how users are told that search may be incomplete while it finishes. Ask Collate how reindexing is triggered and how long it takes at your metadata volume, and measure it in your test environment before you rely on it for recovery. An ingestion run interrupted mid-way is the path most likely to need a human. Requests that hit a temporarily unavailable API may retry, but a metadata ingestion workflow that loses its connection partway through is generally not resumed from its last step. Confirm whether the scheduler re-runs the workflow automatically, whether partially ingested assets are left in a consistent state and whether the next run is a full or incremental pass. Assign a named owner to review failed runs after any infrastructure event. Identity state needs its own check. If the identity provider is not shared across replicas or is reconfigured per pod, a user may hold a session that one replica accepts and another rejects, which appears as random sign-outs or permission errors. Confirm how OpenMetadata stores session state, test a replica restart during an active browser session and document the steps users should take when they see an authentication error. Because OpenMetadata does not allow SSO and basic authentication at the same time, a fallback login path cannot be used to work around an identity outage, so plan that recovery path before a failure occurs.

Build a short runbook that maps each failure path to a detection signal, an owner and a recovery action. Include the checks you ran in the non-production environment, the results and the open questions you sent to the vendor. Most of the manual work in this design sits in three places: confirming which in-flight operations are lost, rebuilding search after a restore and re-triggering ingestion runs that did not finish. Scope those tasks into your rollout plan and into your support arrangements, because a fixed-scope engagement or retainer will only cover them if they are written into the agreed deliverables. When you ask Collate or your platform team for confirmation, ask specific questions: which API operations are idempotent, whether a workflow resumes or restarts, what state the search index is in after a restore, and which settings control replica count and session storage. Record each answer with the version you tested, because release-to-release behavior can change between OpenMetadata releases and patch versions.

Your go-live decision for OpenMetadata high availability should rest on tests run against your own deployment, not on the replica count in a manifest. Run a fixed set of failure tests before cutover, and agree the pass criteria in writing with the platform, data and security owners before the first test starts. Each test should name the component it breaks, the checks that run during the failure, and the result that counts as a pass.

Start with the OpenMetadata server replicas on Kubernetes. Stop one replica at a time while a scripted loop calls the REST APIs (for example, searching for a table and fetching its lineage) and a second loop logs into the UI and opens an asset page. Record any failed request, the replica that served it if your ingress exposes that, and the time until responses return to normal. Repeat with each replica so that no single pod is the only one you have exercised.

Next, fail over the database. With a managed MySQL or PostgreSQL service, trigger the provider's failover method and confirm that the OpenMetadata servers reconnect without a manual restart. Write a test asset before the failover and read it back afterward. Confirm the engine versions you run meet the bare metal prerequisites published for OpenMetadata (MySQL 8.0.42 or later, or PostgreSQL 15 or later), and ask your database provider to confirm how failover affects open connections in your tier.

Then remove one search node, either Elasticsearch 9.x or OpenSearch 3.x, and run the same search queries you used for the baseline. Compare result counts and ranking for the test assets. Check that a scheduled ingestion run still reports a valid status and does not stall. Restart the ingestion scheduler while a run is in progress, then confirm whether the run resumes or is marked failed and rerun, and check for duplicate entities. Confirm the scheduler behavior in your exact version, because it is the behavior you will depend on in production.

Finally, restore the most recent backup into a separate environment and compare counts of entities, lineage edges and data quality test results against a snapshot taken from the source at the same point. Any difference must have a documented explanation. Keep the evidence for every test in one place: timestamped test logs, the recovery times you measured in your environment, and the configuration versions in effect (server image tag, Helm chart version, database and search engine versions, Java 21 runtime and ingestion image). A failed test should block go-live until the cause is found, fixed, and the failed test and the full suite pass again with the evidence recorded.

After go-live, the high availability design only protects you if someone watches it. Assign an owner for each signal in the cluster: the OpenMetadata server replicas on Kubernetes, the MySQL or PostgreSQL database, the Elasticsearch or OpenSearch cluster, and the ingestion scheduler. For the servers, track pod restarts, readiness failures and response times at the load balancer. For the database, track connection saturation, replication lag if you run replicas, and storage growth. For the search cluster, track cluster health status, unassigned shards, heap pressure and index size. Ingestion needs the most attention because a failed run leaves the catalog quietly out of date. Review run outcomes daily, route failed workflows to the data owners who depend on them, and treat repeated failures from one connector as a defect to fix, not as background noise. Confirm which alert channels and escalation paths your operations team already uses before adding OpenMetadata to them.

Backups must cover the database and the search indexes as separate assets, and the restore procedure must be tested for both. The database holds the authoritative metadata, so its backup schedule and retention should follow your recovery point and recovery time objectives, agreed with the business owners of the catalog. Search indexes hold derived data, but rebuilding them after a loss takes time and load on the cluster, so decide whether a snapshot schedule for the search cluster is worth the storage cost. Ask your implementation partner to confirm which reindexing options your OpenMetadata version provides after a restore, and how long a full reindex takes on your data volume. Run a restore into a separate environment on a fixed schedule, record the result, and keep the record with the runbook. An untested backup is an assumption, and auditors will ask for evidence that the restore worked.

Plan upgrades around the release cadence rather than around a fixed annual window. OpenMetadata 2.0.0 was released on August 24, 2026, and maintenance releases 2.0.1 to 2.0.5 have followed at about one to two week intervals. Each patch should be read in its release notes, tested in a staging cluster with a copy of production metadata, and promoted on a planned date. Keep the server, ingestion and search versions recorded together so that a rollback has a known target. Before each upgrade, take a fresh database backup, confirm the restore test from the last cycle is current, and check that the connector set used in production still runs against the new ingestion images. Agree in advance who approves each promotion and how long the team will hold the previous version before retiring it.

The 2.0 release contains breaking changes that must be handled before the upgrade, not during it. Embedding credentials moved into an llmConfiguration block, so any configuration that still holds them in the old location needs to be updated and tested. The Databricks connector uses a new authentication format, so service credentials and connection definitions for Databricks sources must be rewritten and verified against a test workspace. Custom ingestion images must be rebuilt for Python 3.12 because the official ingestion images moved to Python 3.12 with Apache Airflow 3.3.0. Great Expectations must be migrated to 1.3 before you upgrade, which means data quality test suites that depend on it need to be reviewed on a staging system first. Build this work into the upgrade plan as its own tracked item, with an owner and a test result for each change.

Keep documentation current enough that an auditor or a new operator can follow the platform without interviewing the original builders. The core set is an architecture diagram of the replicas, database, search cluster and scheduler, a runbook covering monitoring, alert response and escalation, and a change log recording each OpenMetadata version applied, the date, the approver and the staging test result. Add backup schedules, retention settings and the dated results of each restore test, along with an inventory of configuration items and where their secrets are stored, without the secret values themselves. Record how ingestion failures are triaged and who owns each connector. DeployOpen can prepare these records and the controls behind them, but independent auditors decide whether your controls meet the standard you are assessed against.