Which DataHub install fits your stage
A team evaluating DataHub or preparing to move it toward production needs to decide which install path matches the stage it is in. The Docker quickstart fits evaluation, and the Helm prerequisites and main charts fit production. The supplied facts describe the quickstart as requiring Docker with 8 GB of RAM and Python 3.10 or later, and they do not describe it as a production install, so treat it as a way to explore the product, connect a few sources and test lineage and glossary workflows on a single machine.
The quickstart is the right choice when the goal is to learn the product, validate a handful of integrations, or let a small group of stakeholders see what discovery and governance look like before any infrastructure commitment. It is the wrong choice when the instance must hold shared production metadata, meet internal availability expectations, or be upgraded under change control. Production calls for the Helm prerequisites chart for Kafka, the database, Elasticsearch and the graph index, followed by the main chart for the GMS, MAE and MCE consumers and the frontend. Plan to run those dependencies as external services that your team can back up, monitor and patch, and confirm with your platform team where they will live.
Before choosing the Helm path, check two things. First, the Helm chart runs a system-update job on upgrades to apply schema changes and reindex search when needed, so schedule upgrades with a tested backup and a maintenance window. Second, the Kubernetes guide's own minimum of more than 7 GB of RAM for DataHub and its dependencies on Minikube is a local test figure, not a production sizing, so ask your vendor or DeployOpen for sizing based on your source count, metadata volume and retention needs. A quickstart that succeeds does not prove the production footprint will fit.
How each install fits evaluation, pilot and production
The Docker quickstart is built for someone who needs to see DataHub working before committing infrastructure. The README states it requires Docker with 8 GB of RAM and Python 3.10 or later, so the first decision is whether a workstation or a lab VM meets that bar. The README also lists more than 150 integrations, including Snowflake, BigQuery, Databricks, dbt, Looker and Tableau. For an evaluator, that breadth is a reason to try it early: you can connect the sources your estate actually uses and judge discovery, lineage and glossary behavior on your own metadata.
- Hardware floor: 8 GB of RAM is the documented minimum for the quickstart, so confirm your machine has headroom for the browser and connectors.
- Runtime: Python 3.10 or later must be on the path used by the ingestion CLI.
- Source coverage: Test two or three of your own systems rather than the demo data.
- Feature check: Column-level lineage, the business glossary and data quality assertions are listed features, so verify them against your sources.
- Scope limit: A single-machine stack does not show how the services behave under production load or failover.
Before leaving evaluation, confirm which integrations and features your use cases depend on in the current release notes.
A production install on Kubernetes uses the Helm charts from the acryldata/datahub-helm repository together with a separate prerequisites chart for dependencies. The DataHub Kubernetes guide describes four main components: GMS, the MAE Consumer, the MCE Consumer and the Frontend. These need Kafka, a MySQL, PostgreSQL or MariaDB database, Elasticsearch for search, and a graph index on Neo4j or Elasticsearch. The guide also says more than 7 GB of RAM is needed to run DataHub and its dependencies on Minikube, which is a floor for a test cluster rather than a sizing guide for production.
- Components: Plan capacity and failure handling separately for GMS, both consumers and the Frontend.
- Kafka: Streaming metadata changes depend on Kafka, so decide whether to run it on your existing platform.
- Database and search: Use managed or externally operated MySQL or PostgreSQL and Elasticsearch or OpenSearch where your standards require it.
- Graph index: Choose Neo4j or Elasticsearch for the graph index and confirm the operational support you will have for it.
- Upgrades: The system-update job applies schema changes and reindexes search when needed, so schedule upgrades with that job in mind.
Confirm the sizing, supported versions and external service options with the current DataHub Kubernetes documentation before you write a capacity plan.
A security reviewer is less interested in the install command and more interested in where metadata flows, who can reach it and where secrets live. The Helm design places several stateful services in the cluster, each with its own access path. Kubernetes authenticates users through X.509 client certificates, bearer tokens including OpenID Connect and JWT, webhook token authentication or an authenticating proxy, so the cluster identity model has to be defined before DataHub is layered onto it. Kubernetes stores Secrets unencrypted in etcd by default, which means DataHub credentials need the same controls as the rest of the cluster.
- Identity: DataHub supports OpenID Connect SSO with providers such as Google, Okta and Azure AD. On self-deployed DataHub Core, the operator configures the frontend server.
- Secrets: Enable encryption at rest for etcd, restrict list and watch access with least-privilege RBAC, or use an external store through the Secrets Store CSI Driver.
- Pod security: Apply the Pod Security Standards to DataHub namespaces and review the CIS Kubernetes Benchmark for the cluster.
- Data path: Map where metadata is stored in MySQL or PostgreSQL, indexed in Elasticsearch or OpenSearch and streamed through Kafka.
- Evidence: Ask the platform team to produce the configuration and access records an auditor would request.
Confirm the SSO and secrets configuration in your own environment, since the controls depend on your cluster and identity provider.
Docker quickstart and Helm deployment side by side
This table compares the datahub Docker quickstart with the Helm charts for Kubernetes across stage, sizing, layout, external services, upgrades, availability, SSO and skills. Read it by matching your stage first, then checking the production rows against your own load tests and the DataHub documentation.
| Dimension | Docker quickstart | Kubernetes with Helm |
|---|---|---|
| Intended stage | Evaluation, demos and proof of concept on a single machine | Production or pre-production, with the Helm prerequisites chart and main chart |
| Host requirements | Docker with 8 GB of RAM and Python 3.10 or later, per the DataHub README | More than 7 GB of RAM to run DataHub and its dependencies on Minikube, per the DataHub Kubernetes guide; confirm production sizing with your own load tests |
| Component layout | All services run as containers on one host | Four main components (GMS, MAE Consumer, MCE Consumer, Frontend) scheduled as separate workloads |
| External services | Kafka, database, search and graph services are bundled for the quickstart | Kafka, MySQL, PostgreSQL or MariaDB, Elasticsearch and a graph index on Neo4j or Elasticsearch; a production install should point to services you operate or buy |
| Upgrade mechanism | Re-run the quickstart at a new version; no documented migration job for production data | The system-update job applies schema changes and reindexes search when needed; plan a backup before each upgrade |
| High availability | Not designed for high availability on one host | Depends on replicas and the availability of external Kafka, database and search; confirm the supported topology in the DataHub documentation |
| SSO on self-deployed Core | OIDC must be configured by the operator on the frontend server | OIDC must be configured by the operator on the frontend server; DataHub Cloud enables OIDC through its interface |
| Skills needed | Docker administration and basic Python tooling | Kubernetes operations, Helm, plus running Kafka, a database and search for production |
Why production installs should use external Kafka, database and search
A production DataHub install depends on four stateful services that the quickstart hides inside one Compose stack. Each one can be run from the prerequisites chart or supplied by a separately operated or managed service, and that choice decides who owns backup, patching and capacity.
Kafka for metadata change streams
DataHub's architecture uses Kafka to stream metadata changes between its services, so ingestion and downstream indexing depend on it. In the Helm install, the prerequisites chart can deploy Kafka, but a production team usually wants it managed separately so topic retention, broker sizing and upgrades follow the organization's streaming standards. Check with your vendor or platform team who owns broker upgrades and how partitions and retention will be sized before you commit to the in-cluster option.
Relational database for metadata storage
MySQL or PostgreSQL holds the metadata records that DataHub serves, so it is the system of record for the catalog. Running it inside the cluster means the operator also owns point-in-time backups, restore testing and minor version patching. Confirm with your database team which engine and version your standards support, and whether an existing managed instance can meet your backup and recovery objectives.
Search index for discovery
Elasticsearch or OpenSearch powers search across datasets, owners and glossary terms. If the index is lost, the metadata survives in the database, but discovery is unavailable until the index is rebuilt, which is why the system-update job can trigger reindexing on upgrades. Ask your platform team how long a full reindex takes at your metadata volume, and whether the search cluster will be sized and patched on the same schedule as the rest of your search estate.
Graph index for lineage
Column-level lineage and relationship traversal rely on a graph index, which the Kubernetes guide places on Neo4j or Elasticsearch. Choosing Neo4j adds another stateful service with its own licensing, backup and upgrade path, while Elasticsearch keeps fewer components but changes how lineage queries perform at scale. Confirm the graph backend choice and its license terms with your vendor before you size the cluster.
Shifting operations to the operator
Every external service you choose moves backup schedules, security patches, capacity planning and incident response from the chart to your team. The prerequisites chart is quicker to stand up for evaluation and small pilots, while separately operated or managed services cost more to buy but reduce the number of stateful systems your own staff must patch. Ask each provider for written support terms, recovery targets and sizing guidance, and confirm the figures with them rather than assuming them.
What the system-update job does on each upgrade
The sequence protects the external Kafka, database and search services before the chart changes, lets the system-update job bring schema and search indices in line with the new version, and confirms the result before users return. Operators who rehearse this path in a non-production cluster find problems before they reach production.
Confirm the target release and read its notes
Before touching the cluster, the platform owner checks the DataHub releases page for the target version and reads the upgrade notes for that tag. The current line includes v1.6.0.3, released on September 25, 2026, and v1.8.0 release candidates followed in October 2026, so the team should state which version it is moving to and why. Done means a written change record names the source version, the target version and the owner of the upgrade.
Back up the external stores
The team, working with the database administrator, takes a backup of the MySQL or PostgreSQL metadata store and records the state of the Kafka topics and the Elasticsearch or OpenSearch indices that the deployment depends on. The Helm chart does not replace these stores, so their backups are the rollback path if the migration fails. Done means each backup has a recorded identifier and a tested restore procedure, and the team can say how long a restore takes in its own environment.
Rehearse the upgrade in a non-production cluster
The same chart version, values file and backup steps run first against a copy of production data in a non-production cluster that mirrors the production Kafka, database and search services. This is where schema changes, reindex time and custom values conflicts show up without affecting users. Done means the rehearsal finishes, the team records every warning from the job, and it has agreed a go or no-go decision for production.
Apply the chart with the new version
The operator updates the Helm release to the target chart version with the reviewed values file, so the GMS, MAE Consumer, MCE Consumer and Frontend components move to the new images. The prerequisites chart is upgraded separately only if the release notes require it. Done means the Helm release reports a new revision and the pods for each component reach the ready state.
Let the system-update job run migrations
The Helm chart's system-update job applies schema changes to the metadata store and reindexes the search indices when the new version requires it. Reindexing can be the longest part of the upgrade on large catalogs, so the team should watch the job logs and not assume a finished pod means a finished migration. Done means the job completes without errors and the logs show the migration and any reindex steps reported as finished.
Run post-upgrade checks on search, ingestion and login
After the job completes, the team checks that search returns results for known datasets, that a scheduled ingestion run succeeds against a source it knows, that lineage pages load, and that users can sign in through the configured OIDC provider. Monitoring should show no new error spikes in GMS or Kafka consumer lag. Done means a short written checklist is signed by the platform owner and a data owner, and the rollback decision window closes.
Upgrades, backups, monitoring and skills for each path
This table compares what each DataHub install path asks of your team after go-live: upgrades, backups, dependency monitoring, Kubernetes skills and patching responsibility. Read it as an operating checklist, since the quickstart suits evaluation while the Helm path adds a cluster and its dependencies to maintain. The main takeaway is that the Helm path carries Kubernetes version upkeep on top of DataHub upgrades.
| Area | Docker quickstart (evaluation) | Kubernetes Helm deployment (production) |
|---|---|---|
| Upgrade method | Re-run the quickstart on a new DataHub release. Treat it as a rebuild, not a production change window. | Upgrade the Helm charts. The system-update job applies schema changes and reindexes search when needed, so plan a maintenance window and test on a staging cluster first. |
| Kubernetes version upkeep | Not applicable. Docker runs on the host without a cluster lifecycle. | Plan for Kubernetes minor upgrades. As of September 2026 the project maintains 1.37, 1.36 and 1.35, and 1.35 reaches end of life February 28, 2027 (per kubernetes.io/releases). |
| Upgrade order rule | Not applicable. | The version skew policy requires kube-apiserver to be upgraded first, one minor version at a time, before controller manager, scheduler and kubelets. |
| Backup responsibility | Operator decides whether to keep data. Quickstart data is generally treated as disposable, so confirm this in writing with your evaluation owner. | Your team owns backups for Kafka, the MySQL or PostgreSQL database, search indices and the graph index. Confirm restore order and test restores, since the DataHub docs do not give a single backup procedure. |
| Monitoring of dependencies | Basic container status and logs on one host. Limited alerting unless you add it. | Monitor Kafka lag, database health, search cluster health, pod restarts and the system-update job outcome. Confirm alert thresholds with your platform team. |
| etcd and control plane | Not applicable. | If you self-manage the cluster, etcd needs an odd member count (five recommended in production) and a tested snapshot backup with etcdctl and etcdutl restores. |
| Kubernetes skills needed | Docker and basic container operations. | Helm, Kubernetes RBAC, secrets handling, networking, storage classes and cluster upgrades. Confirm which of these your team already staffs. |
| Who patches what | Operator patches the host and the container image. | Your team patches the cluster, nodes, Helm releases and external services. Confirm with DeployOpen or your vendor what is in scope for each retainer or fixed-scope engagement. |
What a Kubernetes cluster must provide before Helm
The DataHub Helm charts assume a working Kubernetes cluster, and the platform team owns that cluster even though DataHub is the workload. Before installing anything, confirm the control plane topology: a kubeadm high availability setup uses either a stacked control plane, with etcd on the same nodes, or an external etcd cluster, and needs at least three control plane nodes with a TCP load balancer in front of kube-apiserver on port 6443. Check that the etcd member count is odd and that the team has chosen a size, since the Kubernetes documentation recommends five members in production. These decisions are made once, but they determine how the cluster behaves when a control plane node fails during a DataHub upgrade.
Next, own the etcd backup and restore process. Snapshots are taken with etcdctl snapshot save, and restores are performed with etcdutl, so the runbook should name who runs them, where snapshots are stored and how often a restore is rehearsed. This matters for DataHub because its metadata lives in MySQL or PostgreSQL, Kafka and search indices, while the Kubernetes objects that describe the deployment live in etcd. Ask whether your backup scope covers both sides, and whether the database and search backups can be restored to a matching point in time. Kubernetes stores Secrets unencrypted in etcd by default, so encryption at rest for Secrets must be configured explicitly, and DataHub's database and SSO credentials should be kept out of plain values files.
Finally, set least-privilege RBAC and Pod Security Standards for the namespace that will host DataHub. Restrict who can list and watch Secrets, because those permissions expose the same credentials, and use dedicated service accounts for the Helm release rather than cluster administrators. Pod Security Standards define the pod security levels that a namespace enforces, and the DataHub pods should be admitted under the level your security baseline requires, which means confirming the chart's security context settings against that level in a non-production namespace first. Confirm these controls with your cluster team and against the CIS Kubernetes Benchmark, which your assessor may use as a reference, before any production data is loaded.
Questions teams ask before choosing a DataHub install
Can we run the Docker quickstart in production?
The Docker quickstart is built for evaluation, not production. The DataHub README says it requires Docker with 8 GB of RAM and Python 3.10 or later, which is enough to see discovery, lineage and ingestion working on a single machine. A production install needs separate, sized services for the metadata service, consumers and frontend, plus external Kafka, a relational database, search and a graph index that you can back up, monitor and upgrade. Use the quickstart to validate connectors and your metadata model, then plan the Helm path for anything with users, retention or recovery requirements.
What does the Helm install ask of our platform team?
The Helm install asks your platform team to run Kubernetes and the DataHub dependencies as managed services. DataHub is deployed with charts from the acryldata/datahub-helm repository plus a separate prerequisites chart, and the guide lists four main components (GMS, the MAE Consumer, the MCE Consumer and the Frontend) that need Kafka, a MySQL, PostgreSQL or MariaDB database, Elasticsearch and a graph index on Neo4j or Elasticsearch. Your team will own cluster capacity, ingress, certificates, secrets handling, backups and the upgrade process. Confirm with your platform team which of these services you already operate and which you would have to add.
How much does a production install cost to run, and what drives it?
The recurring cost comes from infrastructure and people, not from a DataHub license, since DataHub is Apache-2.0 licensed. Infrastructure covers the Kubernetes nodes, Kafka, the database, search and graph services, storage for backups and any managed-service fees you choose to pay for those components. People cost covers the platform engineers who patch, upgrade and monitor the stack. Cost rises with the number of ingested sources, metadata volume, search index size and high availability requirements. We do not publish sizing or price figures for DataHub. Ask the vendor for current resource guidance, and ask us for a costed scope tied to your environment.
How long does a Helm rollout take?
The duration depends on your cluster, your dependency choices and how much integration work sits around the install, so no fixed timeline applies. The DataHub Kubernetes guide notes that more than 7 GB of RAM is needed to run DataHub and its dependencies on Minikube, which only sets a local floor, not a production schedule. Plan a pilot in a non-production cluster that covers ingress, SSO, one or two ingestion sources and an upgrade rehearsal. The pilot results are the best basis for estimating the production rollout.
Can our own engineers run a DataHub Helm install without outside help?
Yes, if your team already runs Kubernetes, stateful services and Helm releases in production. The harder parts are operating Kafka and search at scale, configuring SSO on self-deployed Core, and planning upgrades. The Helm chart runs a system-update job on upgrades to apply schema changes and reindex search indices when needed, so upgrades need a tested runbook and a recovery plan. Teams without that depth often use outside help for the first install and the first upgrade, then run it internally. Be realistic about the skills gap before you commit.
How does SSO differ between self-deployed DataHub Core and DataHub Cloud?
On self-deployed DataHub Core, your operator configures the frontend server for OpenID Connect single sign-on, with providers such as Google, Okta or Azure AD. DataHub Cloud enables OIDC SSO through its interface. The Core setup means your team manages the identity provider registration, redirect URLs, secrets and configuration changes on every environment. Confirm the exact steps against the DataHub OIDC SSO guide for your version, and include identity testing in your pilot.
What should we ask for in a fixed-scope quote?
Ask for a written scope that names the environments, the Kubernetes platform you will use, the external Kafka, database, search and graph services in scope, the ingestion sources to connect and the SSO provider. Require deliverables for the Helm configuration, secrets handling, backup and restore procedures, an upgrade runbook and a handover plan. Ask how acceptance is tested, what is one-time and what recurs, and what ongoing support or retainer would cover upgrades. Most engagements are fixed-scope and agreed in writing before work starts, so the quote follows from that scope.
Decision points that push a team toward Helm or Cloud
The quickstart fits evaluation on a single machine, and most production questions come down to who runs the dependencies and who holds the evidence. The cards below are the decisions that usually move a team from the Docker quickstart to the Helm charts, or from self-hosting to DataHub Cloud.
High availability and separated dependencies
The quickstart runs every component on one host, which is adequate for evaluation but not for a service that other teams depend on. The Helm guide describes GMS, the MAE Consumer, the MCE Consumer and the Frontend as separate workloads, and it expects Kafka, a relational database, Elasticsearch and a graph index to be available to them. If you need independent scaling, restarts and failure domains, plan for the Helm path and confirm how each dependency will be hosted before you commit.
Existing Kubernetes platform and skills
Helm is only the lowest-effort route when your platform team already runs Kubernetes with the controls it needs. Ask who will own cluster upgrades, ingress, storage classes, secrets handling and pod security settings for DataHub. If the answer is nobody yet, the quickstart or a managed option is a more honest starting point than an unowned Helm release.
Appetite to run Kafka and search
DataHub relies on Kafka to stream metadata changes and on Elasticsearch or OpenSearch for search, so a production install adds two stateful systems to your operating scope. Decide whether your team can handle broker capacity, topic retention, index reindexing and version upgrades for them. If not, budget for a managed or externally run service and confirm that DataHub's supported versions match what that service provides.
Upgrade ownership and system-update runs
The Helm chart runs a system-update job on upgrades to apply schema changes and reindex search when needed. That job touches your metadata store and search indices, so an upgrade is a change to production data, not just a container swap. Teams that need controlled change windows should plan backups of the database and search indices, test the job in a non-production cluster, and assign a named owner for each upgrade.
SSO and compliance evidence needs
DataHub supports OpenID Connect providers such as Google, Okta and Azure AD. On self-deployed DataHub Core, the operator configures the frontend server for OIDC, which adds identity configuration and secret handling to your scope. Auditors will also ask how access, change history and secrets are controlled, so confirm those controls with your compliance lead before choosing a self-hosted path.
Whether DataHub Cloud fits
The DataHub README describes DataHub Cloud as managed, SLA-backed and enterprise-ready, with no infrastructure for you to run. This shifts Kafka, database, search and upgrade operations to the vendor, while data residency, contract terms and the Cloud-only features become your evaluation items. The sibling comparison of DataHub Cloud and self-hosted DataHub covers those trade-offs in detail.
How DeployOpen scopes and delivers a DataHub deployment
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For DataHub, that means deciding whether a Docker quickstart is enough for evaluation or whether a Helm-based install with external Kafka, database and search services is the right production footprint.
How we scope
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- The written scope states whether the target is a quickstart evaluation, a Helm production install, or both.
- Scope names the Kafka, MySQL or PostgreSQL, Elasticsearch or OpenSearch, and graph index services, and who operates each one.
How we build
- We prepare the Kubernetes cluster, Helm prerequisites and main chart configuration in your environment.
- We configure OIDC single sign-on on the frontend server and confirm how login behaves for your identity provider.
- We test upgrades in a non-production environment, including the system-update job that applies schema changes and reindexing.
How we hand over
- Handover documentation covers the deployed topology, service endpoints, secrets handling and backup procedures.
- We document the upgrade runbook, including how to review the system-update job before each DataHub upgrade.
- Operators receive a walkthrough of the deployment so your team can run, troubleshoot and extend the install.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Retainers can cover tracking DataHub release notes and planning upgrades against your change windows.
- Ask for the support scope, response expectations and upgrade responsibilities in writing before you sign.
