Why teams move Superset from Compose to Kubernetes
A Compose-based Superset that now serves many analysts and scheduled reports usually needs rolling updates, restarts that do not interrupt users, and a clear way to scale the web tier and workers independently. The move is mostly a data and configuration exercise. Your dashboards, charts, datasets, roles and saved queries live in the metadata database, so they transfer when that database is carried into the new environment, provided the SECRET_KEY used to encrypt stored database connection passwords is carried over unchanged. The work is less about rebuilding the analytics content and more about externalizing state, rebuilding the image and wiring the routing correctly.
The Superset installation guide describes Docker Compose as the best way to try Superset but less robust than Kubernetes for large production deployments, and it presents Kubernetes as the best-practice production route with rolling updates. That position is the main reason teams make the move, and it also explains the effort: a Compose stack often hides state in named volumes and a bundled PostgreSQL container. The documentation does not recommend the Docker Compose PostgreSQL container for production, so the metadata database has to move to a database service you operate or manage. Confirm with your platform team who owns backups, patching and failover for that database before you start, because those responsibilities do not disappear in Kubernetes.
Before you commit to a date, check which Superset version you run today and which version you will deploy, because the upgrade path and the Kubernetes packaging both depend on that choice. The official Helm chart is marked deprecated in current development documentation, which points new deployments to the Apache Superset Kubernetes Operator. That operator is in alpha with a v1alpha1 API whose schemas may change between releases, so plan for a pinned version and a review step at each upgrade. Installing Superset on Kubernetes is the next read for the current packaging options.
What maps cleanly from Compose and what needs rebuilding
This table pairs each Compose component of a Superset stack with its Kubernetes counterpart and marks what carries over unchanged and what must be rebuilt. Read the first column as the inventory to check against your current compose file, and treat the rebuild rows as the work that drives the migration plan. The main takeaway is that the metadata database and SECRET_KEY move as they are, while the Compose PostgreSQL container and local volumes must be replaced before cutover.
| Component | Docker Compose today | Kubernetes target | Migration effort |
|---|---|---|---|
| Web server | Superset service running Gunicorn behind a published port | Deployment of the Superset image behind a Service and ingress | Rebuild: image, probes and routing change |
| Metadata database | PostgreSQL container with a local volume | Externally managed PostgreSQL 10 to 17 or MySQL 5.7 or 8.x | Replace: move data with a tested export and import |
| Cache and broker | Redis container for caching, async queries and Celery | External Redis or Valkey reachable from the cluster | Replace: point configuration at the new service |
| Celery worker and beat | Separate worker and beat services sharing the image | Separate Deployments, with beat running as a single replica | Rebuild: confirm one beat instance only |
| superset_config.py | Environment variables and mounted config file | ConfigMap for settings, Secret for keys and passwords | Rebuild: split non-secret and secret values |
| SECRET_KEY | Set in the environment or config file | Same value held in a Secret, unchanged | Carry over: changing it breaks stored encrypted credentials |
| Uploaded files and local volumes | Named or bind-mounted host directories | Persistent volumes or object storage | Replace: confirm what each path holds before cutover |
| Helm chart and operator | Not applicable | Helm chart is deprecated; Kubernetes Operator is alpha with a v1alpha1 API | Flag: pin versions and expect schema changes |
Documented paths for moving the Superset install
Superset has no single documented route from Compose to Kubernetes, so the options differ in maturity and in what they migrate. The cards below cover the operator, the Helm chart, the upgrade steps and the asset export fallback, with the checks to run before you commit to any of them.
Apache Superset Kubernetes Operator
The current development docs point new Kubernetes deployments to the operator, which is built with the Go Operator SDK. It is in alpha with a v1alpha1 API, so custom resource fields may change between releases. Confirm the schema works with the exact Superset release and operator version you pin, and plan to re-validate manifests on every upgrade before they reach production.
Helm chart referenced in 6.1.0 docs
The Superset 6.1.0 documentation describes Kubernetes installs with the official Helm chart, but the current development documentation marks that chart as deprecated. Use it only if you are pinned to 6.1.0 and accept that new features and fixes may not reach it. Record which chart version you deploy so the later move to the operator starts from a known state.
Backup before database upgrade
The upgrade guide says to back up the metadata database before any schema change. The metadata database holds dashboards, charts, users, roles and encrypted connection details, so a Compose volume copy is not enough without a logical or consistent backup. Take the backup from the existing Compose database, verify it restores, and keep it until the new environment has run cleanly.
Schema upgrade and init steps
After the backup, run superset db upgrade and superset init against the metadata database in staging first, as the upgrade guide describes. Check UPDATING.md for breaking changes between your Compose version and the target version before you start. Run these steps as an explicit lifecycle task, not as a side effect of every web pod start.
Dashboard ZIP export as fallback
If the metadata database cannot be moved intact or the operator cannot be adopted, dashboards, charts and datasets can be exported as ZIP files and imported into a fresh instance. This moves assets, not users, roles or the full configuration, and database connection passwords must be re-entered. Test an import on a copy first, because UUID and overwrite behavior determine whether assets duplicate or replace.
How the cutover runs with a parallel environment
The sequence moves the Compose stack onto Kubernetes in a staging copy first, proves the restored metadata and worker behavior on a test hostname, and only then switches traffic and retires Compose.
Inventory the Compose stack and configuration
The platform engineer lists every service in the Compose file, every mounted volume, every environment variable, and the superset_config.py overrides, including SECRET_KEY handling, the metadata database connection, cache settings, and Celery broker and result backend. Each item is marked as carry over, replace with an external service, or retire. The inventory is signed off by the team that owns Superset so nothing is discovered during cutover. Done when every entry has a target and the SECRET_KEY source is confirmed to be the same value in staging.
Build and pin a custom image
The engineering team builds a Superset image that bundles the database drivers your warehouses need, the custom configuration module, and any branding or plugin assets, then tags it with an exact version rather than a floating tag. The image is scanned and stored in a registry your cluster can pull from. Confirm the Superset version you pin against the release notes and UPDATING.md for the target version. Done when the image starts locally with the same configuration the Compose stack used and the drivers connect to a test database.
Provision external metadata database and Redis
The platform team provisions a PostgreSQL or MySQL instance within a supported version range for the metadata database, plus a Redis instance for the caching layer and Celery broker, all reachable from the Kubernetes cluster and not from a Compose container. Credentials are stored as Kubernetes Secrets with least-privilege access. Done when the cluster can connect to both services from a test pod and backups are configured for the metadata database.
Back up and restore the metadata database into staging
Before any change, the operator takes a backup of the production metadata database, as the upgrade guide requires. That backup is restored into the new staging database, and the staging Superset is pointed at it using the same SECRET_KEY, so encrypted connection passwords still decrypt. Done when the restore completes and the team can log in to staging and see the expected dashboards and database connections listed.
Run database upgrade and init in staging
Against the restored staging database, the team runs superset db upgrade and then superset init, following the upgrade guide, and reads UPDATING.md for any breaking changes between the source and target versions. These run as Kubernetes lifecycle jobs so they execute once per release. Done when both commands exit cleanly, the migration log is archived, and a sample of charts renders correctly against a test database connection.
Start the operator with web, worker and beat
The team deploys the Superset custom resource through the Apache Superset Kubernetes Operator, defining the web tier, Celery workers, and beat scheduler against the external database, Redis and the pinned image. Because the operator is in alpha with a v1alpha1 API, the manifest version is recorded and reviewed on every upgrade. Done when all pods are ready, the worker processes an async query and a scheduled test report, and beat triggers a scheduled job on time.
Run a parallel check on a test hostname
The new environment is exposed on a separate test hostname while Compose continues to serve users. Owners log in through the same identity provider, open a defined list of production dashboards, compare row counts and rendered values with Compose, and test alerts, reports, and thumbnails. Any difference is logged and resolved before cutover. Done when the signed-off checklist shows matching results for every dashboard in scope.
Cut over DNS or ingress and retire Compose last
Near the agreed change window, the team freezes writes to the Compose metadata database, takes a final backup, reruns the upgrade steps against the final data, and then switches DNS or the ingress route to the Kubernetes service. Compose is kept running in a stopped state for a rollback period defined by the business owner. It is decommissioned only after post-cutover verification passes and the rollback window closes. Done when users reach the new hostname, the verification checklist is complete, and the Compose stack is removed with its volumes archived.
Which Compose services move to which Kubernetes building blocks
The web application is the Flask and React service that users reach in a browser. In a Compose stack it is usually one container. On Kubernetes it becomes a Deployment behind a Service, and the decisions here concern the image, the replica count and where the SECRET_KEY lives. The architecture documentation lists the application as one of four components, so every other tab depends on how this one is configured to reach them.
- Image ownership: Decide who builds the image, which base image it uses, and how database drivers for your warehouses are added before the first deployment.
- Replica count and rollouts: Plan how many web pods run and how each rolling update is checked before traffic moves to it.
- SECRET_KEY handling: Superset requires a user-specified
SECRET_KEYto start. Store it in a Kubernetes Secret, and plan encryption at rest, because Secrets are stored unencrypted in etcd by default. - Configuration source: Decide whether
superset_config.pyis baked into the image or mounted from a ConfigMap and Secret, and who approves changes.
Before you write the first manifest, confirm that every environment variable in the Compose file has a named home in the cluster.
The metadata database holds Superset's own state, including users, roles, dashboards and charts. The architecture documentation tests PostgreSQL and MySQL for this role and does not recommend the SQLite default or the Compose PostgreSQL container for production. On Kubernetes, point Superset at a database instance that your team operates or that a managed provider runs. The configuration documentation lists supported versions as PostgreSQL 10 to 17 and MySQL 5.7 and 8.x, so confirm your engine and version fall inside that range before you commit.
- Managed or in-cluster: A managed service moves backups, patching and failover to the provider. An in-cluster database keeps the data in your cluster, but your team then owns backups, restores and upgrades.
- Engine and version: Choose PostgreSQL or MySQL within the documented range, and check that your team can support that version for the planned life of the deployment.
- Connectivity and credentials: Decide where the connection string is stored, who can read it, and whether network policy limits database access to Superset pods.
- Move method: Decide whether you will copy the existing database or export and import it. Keep the same
SECRET_KEYso the migrated metadata can still be read.
Ask the database owner for a written backup and restore procedure, and rehearse a restore to a separate instance before the cutover.
The caching layer is optional in the architecture, but alerts and reports, async queries and dashboard thumbnails require it along with the worker components. Most installations use Redis with Celery for these features. A Compose stack may run Redis as a sidecar container. On Kubernetes, you choose between a managed Redis-compatible service, a Valkey deployment or a Redis deployment that your team operates.
- Which features you need: If you use alerts and reports, async queries or thumbnails, the cache and workers are required for them. List those features and their owners before choosing a service.
- Redis or Valkey: Architecture documentation names Redis as the common choice. Confirm with the configuration documentation for your Superset version whether Valkey works in your setup.
- Managed or in-cluster: Decide who patches, persists and monitors the store, and what happens to queued tasks during a restart.
- Network path: Limit access so only Superset web and worker pods can reach the cache, and set the connection values in your configuration file to match.
Write down which features depend on the cache so the cutover plan tests each one individually, including scheduled delivery.
Celery worker and beat processes run the background work that the web application queues, such as alerts and reports. In a Compose stack these are often separate containers started from the same image. On Kubernetes, running workers and beat as separate Deployments lets each be scaled, restarted and monitored on its own. Beat schedules tasks, so settle how many beat instances may run before you set any replica count.
- Separate worker deployments: Run worker pods in their own Deployment so you can scale them against task load without changing the web tier.
- Single beat instance: Plan for one beat process unless the architecture and Celery documentation for your version say otherwise, since duplicate schedulers can enqueue the same job more than once.
- Shared image and configuration: Workers must load the same configuration and
SECRET_KEYas the web pods, so use one image tag and one Secret reference for all three. - Health and restarts: Decide how a crashed worker is detected and how in-flight tasks behave during a rolling update.
Before the build starts, list every scheduled report and alert in the Compose stack so you can confirm each one fires from the new beat deployment during verification.
Ingress replaces the published port or reverse proxy from the Compose file. It routes browser traffic to the web Service and, in most designs, terminates TLS. The ingress controller, certificate issuance and the TLS termination point depend on your cluster standards, so record each one as a decision before the build starts. Superset's routing and cookie behavior behind a proxy must match what users see in the browser.
- Controller and class: Choose the ingress controller your platform team already operates, and the ingress class the Superset route will use.
- TLS termination: Decide whether TLS ends at the ingress or at a load balancer in front of it, and who owns the certificate and its renewal.
- Audience and network path: Decide whether Superset is reachable from the internet, a corporate network or only through a VPN, since that sets network policies and allowed hostnames.
- Headers and cookies: Confirm which forwarded headers and secure cookie settings the web application needs behind the ingress, and test them in staging.
- Cutover routing: Plan how the DNS name moves from the Compose host to the ingress address, and how long the old route stays available as a fallback.
Check the proxy and cookie settings in the configuration documentation before you assign the hostname, and test login from every network path your users will take.
What usually goes wrong during the move
Most failed Compose-to-Kubernetes moves trace back to a few decisions made in the first days: how secrets carry over, how background processes run, and whether the old stack can still be restored. Each card below names one risk and the check that prevents it.
Changed SECRET_KEY locks credentials
Superset encrypts database connection passwords with SECRET_KEY. If the Kubernetes deployment starts with a new or regenerated key, saved connections fail to decrypt and dashboards stop loading data. Copy the exact value from the Compose environment into the new secret store before the first start, and confirm that a saved database connection still tests successfully on the new stack. Keep the old value available for rotation through PREVIOUS_SECRET_KEY if you change it later.
Secrets sit unencrypted in etcd
Kubernetes stores Secrets unencrypted in etcd by default. Superset's SECRET_KEY, metadata database password and cache credentials become readable to anyone with broad read access to the cluster's data store. Before you load the migrated values, enable encryption at rest for the cluster, restrict list and watch permissions on Secrets with least-privilege RBAC, or pull values from an external store through the Secrets Store CSI Driver. Ask your platform team who can read Secrets in the target namespace.
Beat runs in duplicate across replicas
Celery beat schedules alerts, reports and cache warm-up tasks. If you scale the beat deployment above one replica, each schedule fires once per pod, so the same report can be sent several times and workers receive duplicate jobs. Run beat as a single replica with a restart policy that does not allow two copies, and keep web and worker pods scalable. Verify after cutover by checking that each scheduled report arrives once across a full schedule cycle.
Stale image missing database drivers
A custom image built for Compose often carries drivers that were installed by hand in a running container. A Kubernetes image built from the stock base will not have them, so connections to warehouses fail with driver import errors only after the move. Build the image from a pinned Dockerfile that installs every driver your dashboards use, and test each database connection from a pod running that exact image before routing users to it.
Ingress and cookie mismatches break sign-in
Session cookies, redirect URLs and the external hostname must match the new ingress. If the host, TLS termination or path prefix changes, users may loop at login, OAuth callbacks may fail against the registered redirect URI, or the browser may reject cookies marked secure. Compare the public URL and proxy headers between the two stacks, update the identity provider's redirect list before cutover, and test sign-in from a clean browser session on the new hostname.
Cutover with no rollback path
Running a one-time migration of the metadata database and switching DNS leaves no easy way back if dashboards or permissions are wrong. Take a verified backup of the metadata database and keep the Compose stack stopped, not deleted, until the new environment passes checks. Write down the exact steps to repoint DNS and restart the old services, and time a rehearsal in staging. Agree with stakeholders who signs off the cutover and how long the old stack stays available.
What makes the Kubernetes move longer or costlier
This table lists the factors that most often stretch a Compose-to-Kubernetes Superset move and shows whether each one is paid once or recurs. Read each row against your own environment, then use the last column to write the questions to put in a quote request. The main takeaway is that the largest drivers are usually the ones your team cannot see until discovery, such as undocumented database drivers and identity rules.
| Driver | Why it changes effort or cost | What to ask for in a quote |
|---|---|---|
| Database connections and drivers | Each database driver must be added to the custom image and tested against its warehouse, so the count of connections sets build and test work. | A list of drivers to bake into the image, with a test plan for each connection and who owns the credentials. |
| Existing cluster and etcd backups | If the cluster, etcd backups and restore procedure already exist, the move reuses them. If not, platform setup becomes part of the project. | A statement of which cluster, backup and monitoring pieces are in scope and which the client already runs. |
| Identity integration complexity | OAuth2, LDAP and group-to-role mapping (AUTH_ROLES_MAPPING) take design and testing against your identity provider, and each rule adds review time. | The identity provider, protocol, number of group mappings and whether mappings change after go-live. |
| Scheduled reports and alerts needing workers | Celery workers and beat run reports, alerts and thumbnails. Higher report volume means more worker capacity, queue tuning and load testing, and that capacity keeps running after cutover. | The expected report and alert volume, queue layout, and who sizes and monitors workers after handover. |
| High availability requirements | Running several web and worker replicas, a managed PostgreSQL metadata database and a managed Redis or Valkey cache adds components to design, secure and back up. | Whether HA is required for each component, the failover tests expected and the recovery targets you will accept in writing. |
| Number of environments | Each environment (development, staging, production) needs its own build, configuration and secrets, and staging should match production closely enough to rehearse the cutover. | How many environments are in scope, whether staging must mirror production, and how promotion between them is handled. |
| Operator alpha risk | The Superset Kubernetes Operator is in alpha with a v1alpha1 API whose schemas may change between releases. Upgrades to the operator can require rework, so this risk recurs. | Who tracks operator releases, how upgrades are tested, and whether the quote covers rework if a schema changes. |
Questions buyers ask before moving Superset to Kubernetes
What does moving Superset from Docker Compose to Kubernetes cost?
The cost is driven by how much of the platform work your team can absorb, not by a fixed package. You pay for discovery of the current Compose stack, image build and registry work, externalizing the metadata database and cache, worker and beat configuration, ingress, testing in staging and the cutover itself. Some costs are one-time, such as the migration and validation. Others recur, such as cluster capacity, backups, patching and Superset upgrades. Ask any partner for a written scope that lists each of these items separately, the environments covered, and what is excluded, so you can compare it with your own internal effort. Superset itself is open source under the Apache License 2.0 with no paid edition, so the license is not a cost driver.
How long does a Compose to Kubernetes migration take?
Duration depends on how many components are moving, how many environments need a staging rehearsal, and how much your cluster already provides. The main drivers are whether the metadata database must be moved to a managed or external PostgreSQL or MySQL instance, whether Redis and Celery workers are already in place for alerts, reports, async queries and thumbnails, the state of your identity and ingress setup, and how many change windows your organization allows. A Compose stack with a simple PostgreSQL container and no workers is a shorter job than one with Celery beat schedules and custom images. Ask for a timeline tied to a written scope, and confirm the staging rehearsal count before work starts.
Can our own team do this migration without a partner?
Yes, if your team has the skills and time to cover the areas that usually cause problems. The work needs Kubernetes operations experience, container image build practice, PostgreSQL or MySQL administration, Redis administration and Superset configuration knowledge, including the SECRET_KEY requirement and the upgrade steps in the project documentation. The Helm chart is documented for Kubernetes installs in the 6.1.0 docs, but the current development documentation marks it as deprecated and points new deployments to the Apache Superset Kubernetes Operator, which is in alpha with a v1alpha1 API that may change. A partner is useful when those skills are thin, when you need a fixed-scope delivery with written deliverables, or when you want a second team to rehearse the cutover.
What access and decisions does the migration require from our team?
Plan for access and decisions before the work starts, because they set the schedule. You will need a non-production and production Kubernetes namespace or cluster, registry credentials for your built image, read access to the current Compose files, environment files and volumes, and a database account for the new metadata database with a tested backup. Secrets such as SECRET_KEY, database passwords, OAuth client secrets and warehouse connection credentials must be handed over through a controlled channel. Kubernetes stores Secrets unencrypted in etcd by default, so confirm encryption at rest and least-privilege RBAC for the namespace. You will also need named change windows, a rollback owner and sign-off on DNS or ingress changes.
Will our dashboards, charts, datasets and users survive the cutover?
They survive when the metadata database is moved with a verified copy and the same SECRET_KEY is used, because Superset stores dashboards, charts, datasets, roles and connection records in the metadata database. Database connection passwords are encrypted with the secret key, so losing or changing it can make saved connections unusable. Verify the result by comparing counts of dashboards, charts, datasets, users and roles before and after, opening a sample of each in staging, and running alerts and reports through the workers. Keep the Compose stack available for a rollback window and freeze edits on the old instance during the final copy. Ask your partner to document this verification as a cutover checklist you can run again.
Should we use the Helm chart or the Kubernetes Operator?
For a new Kubernetes deployment, the current development documentation directs you to the Apache Superset Kubernetes Operator, because the Helm chart is marked as deprecated there. The Helm chart is still described in the 6.1.0 documentation, so teams already on a supported version may need it for a transition period. The operator is built with the Go Operator SDK and is in alpha, so its custom resource schema may change between releases, which affects how much you should invest in custom manifests and how you plan upgrades. Confirm the current status in the project documentation, test the chosen path in staging, and record the decision and the version you pinned.
What goes wrong most often during a Compose to Kubernetes cutover?
The most common failures come from configuration that existed only in the Compose files or in the container's local state. Typical examples include a SECRET_KEY that was regenerated, a metadata database still on the Compose PostgreSQL container that is not recommended for production, Celery workers or beat not started, so scheduled reports silently stop, a cache not reachable from the new pods, and ingress routing that breaks sessions or static assets. Superset requires a user-specified SECRET_KEY to start, and upgrade steps call for a metadata database backup, superset db upgrade and superset init, and a check of UPDATING.md. Test each of these in staging, and keep a written rollback path for each cutover step.
Official sources for the migration
Start with the Superset architecture documentation, because it defines the four components you will split apart: the Flask and React application, the metadata database, the caching layer, and the Celery worker and beat processes. Read it alongside the Installing on Kubernetes page in the current docs, which marks the Helm chart as deprecated and points new deployments to the Apache Superset Kubernetes Operator. Confirm with your team which documentation version matches the Superset release you plan to run, since the 6.1.0 docs still describe the Helm chart as the Kubernetes route. The Superset Kubernetes Operator documentation should be your reference for the custom resource fields, and note that its v1alpha1 API is alpha and its schemas may change between releases.
The configuring Superset documentation covers the settings you must carry over from Compose: SECRET_KEY, the metadata database URI, Redis and Celery broker settings, and PREVIOUS_SECRET_KEY for rotation. Check the supported metadata database versions listed there before you choose a managed PostgreSQL instance, and confirm the exact SECRET_KEY handling with the version you run. The upgrading Superset guide defines the backup, superset db upgrade, superset init and staging sequence you should rehearse during the cutover, and points you to UPDATING.md for breaking changes.
On the platform side, the Kubernetes secrets good practices page explains that Secrets are stored unencrypted in etcd by default, so you should confirm encryption at rest and least-privilege RBAC before you load the SECRET_KEY and database credentials. The Kubernetes etcd operations page covers the etcdctl snapshot save and etcdutl restore steps you need before the cutover, since a restore of the cluster state is only useful if you have tested it. Treat these pages as the baseline your platform team signs off on, and record which version of each page you used so the cutover runbook can be audited later.
How DeployOpen works on this
DeployOpen plans the move from a Compose-based Superset to Kubernetes as a written, fixed-scope engagement inside your own infrastructure. We start from your running configuration and data, then build, test and cut over in stages so dashboards, users and scheduled reports keep working throughout.
How we scope
- Inventory the running Compose stack, its SECRET_KEY handling, database connections and volumes before any design starts.
- Agree in writing which metadata database, cache and worker components move, and which stay as they are.
- Confirm the target cluster, ingress path, Superset version and the Helm or Operator route before the build is priced.
How we build
- Build a Superset image with your extensions and configuration, and manage it through your own registry and pipeline.
- Externalize the metadata database and cache, then configure Celery workers and beat for alerts, reports and async queries.
- Test the upgraded stack in staging, including a restore from a metadata database backup taken from the Compose environment.
How we hand over
- Deliver a cutover runbook with a freeze window, backup checkpoints, dashboard and user verification steps and a rollback path.
- Document the Kubernetes manifests or Operator resources, secret locations, configuration overrides and upgrade steps.
- Walk your platform and security teams through operations, so they can run the cluster and remove the old Compose stack confidently.
How we support
- Support after cutover can be agreed as a retainer covering patching, upgrades and monitoring of the Superset stack.
- Plan Superset upgrades with a backup, staging test and review of breaking changes before each move to a new version.
- Review the Operator's alpha status and API changes with you before each upgrade of the custom resource definitions.
