When a single web process is enough and when Celery workers become necessary
A team deciding how many Superset processes to run can answer the question from the feature list the business actually needs. A web application plus a production metadata database is enough for interactive dashboards, SQL Lab exploration and chart building, where a user clicks, waits for results and moves on. The architecture documentation lists the Flask and React application, a metadata database, an optional caching layer and optional Celery worker and beat processes as separate components, so the single-process option means accepting that the optional pieces are absent. Start with the metadata database choice, since the documentation recommends PostgreSQL or MySQL in production and does not recommend the SQLite default for production use.
Scheduled reports, alerts, async queries and dashboard thumbnails are the features that change the answer. According to the architecture documentation, these require the caching layer and worker components, and most installations use Redis and Celery for them. If stakeholders have asked for emailed or Slack-delivered reports, threshold alerts, or long-running queries that should not hold a browser session open, plan for celery worker and celery beat processes, a Redis broker and cache, and monitoring for each. Confirm the exact worker and beat commands, broker settings and cache configuration against the current Superset architecture documentation and the configuration reference for your release line.
Write the requirement down before choosing a topology. List each report, alert and async workflow, its owner, its delivery channel and how often it runs, then count the services that list needs. A single web process with no background features is simpler to size and upgrade, but moving to workers later means adding Redis, configuring Celery and testing thumbnails and alerts against your metadata database under load. Ask your implementation partner or internal platform team to confirm how the worker processes will be monitored and restarted, because a dashboard that loads while its scheduled reports silently stop is the failure most likely to reach stakeholders first.
Which shape fits which team situation
If analysts open existing dashboards, build charts and run queries in SQL Lab that return synchronously, the architecture can stay small. The Superset architecture documentation lists four components: the Flask and React application, a metadata database, an optional caching layer and optional Celery worker and beat processes. For interactive use, the web application and a supported metadata database are the core pieces, and the cache and workers are optional.
- Required components: the web application and a PostgreSQL or MySQL metadata database, which the documentation says is tested for production use.
- Optional components: the caching layer and Celery workers can be left out while no feature depends on them.
- Metadata database: the SQLite default of Quickstart and PyPI installs is not recommended for production, so plan for a real database from the start.
- Scope limit: the documented dependency means scheduled deliveries and background jobs cannot be added to this shape without changing it.
- Upgrade effort: fewer moving parts means fewer services to back up and patch during the documented upgrade steps.
Before committing, confirm with your own usage that no team will need alerts, scheduled reports or asynchronous queries in the next upgrade cycle.
Scheduled reporting and alerting need the worker components. The Superset architecture documentation says alerts and reports require the caching layer and worker components, and most installations use Redis and Celery for them. A web process alone cannot evaluate schedules or send deliveries on time, so the extra services are part of the design rather than an add-on.
- Redis: serves as the caching layer and the broker that the Celery processes depend on.
- Celery worker: runs the jobs that execute reports and alert checks.
- Celery beat: triggers scheduled runs at their configured times, so it must run as its own process.
- Outbound delivery: the host running workers needs network paths to mail or messaging endpoints, which should be confirmed with your mail or chat administrators.
- Monitoring: a stopped beat process silently halts schedules, so add health checks for each worker and beat process.
Before committing, confirm the exact Redis and Celery settings for your Superset version in the configuration documentation and test a scheduled delivery in staging.
When analysts run long queries that should not block the browser session, asynchronous execution in SQL Lab depends on the worker components. The architecture documentation ties async queries to the caching layer and workers, so a team that expects multi-minute warehouse queries should plan for them early. The shape of this workload also determines where the cost of running the extra services lands.
- Async execution: queries are handed to workers, which means worker capacity and queue health affect how long analysts wait.
- Result storage: the caching layer holds results between the worker and the browser, so Redis memory and retention settings need review.
- Separate worker pools: heavy query work can be isolated from scheduled reports, though the split is a design choice to confirm for your version.
- Sizing: no sizing figures are supplied here, so ask your DeployOpen or internal architect to test expected concurrency in staging against your own warehouse.
- Database limits: the source warehouse, not Superset, often sets the real query ceiling, so check its concurrency limits too.
Before committing, measure worker queue behavior with a realistic set of concurrent queries from your own analysts.
Embedded dashboards shown to customers combine interactive use with the same dependencies as any production deployment. The worker components matter if the embedded views rely on asynchronous queries, cached results or scheduled refreshes, and dashboard thumbnails in listings depend on workers as well. Access control and identity integration also change the design, because the web tier must sit in front of customer traffic.
- Identity: OAuth2 and LDAP group mapping through AUTH_ROLES_MAPPING affect who can reach which dashboards, and the design should be checked against the configuration documentation.
- Secret management: a user-specified SECRET_KEY is required at startup, so it must be stored and rotated under your key management process.
- Thumbnails: dashboard thumbnails depend on the caching layer and workers, which matters when the customer-facing UI shows previews.
- Availability: a web process alone has one failure point for customer traffic, so plan redundancy in your own infrastructure.
- Licensing: Superset has no paid edition, but embedding design and tenant isolation should be reviewed against your product requirements.
Before committing, confirm with your architect which embedding features and thumbnails your release will expose and test them with the full worker stack.
Web-only and web plus Celery compared
This table compares a Superset install that runs only the web application and metadata database against one that adds a caching layer and Celery worker and beat processes. Read each row as the question to answer for your own deployment, and treat the web-only column as fine for evaluation and light use only where your features do not need workers.
| Factor | Web application and metadata database only | Web plus Redis and Celery workers |
|---|---|---|
| Components to run | Flask and React application and a metadata database | Application, metadata database, Redis caching layer, Celery worker and Celery beat |
| Features available | Interactive dashboards and SQL Lab queries; no scheduled alerts, reports or thumbnails | Adds async queries, alerts and reports, and dashboard thumbnails |
| Metadata database requirement | PostgreSQL or MySQL for production; SQLite default is not recommended for production | Same PostgreSQL or MySQL requirement, shared by web and workers |
| Caching layer | Optional; query results are not cached by a dedicated service | Required for the worker-dependent features; most installs use Redis |
| Monitoring surface | Web process health, database health and application logs | Adds Redis health, worker and beat process health, queue depth and scheduled task runs |
| Sizing inputs | Concurrent dashboard viewers and query load on the database | Adds queue volume, report frequency, worker concurrency and thumbnail rendering load |
| Upgrade steps | Back up metadata database, run database upgrade and init, test in staging | Same steps, plus restart and version-match workers and beat with the web application |
| If a worker is down | Not applicable; no worker-dependent features exist | Async queries stall, scheduled alerts and reports are missed, and thumbnails do not refresh |
| Who should confirm | Platform team checks the chosen features against the Superset architecture documentation | Platform team confirms Redis and Celery sizing, high availability and alerting with the vendor docs and their own environment |
The extra services a Celery deployment must run and monitor
A single web process needs the Superset application and a metadata database. Once you add async queries, alerts and reports, or dashboard thumbnails, the caching layer and Celery workers join the stack, and each one fails in its own way. The cards below cover each component, what breaks visibly when it stops, and what to watch.
Redis as cache and broker
Redis serves two roles in a Celery setup: a caching layer for query results and a message broker that queues tasks for the workers. The Superset architecture documentation says most installations use Redis and Celery for async queries, alerts and reports, and thumbnails. If Redis stops, cached dashboards reload slowly, and queued work stops moving. Monitor memory use, eviction behavior, connection counts and persistence settings, and confirm with your team whether the broker and the cache share one instance.
Celery worker processes
Workers run the background jobs: long SQL queries submitted asynchronously, scheduled alert checks, report delivery and thumbnail generation. When no worker is running, these features look configured but never complete, so users see spinners or missing reports rather than a clear error. Monitor queue depth, task failure rates, worker memory and task duration. Size the worker count against your heaviest query and report schedules, and test each job type after every upgrade.
Celery beat scheduler
Beat is the scheduler that triggers recurring work, such as alert evaluations and scheduled report sends, and places those tasks on the queue. It is a separate process from the workers. If beat stops, workers stay healthy and idle, and scheduled alerts and reports silently stop firing. Monitor that beat is running, that it has a single active instance, and that scheduled tasks appear on the queue at their expected times. Keep a check that alerts still fire on a known test condition.
The Superset web process
The web process serves the Flask and React application, dashboards and SQL Lab. It also enqueues work for the workers, so a worker outage often shows up first as a web symptom, such as a query that never returns. Monitor HTTP health, response times, error rates and the SECRET_KEY configuration that must be set before startup. Confirm how many web replicas you run and whether they share one configuration.
The production metadata database
Superset stores dashboards, charts, users, roles and scheduled jobs in its metadata database. The architecture documentation says PostgreSQL and MySQL are the tested options and that the SQLite default and the Docker Compose PostgreSQL container are not recommended for production. Monitor connection limits, storage growth, replication or backup status and the time of the last successful backup. Confirm the supported version range in the configuration documentation before you provision.
Monitoring and log stack
Each extra component writes its own logs, and a Celery failure often leaves no trace in the web logs. Plan central log collection across the web process, workers, beat and Redis, with alerts on task failures, queue backlog and missing scheduled runs. Ask your team who owns on-call for each component and which dashboards show queue depth. The monitoring stack is also one more service you must back up, patch and size.
How Celery changes sizing and upgrade planning
Adding Celery workers and beat turns a single Superset deployment into several processes that must be sized and upgraded as one system. Each worker process pulls jobs from the Redis broker and runs async queries, alert and report jobs, and thumbnail generation, so its CPU, memory and database connection use add to the load on the metadata database and the caching layer. Confirm in your own environment how many worker processes you need for your expected alert, report and query volume, and measure connection counts on the metadata database after you add them. The beat process must run as a single scheduler, because two beat processes would trigger scheduled alerts and reports twice. Check that your process manager, Kubernetes deployment or Compose file runs exactly one beat instance and that a restart policy will not start a second one during a rolling update.
Upgrades must cover the web process and every worker together, because they share the same codebase, configuration and metadata database. Follow the documented sequence: back up the metadata database, run superset db upgrade and superset init, and test the upgrade in staging first. Read UPDATING.md for breaking changes between your current and target versions before you schedule the change, and plan for the web and worker images to be replaced in the same window. The Superset upgrade documentation covers these steps, and the Superset upgrading guide is the natural next read for the exact order.
Before any upgrade or secret rotation, confirm three items: that SECRET_KEY is set explicitly in every process's configuration, since Superset requires a user-specified key to start; that you know how PREVIOUS_SECRET_KEY will be used to rotate the key without breaking existing encrypted data; and that your staging environment uses the same Redis, Celery and metadata database versions as production. Also confirm the supported metadata database versions for your target release in the configuration documentation, and the version of Redis you plan to run, since that is a dependency your team will now own.
Upgrades, backups, monitoring and skills for each shape
This table compares what each Superset shape asks of the team on upgrades, backups, monitoring, on-call skills and failure modes, using the steps Superset documents. Read it as a list of operating duties: the single web process keeps them to the web app and metadata database, while the Celery shape adds Redis and worker and beat processes to every duty. The main takeaway is that the Celery shape roughly doubles the surfaces the team must patch, back up and watch, even though the upgrade steps stay the same.
| Area | Web process only | Web plus Redis and Celery |
|---|---|---|
| Upgrade steps | Back up the metadata database, run superset db upgrade and superset init, then test in staging | Same documented steps, plus stopping and restarting worker and beat processes so they run the same Superset version as the web app |
| Breaking changes | Check UPDATING.md for changes between versions before upgrading | Check UPDATING.md too, and confirm that task and cache configuration still matches the new release |
| Backup scope | Metadata database, including dashboards, charts, datasets, users and roles, plus the SECRET_KEY | Metadata database and SECRET_KEY, plus Redis if it holds state you need to keep; cached results can usually be rebuilt |
| Monitoring | Web process health, metadata database connections and query latency | Adds Redis availability, Celery worker count and queue backlog, and beat schedule execution for alerts and reports |
| Failure modes | Dashboards and SQL Lab fail or slow down when the web process or metadata database fails | Async queries, alerts, reports and thumbnails can stall while the web UI still loads, so a silent worker outage may go unnoticed |
| On-call skills | Flask application deployment, database administration and reverse proxy troubleshooting | Adds Redis operations, Celery worker and beat troubleshooting, and reading task logs alongside the web stack |
| What the team owns | Web containers or processes, metadata database, configuration and backups | All of the single shape, plus Redis, worker and beat process lifecycle, scaling and their monitoring |
| Confirm with the vendor | Metadata database versions supported for your Superset release | Redis and Celery versions that your Superset release documents, and the worker configuration for your deployment method |
Questions teams ask before running Celery workers
Can a web-only Superset send scheduled reports?
No. According to the Superset architecture documentation, alerts and reports, async queries and dashboard thumbnails require the caching layer and the worker components. A web-only install runs the Flask and React application with a metadata database, so scheduled reports and alerts depend on adding Redis and Celery worker and beat processes. If your plan includes scheduled delivery, treat the workers as part of the base design, not an optional add-on. Confirm the exact configuration for your chosen Superset version in the Superset architecture documentation before you commit.
How much extra effort does the Celery setup add?
It adds services, not just configuration. Besides the web application and metadata database, you run a cache broker (most installations use Redis), one or more Celery worker processes and a Celery beat process for scheduled tasks. Each one needs its own process supervision, health checks, logs, resource limits and restart behavior. Worker sizing also matters because report rendering and thumbnail generation can be resource-heavy. Expect more work in monitoring, backup scope and upgrade testing. The exact effort depends on your platform (Kubernetes, Docker Compose or a PyPI install) and on how many dashboards and recipients you serve, so ask for the component list and monitoring plan in any quote.
Can our team run Celery workers without outside help?
It depends on your team's experience with Python services, Redis, and container or Kubernetes operations. The Superset documentation gives you the architecture, configuration keys and upgrade steps, but it does not replace on-call readiness for queues that can back up silently. Teams with a platform group that already runs Redis and Python workers usually have the skills in-house. Teams without those skills should either train staff or consider outside help for the initial build and runbooks. DeployOpen plans, implements, hardens, documents and supports self-hosted platforms inside your environment, and most engagements are fixed-scope with the scope agreed in writing before work starts.
How long does a move from web-only to Celery take?
The timeline depends on scope, and it must be quoted against a written scope. The main variables are the deployment method, whether Redis is already available, the number of alerts and reports to migrate, how many environments need the change, and how much testing your staging process requires. The Superset upgrade documentation recommends backing up the metadata database, running the database upgrade and initialization commands, and testing in staging first, so plan for staged validation rather than a single cutover. Ask any provider, including DeployOpen, for a fixed-scope proposal with deliverables and dates listed.
What does the work need from our side?
You need to supply access and decisions. Typical inputs include a target environment (cloud account, data center or air-gapped network), a metadata database you are willing to run in production (the Superset documentation tests PostgreSQL and MySQL and does not recommend the SQLite default for production), a secrets process for SECRET_KEY, and named owners for alerts, report recipients and on-call escalation. You also need a staging environment that matches production closely enough to test scheduled delivery, and a list of the dashboards and datasets that will produce reports. Confirm any SMTP or messaging channel your reports use, since those depend on your own mail or chat systems.
What drives the cost of running Celery workers?
Cost comes from three places: people, infrastructure and any outside services. Superset itself is Apache-2.0 with no paid edition, so license fees are not the driver. Infrastructure cost rises with the number of worker and beat processes, the Redis footprint, the metadata database size, and the compute needed for report rendering and thumbnails. Ongoing cost includes monitoring, on-call time, upgrades and staging environments. One-time cost covers the initial build, documentation and testing. Ask any quote to itemize these components and to state which parts recur, such as support retainers or upgrade work.
Moving from a web-only install to a Celery deployment
This sequence takes a running web-only Superset to a deployment with Redis, Celery workers and beat, validated against the features that depend on them, with a rollback path agreed before the cutover.
Confirm which features need workers
The team lists the features it intends to use, such as async SQL Lab queries, alerts and reports, and dashboard thumbnails. The Superset architecture documentation says alerts and reports, async queries and dashboard thumbnails require the caching layer and worker components, so any of these on the list means Redis and Celery are in scope. Ask the platform owner to write the list down and mark which features are optional. The output is a signed-off feature list that drives every later step.
Provision Redis and a production metadata database
The platform team provisions Redis for the caching layer and Celery broker, and moves the metadata database off the SQLite default. The architecture documentation says SQLite and the Docker Compose PostgreSQL container are not recommended for production, and that PostgreSQL and MySQL are the tested options. Confirm the exact versions against the configuration documentation, which lists PostgreSQL 10 to 17 and MySQL 5.7 and 8.x. Take a backup of the metadata database before any migration, and confirm that restore has been tested. The output is a reachable production database and a verified backup.
Migrate the metadata database and secrets
The team copies existing metadata into the new database and confirms the SECRET_KEY set in the current install is carried over unchanged. The configuration documentation says Superset requires a user-specified SECRET_KEY to start, and supports rotation through PREVIOUS_SECRET_KEY, so a changed key can break access to stored credentials. Run the upgrade steps from the upgrade documentation, including superset db upgrade and superset init, in a staging copy first. The output is a staging instance that loads with existing dashboards, users and data sources intact.
Deploy Celery workers and beat
The platform team starts the Celery worker and beat processes against the same Superset configuration and the Redis broker, using the same application image or code version as the web process. Beat schedules alerts and reports, so it must run as a single instance that the team monitors. Confirm process health through the team's supervisor or orchestrator, and confirm each worker can reach the metadata database and the cache. The output is running worker and beat processes with logs that show a successful connection to Redis.
Validate alerts, reports and thumbnails in staging
Testers create one async SQL Lab query, one scheduled report and one alert in staging, then confirm each completes and that dashboard thumbnails render. Each check should be run against the features on the signed-off list, and failures should be traced to the worker, the broker or the web process. Confirm delivery channels such as email or messaging work from the worker host, since outbound network rules often differ from the web tier. The output is a dated test record showing each feature passed or has a documented fix.
Rehearse the cutover and rollback
The team writes the cutover runbook: the freeze window, the final metadata backup, the order in which web, workers and beat are switched, and the steps to return to the web-only install. The rollback should be tested in staging by restoring the pre-cutover backup and confirming the earlier version still serves dashboards. Check UPDATING.md for breaking changes between the installed and target versions before the rehearsal. The output is a runbook with a rehearsed rollback that the on-call owner has read.
Cut over and monitor the new processes
The team applies the runbook in production, starts workers and beat, and checks that scheduled reports and alerts run on their next cycle. Monitoring should cover queue depth on Redis, worker restarts, beat health and the metadata database connection count, with alerts routed to the platform owner. Keep the rollback path available until the team has seen at least one full cycle of scheduled jobs succeed, and record the result. The output is a production deployment with monitoring active and a signed-off decision to close the rollback window.
When to drop workers and keep the web process only
A web-only Superset is a sound choice when the business needs interactive dashboards and SQL Lab, and nothing else. In that setup the deployment is the Flask and React application plus its metadata database, with the caching layer and Celery worker and beat processes left out. The architecture documentation ties async queries, alerts and reports, and dashboard thumbnails to the caching layer and workers, so if none of those features are in use, there is no need to run Redis or Celery at all. Fewer services mean fewer failure points, fewer components to patch, and a smaller upgrade surface. Before you commit, confirm which features your users and owners rely on, and check the Superset architecture documentation against that list.
The trade-off is that scheduled delivery and alerting move somewhere else. If a weekly PDF to finance, a threshold alert on a revenue table, or a thumbnail-driven dashboard gallery is part of the requirement, a web-only design cannot cover it, and you will need another scheduler or alerting tool and a clear owner for it. Ask each dashboard owner whether any chart is sent on a schedule, triggers a notification, or relies on cached async results, and record the answers before you decide. Also remember that the metadata database remains a single dependency in either design, and the supported versions and SECRET_KEY requirement apply regardless of worker choice.
Keep the decision reversible. Adding workers later means provisioning Redis, configuring the cache and worker settings, and running the upgrade steps on a larger set of components, so plan that work as a deliberate change rather than a quick toggle. Test the web-only build in staging with a representative set of dashboards, then review the findings with the teams who own the reports, so that any scheduled delivery you move to another tool has a named owner and an acceptance check.
How DeployOpen scopes and runs Superset deployments
DeployOpen plans, implements, hardens, backs up, upgrades, documents and supports self-hosted open-source platforms inside the client's own infrastructure. For Apache Superset, that means deciding in writing whether the deployment stays a web process with a metadata database or adds Redis, Celery workers and beat, then building and handing over the result with the operating procedures to match.
How we scope
- Confirm which features need workers: async queries, alerts and reports, and dashboard thumbnails.
- List the metadata database, caching layer, worker processes and monitoring each option adds.
- Agree the scope, deliverables and timelines in writing before any work starts, as engagements are fixed-scope.
- Record the Superset version, the metadata database version and the target hosting environment in the scope.
How we build
- Deploy Superset inside your cloud account, data center or air-gapped network with the chosen process layout.
- Configure SECRET_KEY, identity integration and role mapping as documented for your environment.
- Test async queries, scheduled reports and thumbnails in staging before anything reaches production users.
- Run security testing of the deployment as a separate, agreed activity where the scope includes it.
How we hand over
- Document the process layout, service dependencies, queue and worker health checks, and restart order.
- Provide the upgrade runbook covering metadata database backups, superset db upgrade, superset init and UPDATING.md review.
- Walk your operations team through monitoring for Redis, workers and beat, with the alerts agreed in scope.
- Deliver configuration and evidence for compliance assessments, noting that auditors and certification bodies issue any attestations.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Retainers can cover Superset release reviews, staging upgrades and worker queue maintenance on your schedule.
- Ask for the support scope, response arrangements and upgrade windows in the written quote before committing.
- Quotes follow from the written scope, so any change to the worker design is priced as a scope change.
