Why Apache Superset async queries with Celery matter for heavy SQL
Async queries move long SQL Lab executions off the web tier and onto Celery workers, so a heavy query no longer holds a web worker open until the browser gives up. The results are written to a results backend and the analyst picks them up when they are ready. This matters most for analysts who run wide scans or large aggregations against a warehouse, where a query that takes several minutes will regularly outlast a browser session or a proxy timeout on a synchronous request. The Superset architecture documentation describes four components: the Flask and React application, a metadata database, an optional caching layer, and optional Celery worker and beat processes. It lists the caching layer and workers as required for async queries, alongside alerts, reports and dashboard thumbnails, and notes that most installations use Redis and Celery for them. Async execution is therefore not a switch you flip on the web app alone. It is a second tier of infrastructure that you must size, secure and monitor.
Superset is an *Apache Software Foundation* project released under the Apache License 2.0, with no paid edition, so the cost of this tier is the infrastructure and the people who run it, not a license. Before you commit, confirm with your team which broker and results backend you will operate, who owns the worker pool, and how long a query may run before it is stopped. Those three answers shape the worker sizing, timeouts and user experience covered in the sections that follow.
Choosing the broker and results backend
The architecture documentation states that async queries, alerts and reports, and dashboard thumbnails require the caching layer and worker components, and that most installations use Redis and Celery for them. Redis is the broker most teams start with because it is widely deployed and familiar to platform operators. In a single-site rollout, one Redis service can carry queued query messages for the Celery workers that run SQL Lab queries. Before you adopt it, decide how the instance will be persisted, monitored and patched, since queued work depends on it.
- Single instance: one Redis service is usually enough for a pilot or a single-site rollout with modest query volume.
- Separate roles: consider a broker instance separate from the cache so that cache sizing and eviction settings do not interact with queued tasks.
- Network placement: allow connections only from the web tier and the workers, over a private network.
- Authentication and encryption: set a password and enable transport encryption where your Redis version and Celery client support it.
- Restart behavior: test what happens to in-flight queries when the broker restarts.
Confirm the Redis version, persistence settings and high availability features against your Redis distribution and the Celery broker documentation before you commit to the design.
Celery can use more than one broker, but the options Superset accepts are set by its configuration and by the Celery version the release bundles. Do not assume that a broker Celery supports will work with Superset without checking. RabbitMQ is the option teams most often compare with Redis, and some organizations already operate it with the durability and monitoring practices they want. Choosing it adds a service to your operating model, and its high availability model differs from Redis, so the failover tests change too.
- Confirm support first: check the broker connection settings in the configuration documentation before choosing anything other than Redis.
- RabbitMQ: a service your team already runs, backs up and staffs may outweigh a technical preference for Redis.
- Managed brokers: a managed service can reduce patching work, but you must confirm the connection string format, TLS support and the failover behavior the provider documents.
- Platform standards: your existing monitoring, alerting and on-call practices often decide the broker more than benchmark results do.
- Load testing: run a test with a representative mix of long and short queries on the chosen broker before the first user onboarding.
Ask the vendor or your Superset release notes which broker versions and Celery versions your Superset version has been tested with.
The broker moves task messages between the web tier and the workers, while the results backend holds the output that the web tier retrieves once a query finishes. The architecture documentation describes the caching layer and workers as the components async queries need, but the exact results storage options are defined in the configuration documentation for your version. Query results can contain sensitive business data, so the results store belongs inside your data classification and retention policy, not only in the operations runbook.
- Shared reachability: the web tier and every worker that can finish a query must read and write the same results store. A path that only one host can read will fail once the web tier runs on another node.
- Retention: set expiry so stored results do not outlive your query history and data handling rules.
- Redis as results store: many installations reuse Redis here, so use a separate database or instance to keep queued tasks and stored output apart.
- Access control: decide who can read stored results at the storage layer, since database users with direct access may see them.
- Cleanup: confirm how expired results are removed and whether that process runs without manual steps.
Check the results backend options in the configuration documentation for your version, and confirm the access rules in writing with your security team.
A single-site deployment with one broker and one worker host is simple to build and easy to troubleshoot, but a host failure stops every async query until it is restored. A high-availability design adds redundancy at each layer, and it adds configuration that you must test. The architecture documentation notes that alerts and reports depend on Celery beat as well as workers, so the scheduler needs the same planning as the queue. Multi-site designs raise questions about latency and data residency that a single site does not.
- Single node: acceptable for a pilot, with the understanding that a single host failure interrupts async queries and scheduled work.
- Single site, HA: a replicated broker with failover, at least two worker hosts and shared results storage. Confirm the reconnection and failover settings with the Celery documentation.
- Beat scheduling: plan how you prevent duplicate or missed scheduled runs when beat is restarted or moved during a failover.
- Multi-site: confirm latency, data residency and the failover model before spreading one queue across sites.
- Failure tests: write a test plan that records the user-visible effect of each broker, worker and storage failure.
Ask your team to document the expected behavior of each failure before go-live, since that record is what operators will use during an incident.
Configuration options and when each fits
This table compares the main Celery and SQL Lab settings that decide how Superset moves long queries onto workers. Read each row as a decision: the setting, when it fits, what it costs you, and the value to confirm for your version. The main takeaway is that timeouts, concurrency and result retention should be set together, because each one changes warehouse load and how long users wait.
| Setting | Fits when | Trade-off | Confirm before rollout |
|---|---|---|---|
| Broker URL (CELERY_CONFIG, broker_url) | Redis is the broker most installations use, reachable from both the web tier and workers. | If the broker is down, no async queries or scheduled jobs run. It becomes a shared dependency to monitor. | Broker URL format, TLS options and authentication for your broker version. |
| Results backend (RESULTS_BACKEND) | Users need to return to a finished query after a page refresh or session change. | Result data is stored outside the web process, so storage and access controls must cover it. | Backend type, key prefix and whether results are encrypted at rest in your setup. |
| Worker concurrency (celery --concurrency) | Set to the number of heavy queries your warehouse can run at the same time. | Higher values raise warehouse load and memory use per worker. Lower values make users queue. | Warehouse connection and concurrency limits, and the memory each query needs in testing. |
| Pool type (celery --pool) | Prefork suits CPU-heavy result handling. Threads or gevent suit workers that mostly wait on the database. | Prefork uses more memory per process. Threads and gevent depend on driver compatibility. | Which pool your database driver supports reliably under concurrent load. |
| Task time limits (CELERY_CONFIG, task_time_limit and task_soft_time_limit) | A hard cap stops runaway queries from holding a worker slot indefinitely. | Low limits cancel valid reports. High limits let a few queries tie up workers for long periods. | Limit values against your warehouse policy and the longest legitimate report. |
| SQL Lab timeouts (SQLLAB_TIMEOUT, SQLLAB_ASYNC_TIME_LIMIT_SEC) | Synchronous runs stay short. The async limit matches what analysts should be allowed to run. | A long async limit lets analysts hold workers. A short one cancels work that was still progressing. | Default values in your version and whether the warehouse enforces its own statement timeout. |
| Result retention (RESULTS_BACKEND expiry) | Short retention fits sensitive result sets that should not linger in shared storage. | Short retention means users re-run queries and spend warehouse capacity again. | The retention setting for your version and who may read results after expiry. |
Sizing workers and setting timeouts for heavy queries
Worker count, warehouse capacity and timeouts form one chain: a query waits for a worker, the worker waits on the database, and the browser gives up on the web tier first if the limits are not aligned. The cards below cover the drivers that move cost and user wait time, and each one should be load tested in staging before production sizing is fixed.
Concurrent long queries
The number of long SQL Lab queries that run at the same time sets how many Celery worker slots you need. If analysts launch heavy queries during the same window, the rest queue behind them and wait. Ask your team to estimate peak concurrency from current SQL Lab history, then load test that peak in staging with real query shapes before choosing worker counts.
Warehouse capacity
Workers do not make a query faster. They only hold it while the warehouse runs it. If the warehouse has a fixed number of concurrent sessions or compute slots, extra workers simply move the wait from Superset into the database. Confirm the connection and concurrency limits with your database owner, and size the worker pool so it never exceeds what the warehouse can serve.
Memory per query
Each worker process holds result data in memory while it fetches and serializes rows before writing them to the results backend. Large result sets raise memory use per process, which limits how many queries a single worker host can run safely. Check the row limits in your SQL Lab settings, measure worker memory during the load test, and set per-host concurrency below the point where memory pressure begins.
Number of analysts
Headcount drives demand indirectly. More analysts with SQL Lab access produce more overlapping queries, more result downloads and more history records in the results backend. Plan capacity for the active analyst group rather than total licensed users, and revisit it when a new team gets SQL Lab access, since a single new group can change peak load without any change to the platform.
Timeouts across three tiers
The browser, the web server and the Celery task each enforce their own time limit. If the web tier gives up while the worker keeps running, users see errors for queries that still consume warehouse capacity. Align the limits so the worker limit is the one that ends a runaway query, then confirm the web server and proxy timeouts are longer than the task limit. Test each boundary with a query that deliberately runs past it.
Wait time and retries
A query waiting in a queue feels slower to users than one that is running, and the wait grows fast once workers are saturated. Decide what a user sees while queued, how long results stay available in the results backend, and whether failed queries are retried. Retries multiply warehouse load, so turn them off for expensive statements unless your team has reviewed the query cost first.
Setting up Celery workers for SQL Lab
This sequence takes a Superset instance from a web-only setup to one where long SQL Lab queries run on Celery workers, return results to the browser, and can be confirmed with a test query that runs longer than the web request would tolerate. Each step ends with a check so the team knows it can move on.
Provision the caching layer and message broker
Platform engineering provisions a Redis instance (or the broker your team has chosen) that the Superset web tier and the Celery workers can both reach over the private network. Architecture documentation notes that async queries need the caching layer and worker components, and most installations use Redis and Celery for them. Confirm the broker is reachable from the web host and from each planned worker host, and that it is not exposed to other networks. The step is done when a simple connection test succeeds from every host that will run Superset or a worker.
Set the metadata database and SECRET_KEY
Before any async work, confirm the metadata database is PostgreSQL or MySQL in a supported version, not the SQLite default used by quickstart and PyPI installs. Set a user-specified SECRET_KEY in the configuration, since Superset will not start without one. Web and worker processes must load the same configuration file so they read the same metadata database and key. The step is done when the web application starts with that configuration and the metadata database shows the expected Superset tables.
Configure the Celery broker and results backend
In the Superset configuration, set the Celery broker URL to the broker from the first step and configure a results backend so finished query output can be retrieved by the web tier. Enable the async query path in the configuration and confirm the feature settings for SQL Lab are present in the same file used by workers. Keep the results backend's data retention in mind, because query results will be stored there until they are read or expire. The step is done when the web application loads without configuration errors and the settings appear in the running process.
Start the Celery worker and beat processes
Start one or more Celery worker processes using the same Superset application context and configuration, pointed at the same broker. Start the beat process if scheduled tasks such as reports or cleanup will run on this deployment, because beat handles scheduled work rather than SQL Lab queries. Set worker concurrency and the query timeout values deliberately with your database team, since these decide how many heavy queries can run at once and when they stop. Check the worker logs for a clean connection to the broker and a ready state. The step is done when the worker lists itself as online in the Celery inspection output.
Enable async execution for the relevant databases
In Superset, open the database connection settings for each warehouse that analysts will query from SQL Lab and enable asynchronous execution for those connections only. Limit the change to the databases the team has agreed should run long queries in the background, and confirm the SQL Lab role grants access to them. Record which connections have async execution on so the change is reviewable later. The step is done when a user with the SQL Lab role sees the async option or a query runs through the worker path when submitted.
Verify with a long test query
Run a test query in SQL Lab against a non-production database that takes longer than a normal web request, such as a deliberately slow aggregation agreed with the database owner. Watch the worker logs to confirm it picks up the task, and confirm the browser shows a running state that updates to a finished result without the page timing out. Confirm the result set returns to the browser and that a second user with the same access sees the query history entry. If the worker never picks up the task or the result does not return, check the broker URL and results backend configuration before enabling wider use. The step is done when the test query completes through the worker and the result renders in the browser.
How async execution changes the analyst experience
With async queries enabled, a long SQL Lab query no longer holds the browser request open until the warehouse returns. The analyst submits the query, the web tier hands the work to a Celery worker, and the interface shows a status while the query runs. The results appear when the worker finishes, and the analyst can leave the editor and return later. Confirm the exact status labels, polling behavior and notification options on your Superset version, because the interface changes between releases and the documentation does not spell out every screen state.
Query history becomes the place where analysts find output from a submitted query. Before rollout, confirm how long results remain retrievable in your results backend and whether history pruning or result expiry could remove output an analyst expects to revisit. Train analysts to save or export results they need to keep, and to rerun a query from history only when the underlying data may have changed. Dashboards behave differently: charts that fail or time out report that inside the dashboard, and the wait is shared across the page, so communicate that a slow chart may reflect worker queueing rather than a broken query. Check how your version handles chart queries and caching before promising analysts a particular behavior.
Training should cover three points: which queries run asynchronously in SQL Lab, where to find results after submitting, and what a timeout or cancelled query looks like on your deployment. Ask the team that operates the workers how queue depth and the configured query timeout will show up for users during peak hours. Analysts who understand that a query may wait in a queue, rather than fail, file fewer duplicate submissions and fewer support tickets. Document these behaviors in your internal guide and update it after each Superset upgrade, since UI and history behavior can change between releases.
Security and operational risks of running workers
Moving queries onto Celery workers adds components that hold credentials, carry query results and depend on shared secrets. Each card below names one risk, the control that reduces it and the part of your self-hosted trust boundary it sits in.
Broker access controls
The broker, typically Redis, carries task messages that describe which query runs against which database. Anyone who can publish to or read from it can inject tasks or see query text. Restrict network access to the broker to the web and worker hosts, require authentication on the broker, and confirm that it is not reachable from user subnets. Ask your team to document who administers the broker and how its credentials are rotated.
Results backend access controls
The results backend stores query output so the web tier can fetch it after a worker finishes. That output can contain sensitive rows, so the backend holds data, not just metadata. Limit its access to the Superset web and worker services, enable encryption in transit where your platform supports it, and treat it with the same classification as the warehouse data it mirrors.
Worker database credentials
Workers connect to the analytics databases with the credentials configured in Superset, so a compromised worker can query anything those accounts can read. Use read-only database users scoped to the schemas SQL Lab needs, keep credentials in your secret store rather than plain configuration, and confirm that worker hosts have no direct path to databases they should not reach.
Query result retention
Results sit in the backend until they expire, so retention settings decide how long sensitive output remains readable outside the warehouse. Confirm the expiry time configured for your results backend, decide whether analysts may export stored results, and check that cleanup runs on schedule. Record the chosen retention in your data handling policy so auditors can compare it with practice.
SECRET_KEY and session impact
Superset requires a user-specified SECRET_KEY to start, and the web and worker processes must agree on the configuration that depends on it. Changing or losing the key affects sessions and signed values, and a leaked key lets an attacker forge them. Store the key in a secret manager, use PREVIOUS_SECRET_KEY during rotation so users are not all signed out at once, and test a rotation in staging first.
Worker health monitoring
A worker that stops consuming tasks leaves analysts with queries that never finish, and nothing in the web interface may say why. Monitor worker process liveness, queue depth at the broker and task failure rates, and alert on a queue that grows while workers report healthy. Assign an owner for these alerts and include a restart procedure in your operations runbook.
Questions teams ask before running Celery for Superset
Do I need Celery workers to run long queries in Superset?
Not for every deployment, but you need them for asynchronous SQL Lab execution. Superset's architecture documentation lists Celery worker and beat processes as optional components, and says async queries, alerts and reports, and dashboard thumbnails require the caching layer and worker components. If your analysts only run short queries and can wait on the web request, you can run without workers at first. Once queries routinely exceed what users will wait for in a browser, asynchronous execution moves that work off the web tier. Confirm the exact configuration for your version in the architecture documentation.
What drives the cost of running Celery with Superset?
The cost comes from four areas: the design and build of the broker and results backend, the worker sizing and capacity you provision, the staging and testing effort, and the ongoing operation of the components. Most of the infrastructure cost is recurring compute, storage and the Redis or other broker service you run. Costs rise with the number of concurrent heavy queries, the timeout you allow, the number of environments and the number of teams that need their own tuning. Ask any partner for a quote that separates one-time build work from recurring operations, and that names which components they will deploy, configure and hand over.
How long does an async Celery rollout take?
Duration depends on scope, not on a fixed template. The main variables are how many environments need workers, whether you already run Redis or another broker, how many database connections and workloads must be tested, and how much load testing you require before users rely on asynchronous queries. A rollout limited to one staging and one production environment with an existing broker is a different job from a multi-region rollout with custom timeouts and monitoring. No duration figure is given here because none was supplied. Ask for a timeline tied to a written scope with named milestones.
What does my team need to provide for a Celery rollout?
You will need someone who owns the Superset deployment, access to the metadata database and the target environments, and a person who can make decisions about the broker, results backend and retention. Your platform team typically provides the Kubernetes or virtual machine capacity, networking between web, worker and broker components, and secrets management. Your data team should confirm which databases and queries are expected to run long, and your security team should review where query results are stored. Name these owners before the work starts so that testing and handover are not delayed.
Can my team set up Celery for Superset ourselves?
Yes, if you have engineers who run Python web applications, Redis or another broker, and container or virtual machine platforms. The Superset documentation covers the components and configuration, and the upgrade guide tells you to back up the metadata database, run database upgrades, test in staging and check UPDATING.md for breaking changes. The harder parts are usually worker sizing under real query load, timeout choices that match your databases, and results backend handling for sensitive data. Teams that lack these skills often choose a fixed-scope engagement for the build and then operate the platform themselves.
What is included in a fixed-scope quote for Celery workers?
A fixed-scope quote should list the components in scope, the environments, the deliverables and the acceptance tests. Ask for the broker and results backend choice with reasons, the worker deployment design, the timeout and concurrency settings for each database, the secrets handling for broker credentials, monitoring and alerting for the worker and beat processes, and a documented runbook. Also ask what is excluded, such as the database performance tuning, the broker hosting service, or the user training. DeployOpen plans and delivers work under written scopes like this, so the quote should follow directly from that document.
What breaks most often with async Superset queries?
The most common failures are workers that cannot reach the broker or metadata database, results backends that fill up or are not cleaned, and timeouts that do not match the database's own limits. Users also report confusion when a query finishes in the background and they must return to SQL Lab to see results. Check worker logs, broker health and results storage growth before you tune concurrency. Test each database with long queries in staging, and confirm the behavior when a worker restarts during a running query.
How DeployOpen scopes and supports Superset async workers
DeployOpen plans, implements, documents and supports self-hosted Superset in infrastructure you control, including the Redis, Celery worker and metadata database layers that async SQL Lab queries depend on. Each engagement starts from a written scope that sets the workers, broker, results backend and timeouts before any configuration is changed.
How we scope
- Inventory which SQL Lab users run long queries and which databases they target.
- Agree the broker and results backend choice with your platform team in writing.
- Define worker count, concurrency, queues and timeout values for each query class.
- Record the metadata database version, which must be PostgreSQL 10 to 17 or MySQL 5.7 or 8.x.
- Confirm the fixed-scope deliverables, acceptance tests and any ongoing retainer before work starts.
How we build
- Deploy Redis or your approved broker, Celery workers and the Superset async query configuration in your environment.
- Set SECRET_KEY and session handling in line with the configuration documentation before the first start.
- Tune worker concurrency and result expiry against measured load in a staging environment.
- Test async execution with long queries, cancellation, failed workers and broker restarts.
- Verify that row level security and database permissions still apply to queries run by workers.
How we hand over
- Provide documentation of the worker topology, configuration keys and timeout rationale.
- Deliver runbooks for restarting workers, draining queues and clearing stuck query results.
- Walk through the user experience for async queries so analysts know how status and results appear.
- Hand over backup steps for the metadata database and the results backend where they hold data.
- Document the upgrade procedure, including the metadata database backup and UPDATING.md review.
How we support
- Support is available as a retainer covering monitoring reviews, patching and upgrade work.
- Retainers can include periodic review of worker sizing as query volume changes.
- Upgrades are planned against the Superset upgrade guide and tested in staging before production.
- Support scope, response expectations and review cadence are agreed in writing with your team.
- Ask for the retainer terms and the written scope together so responsibilities are clear.
