Sizing Gunicorn workers for Apache Superset before users complain about slow pages
If your Superset web tier queues requests or times out when several people open shared dashboards at once, the fix usually starts with the Gunicorn settings for the web process: worker class, worker count, threads and timeouts. Those settings decide how many requests the application can serve at the same time and how long a slow response is allowed to hold a connection. Sizing them deliberately is cheaper than discovering the limit during a quarter-end review, when every analyst opens the same dashboard within minutes of each other.
The web tier serves the Flask and React application, which is one of the four components in the Superset architecture documentation. The other three are the metadata database, an optional caching layer and optional Celery worker and beat processes. Gunicorn runs the web tier and governs its concurrency and timeouts, while Celery workers handle asynchronous work such as async queries, alerts and reports, and dashboard thumbnails. The architecture documentation notes that those asynchronous features need the caching layer and worker components, so adding Celery capacity does not relieve a web tier that is saturated by page requests. Teams should plan this page when they move beyond a single-user test instance, when dashboards are shared with broad audiences, or when a reverse proxy or load balancer starts reporting upstream timeouts. Confirm the exact Gunicorn options and their defaults for your Superset version in the Apache Superset architecture documentation and in your own deployment configuration before changing them.
Measure before you change anything. Record request latency, queue depth and timeout errors at the proxy during a normal peak, then change one setting at a time in staging. Keep in mind that every extra worker process holds its own memory and database connections to the metadata database, so the worker count you choose must fit the memory and connection limits of the host and of PostgreSQL or MySQL. Ask whoever runs your platform to confirm the proxy timeout is longer than the Gunicorn timeout, so the proxy does not cut off a request that the application is still completing. Load that reflects a shared dashboard fan-out is a better test than a single user clicking through pages.
Gunicorn settings and when each choice fits
This table compares the Gunicorn settings that shape Superset's web tier, with the situation each one suits and the trade-off it creates. Read each row against your user count, dashboard load pattern and warehouse latency, then confirm defaults and supported values in the Superset configuration documentation for your version.
| Setting | Choose it when | Trade-off it creates | Confirm before rollout |
|---|---|---|---|
| Sync worker class | Dashboards mostly wait on the warehouse and each request is short, and you want predictable memory per process | Each worker handles one request at a time, so slow warehouse queries tie up a whole process | Default worker class in your version's configuration |
| Threaded worker class (gthread) | Many users open dashboards at once and each request spends most of its time waiting on I/O | Threads share one process memory, so a memory leak or large result set affects more requests | Whether your version's documented examples recommend threads |
| Gevent or other async worker class | Only when every database driver in your stack is verified to be compatible with monkey patching | Blocking drivers can stall the whole event loop, which the BigQuery caveat in this guide explains | Driver compatibility notes for each warehouse you connect |
| Worker count | Sized from CPU cores and memory per process, starting low and raising it after load tests | More workers raise memory use and database connections, and too many can slow the host | Recommended formula in the current Superset deployment guide |
| Threads per worker | Requests are I/O bound and you want concurrency without adding processes | Thread count multiplies concurrent warehouse calls, which can increase load on the warehouse | Whether your Gunicorn version supports the thread setting you plan to use |
| Request timeout | Long chart queries are expected and you want the web tier to wait for them | A high timeout holds workers longer during stalls, so fewer requests get served | Timeout values in the Superset configuration and any proxy timeout in front of it |
| Keep-alive | Clients sit behind a load balancer that reuses connections and you want fewer reconnects | Idle connections hold worker capacity, so the value needs to match the balancer idle timeout | Keep-alive values in your load balancer and Gunicorn documentation |
| Web replicas behind a load balancer | Concurrent users outgrow one host, or you need restarts without downtime | Session and cache state must be shared across replicas, or users see inconsistent behavior | Shared cache and session configuration before adding replicas |
Choosing a worker class for your workload mix
Most dashboard viewers load a page, change a filter and read results that are cached or quick to return. For this mix, the web tier spends much of its time waiting on the network and on short calls to the metadata database, so the main question is how many users one node can serve without holding a large amount of memory for each connection. Synchronous workers handle one request per process. They are simple to reason about and easy to troubleshoot, but each process consumes memory whether it is busy or idle. Threaded and async-style workers share a process across several connections, which can serve more light users per unit of memory.
- Sync workers: predictable behavior and simple logs, but concurrency comes from running more processes, so memory grows with each one.
- Threaded workers: several requests share one process, which suits light traffic if the code paths you use are thread-safe.
- Async-style workers: can hold many idle connections cheaply, but only pay off when the code paths do not block.
- Memory per process: Superset loads a large dependency tree, so measure resident memory for one idle worker in your environment before choosing a count.
- Proxy connection handling: check whether your reverse proxy holds keep-alive connections open on behalf of browsers, since these can tie up worker slots.
Check the memory footprint of one idle worker and whether your proxy settings allow idle browser connections to release worker slots.
When charts run heavy warehouse queries, the worker spends most of its time waiting on the database. A synchronous worker is blocked for the full query, so each in-flight chart consumes a slot. Threads help during these waits because blocking network I/O releases the interpreter lock, letting other requests proceed in the same process. Async-style workers can also wait efficiently, but most warehouse drivers are blocking, and a blocking call inside an async-style worker can stall every request that process is handling. The number of simultaneous chart queries one node can run is roughly the number of worker processes multiplied by threads, which makes the web tier sizing a direct input to warehouse load.
- Slots set the ceiling: concurrent chart queries cannot exceed the worker processes multiplied by threads, so a few heavy dashboards can exhaust every slot.
- Driver compatibility: confirm whether your warehouse driver is blocking, and test it with the async-style worker before you commit to that class.
- CPU work stays serialized: threads speed up waiting, but pandas-style transformations and result processing still run one at a time per process.
- Timeout interaction: a worker timeout shorter than the slowest chart query kills the request mid-flight and leaves the user with an error.
- Warehouse concurrency: more slots mean more simultaneous queries against the warehouse, which affects both performance and cost.
Check the slowest chart queries in your logs against the configured worker timeout and the concurrency limit your warehouse enforces for the service account.
Many production deployments serve quick dashboard views and long-running requests from the same web tier. Long requests such as large exports, slow SQL Lab runs or heavy dashboard loads can occupy slots for long periods, which makes light browsing feel stalled even when the server has spare CPU. The choice of worker class matters less here than the decision about where long work runs. Threads and async-style workers increase the number of requests a node can hold, but they do not remove the risk that a small set of long requests dominates. Per the Superset architecture documentation, async queries, alerts, reports and thumbnails need the caching layer and worker components, which moves long-running work off the web request path.
- Slot exhaustion: a few long requests can use every available slot, so quick page loads queue behind them.
- Dedicated web pools: a separate set of web replicas for heavy paths is a planning option; confirm that your load balancer can route by path or host.
- Async offload: moving long queries to asynchronous execution frees web slots, provided the workers and caching layer are sized for that work.
- Layered timeouts: set the worker, proxy and browser timeouts so the slowest layer fails clearly and the others do not mask the error.
- Restart behavior: long requests complicate rolling restarts, so confirm how in-flight requests drain during deployments.
Check which request paths hold a worker longest in your logs and whether each one can move to asynchronous execution.
The worker class should follow the workload you measure, not the default you inherited. Start by identifying the dominant pattern in your access logs, then test one change at a time in staging with a load profile that resembles your peak usage. Record memory per worker, response times and error rates for each configuration, and keep the chosen settings in version-controlled configuration so they survive redeployments. Retune after major Superset upgrades, because dependency changes can alter memory use, and after you add dashboards with many charts or new warehouse connections. The upgrade documentation recommends testing in staging first, which is a sound place to validate worker settings.
- Measure the mix: separate light page views, chart queries and long requests in your logs before choosing a class.
- Change one variable: adjust worker class, count or threads individually so the effect of each change is clear.
- Watch the signals: rising 502 or 504 responses, worker restarts and memory growth indicate the configuration is under strain.
- Store settings as code: keep the Gunicorn configuration in version control next to your Superset deployment files.
- Plan recurring tuning: schedule a review after each upgrade and after significant dashboard or connection changes.
Before you go live, confirm that the staging results match the peak usage you expect and that a rollback to the previous configuration is documented.
Why the BigQuery gevent caveat matters for worker class
Gevent workers can interact badly with some Google client libraries, so a BigQuery deployment needs a check before gevent is chosen. Gevent-based worker classes patch the Python standard library to run many connections concurrently inside one process. That model suits I/O-heavy web traffic, but a client library that does not cooperate with the patching can block, hang or behave unpredictably under concurrency. The risk sits in the combination of your worker class, your Superset version and the exact Google client library versions installed in the same image, not in any one component alone.
Before you change the worker class, confirm current guidance in the Superset documentation for your release, and check the compatibility of the BigQuery connector and Google client libraries your build pins. Read the release notes for those libraries rather than relying on memory of past behavior, and record the versions you tested. If the documentation or your library versions raise a concern, keep the synchronous worker class for the BigQuery-facing tier until the combination is proven. Ask any partner or internal team to write down which versions were checked and where the source of that guidance lives.
Test under load in staging before touching production. Use a copy of your metadata database, a representative set of BigQuery-backed dashboards and chart requests with realistic concurrency, and watch for hung requests, worker timeouts, unexpected errors from the client library and changes in query latency. Compare the same test on the current and candidate worker classes, and keep a rollback path that restores the previous Gunicorn settings. Only promote the change once the staging run completes cleanly at your expected peak.
How to set Gunicorn sizing and verify it under load
This sequence takes the web tier from a measured baseline to a sized Gunicorn configuration that has been load tested against realistic dashboard use and shows no proxy 502 or 504 responses. Each step leaves a written record that the next step depends on.
Record the baseline for concurrency and latency
Before changing anything, the platform owner records how the current Superset web tier behaves: peak concurrent users, requests per second, p95 response time per endpoint, and the proxy timeout in front of Gunicorn. The usual sources are the proxy access logs, Gunicorn logs and any application metrics already collected. Ask the Superset admins which dashboards carry the most traffic, since those define the load profile. Done means a one-page table exists with these figures and the date they were captured.
Inventory the slowest expected chart queries
Identify the longest queries that users are expected to trigger from dashboards and SQL Lab, and note which warehouse engine runs them. Each query's runtime becomes a floor for the Gunicorn timeout, because a worker that is killed mid-query returns an error to the user. Confirm with the data team which datasets have known long scans and whether any of those charts are candidates for async execution through Celery. Done means a list of the slowest expected requests, each with a measured or agreed duration and an owner.
Choose the worker class and worker count
Select the Gunicorn worker class and starting worker count from the baseline and the engine's known behavior. Confirm the worker class supported by the Superset image you run and check the Superset configuration documentation for the current recommended settings. Be cautious with the BigQuery gevent caveat: if the query path uses that driver, confirm whether the gevent worker class is safe for your version before selecting it. Record the chosen class, worker count and the memory each worker needs on one container. Done means a written configuration draft that names the class, count and memory budget per replica.
Set threads and timeouts to match the slowest request
Set the threads per worker and the Gunicorn timeout so that the slowest expected query finishes inside the worker timeout, and the Gunicorn timeout is shorter than the proxy timeout so the application fails cleanly before the proxy does. Keep the proxy timeout and the Gunicorn timeout as separate settings you can show side by side. Note that a longer timeout holds worker slots longer, which reduces capacity under load. Done means a timeout table listing the proxy value, the Gunicorn value and the slowest query each one covers, and a reviewer has signed off the ordering.
Deploy the configuration to staging
Apply the Gunicorn settings to a staging Superset that matches production in web replica count, proxy configuration, metadata database, cache and warehouse connections. Keep the Superset version identical to production so that the results are comparable. Restart the web tier and confirm each replica starts, logs the expected worker count and passes health checks behind the load balancer. Done means staging shows the configured workers on every replica and the load balancer marks all replicas healthy.
Load test with realistic dashboard use
Run a load test that reproduces real dashboard sessions: page loads, filter changes, chart refreshes and export actions, weighted by the traffic mix recorded in step one. Use a tool your team already knows and keep the test script in version control. Ramp concurrency in steps and record latency, worker saturation and memory at each level. Note any point where requests queue because all workers are busy. Done means a results file exists for each load level with latency percentiles, error counts and the concurrency where latency first rises sharply.
Confirm no 502 or 504 responses from the proxy
Review proxy logs and the load test output for 502 and 504 responses, and match any that appear to a worker timeout, a worker crash or a proxy timeout. A 502 usually points to a worker that died or closed the connection, and a 504 usually points to a request that outlasted the proxy timeout. Adjust the worker count, threads or timeouts, then rerun the test until no 502 or 504 appears at the target concurrency. Done means a final run at target load with zero 502 and 504 responses, and the run is saved next to the configuration it validated.
Scaling web replicas behind a load balancer
Running more than one Superset web replica changes how logins, secrets, health checks and upgrades behave. The cards below cover the decisions to settle before adding a second replica behind your load balancer, and what to confirm on your platform.
Session and login affinity
Flask-AppBuilder keeps the login session in a signed cookie, but Superset also depends on server-side state for some features, so a request can land on a replica that has not seen the user's earlier steps. Decide whether your load balancer needs session affinity for login and for long chart requests, and test a login flow across replicas before go-live. Confirm the cookie name, the affinity mechanism and how long a sticky assignment lasts on your ingress or load balancer.
Shared SECRET_KEY across replicas
Every replica must start with the same SECRET_KEY, because Superset refuses to start without a user-specified key and session cookies signed by one replica must be accepted by the others. Store the key in your secrets manager and inject it identically into each pod or instance, then check that rotation with PREVIOUS_SECRET_KEY is planned as a coordinated change. A mismatch often shows up as users being logged out at random, so confirm the value is identical on every replica before you compare logs.
Sticky sessions versus shared cache
Sticky sessions keep a user on one replica, which helps with in-process state but concentrates load and hides failures. A shared Redis cache lets any replica serve any request and is the usual choice when the caching layer is already in place. Decide which trade-off fits your traffic, and confirm that your Redis deployment is reachable from every replica and that cache keys do not depend on the replica that wrote them.
Health checks for web replicas
A load balancer should only send traffic to replicas that can answer requests and reach the metadata database. Point the probe at a lightweight endpoint, set a timeout that is shorter than your slowest expected chart request, and avoid probes that run a warehouse query. Confirm the health path your deployment exposes, the success codes your load balancer expects, and how it behaves when a replica is still loading after a restart.
Rolling upgrades and one-time migrations
Database migrations such as superset db upgrade and superset init change shared metadata and should run once per release, not from every replica at startup. Plan a separate migration job, take a metadata database backup, and roll web replicas only after that job succeeds. Check UPDATING.md for breaking changes between versions and confirm with your platform team how the rollout handles old and new replicas serving traffic at the same time.
Questions teams ask before sizing Superset web workers
What drives the cost of sizing Superset web workers?
The cost comes from how much measurement and tuning the work requires, not from the number of workers themselves. The main drivers are how many dashboards and user groups must be tested, whether a staging environment matches production, how many identity and proxy layers sit in front of Superset, and how many warehouses the dashboards query. Costs are one-time for the sizing exercise and recur whenever user counts, dashboard designs or Superset versions change. Ask any provider to price the load test scenarios, the staging environment, and the follow-up retest as separate line items in the written scope.
How long does a Superset worker sizing exercise take?
Duration depends on the environment and on how much test coverage already exists. A team with a staging copy of production, a load test tool already in use, and documented dashboard usage can move quickly, while a team that must build test data, define user journeys and wait on warehouse access will take longer. No fixed duration applies to every deployment. Ask for a timeline that lists the baseline measurement, each test round, the tuning changes between rounds, and the final validation, so you can see which steps depend on your team's availability.
What does the sizing work need from our team?
The work needs several inputs before any test runs. Your team should provide a load test tool or agree on one, a staging environment that mirrors the production web tier, read access to the warehouse or a representative replica, and details of the identity provider and reverse proxy in front of Superset. Your team should also share dashboard usage data, such as peak concurrent users and the most viewed dashboards. Missing proxy timeouts or login flow details are a common cause of rework, so confirm them early.
Can our internal team size Superset web workers alone?
An internal team with platform engineering experience and access to a staging environment can do this work alone. The skills needed are load testing, reading Gunicorn and proxy logs, and understanding how Superset's web requests call the warehouse. The main risks for an internal team are testing with unrealistic traffic, missing the effect of slow warehouse queries on worker occupancy, and not repeating tests after upgrades. If your team lacks time or load testing experience, a partner can take on the test design and tuning under a written scope while your engineers keep ownership of the result.
What goes wrong when Superset web workers are sized incorrectly?
Under-sized workers cause requests to queue, which users experience as slow page loads or gateway timeouts. Over-sized worker counts can exhaust memory on the host or push too many simultaneous queries to the warehouse, which raises cost and can trigger warehouse limits. Mismatched timeouts create a different failure: the proxy gives up while Gunicorn keeps working, so users see errors for queries that still run. Each of these failures shows up under peak load, which is why testing with realistic concurrency matters more than testing with one user.
Does the BigQuery gevent caveat change how we size workers?
Yes. Some BigQuery client behavior does not combine well with gevent workers, so the worker class you choose can affect whether queries complete reliably. Treat the worker class as a decision to validate, not a default to copy from another deployment. Confirm the Superset version, the BigQuery client library version and the Gunicorn worker class together in staging, then repeat that test after every upgrade. Your Superset and Google Cloud documentation are the primary references for the current guidance.
Who should own Superset web tier sizing after the first exercise?
Ownership should stay with the team that runs the Superset platform, because sizing changes whenever dashboards, users, identity settings or versions change. Assign one person to track the peak concurrency numbers, one to own the load test scripts and one to approve changes to worker counts or timeouts. A partner can deliver the initial sizing and documentation, and an ongoing retainer can cover retests after upgrades. Ask for the test scripts and tuning records as deliverables so that your team can repeat the work without the partner.
How DeployOpen scopes and supports Superset web tier sizing
Web tier sizing for Apache Superset starts with a written scope. Before any work begins, DeployOpen and the client agree in writing on which Superset deployment is in scope, the Gunicorn settings to be tested (worker class, worker count, threads and timeouts), the load profile that represents real users, and the deliverables, such as a tested configuration, a sizing note and a runbook for the operations team. Most engagements in this area are fixed-scope, so the quote follows from that document rather than from a standard package. The scope also records which data sources the dashboards query, because web tier timeouts interact with warehouse query behavior and that dependency has to be visible before the first test run.
Changes to the web tier go through staging first. DeployOpen applies candidate Gunicorn settings to a staging copy of the client's Superset, built from the same version, configuration and metadata database as production, and runs load tests against it in the client's own infrastructure, whether that is a cloud account, a data center or an air-gapped network. Test results, error rates and warehouse query counts are recorded so the client can confirm the chosen values before they reach production. Before any production change, the metadata database is backed up and the upgrade guidance in the Superset documentation is followed where a version change is involved, because a version change alters what the web tier must handle.
Ongoing support is optional. Many clients ask for a retainer to cover re-testing after a Superset upgrade, a rise in dashboard count or a change in identity provider, and to review logs for worker timeouts and restarts. Retainers are scoped in writing like the initial work, so the client knows what recurs and what is one-time. DeployOpen does not publish a price list. To get a quote, ask for a written scope that names the environments, the load profile, the testing hours, the deliverables and any retainer terms, and confirm which of those costs your own team would otherwise carry.
