Why the Airflow 3 API server needs its own sizing plan
If you are moving from the Airflow 2 webserver to the Airflow 3 API server, the first question is whether the default setup can carry your users and automation. In Airflow 3 the webserver is replaced by an API server started with airflow api-server, and the DAG processor must run as a separate component started with airflow dag-processor, according to the Airflow 3 upgrade guide. That split changes what the API server is responsible for, so an Airflow 2 webserver sizing is not a safe starting point. Treat the API server as its own service with its own capacity plan, separate from the scheduler and the DAG processor.
The API server does more than render the UI. It serves the REST API, and task code in Airflow 3 reaches the platform through the REST API or Task SDK rather than database sessions, so task traffic also passes through this component. Because of this, a busy set of workers can create load on the API server even when no one opens the UI. Plan a dedicated sizing exercise when you have a large number of UI users, heavy API consumers such as CI jobs, monitoring pollers or external systems that trigger runs, or a large DAG estate that makes page loads and DAG listings expensive. Ask your team to measure request rates from each client type before choosing a worker count.
The one decision to make early is how many API server workers and replicas you need, and whether your deployment can recycle workers without downtime. Airflow 3.1.0 reduced the default number of API server workers to 1, and Airflow 3.2.0 added gunicorn support with zero-downtime worker recycling, according to the release notes. Confirm which Airflow version you will run and which settings your version exposes before you write the plan. Also confirm the shared secret key, the clock synchronization between components and the load balancer behavior in front of the API server, since each of these affects whether replicas behave consistently.
What changed when the default dropped to one worker
Airflow 3.1.0 reduced the default number of API server workers to 1, so a stock install serves every UI page load, DAG list request, log fetch and API call from a single worker process unless you configure more. One worker is a starting point, not a sizing decision. Teams that upgraded from Airflow 2 and kept the webserver habits they knew may assume the old multi-worker behavior still applies, and that assumption is where slow dashboards and delayed API responses usually start. A single worker can queue requests behind a slow one, so a heavy DAG grid view or a long log download can make the whole UI feel unresponsive even when the database and scheduler are healthy.
The practical consequence is that worker count becomes a setting you own and must justify. Before you change anything, measure how many concurrent users, automation clients and monitoring scrapes hit the API server during your busiest hour, and watch request latency and error rates while those requests run. Then set the worker count deliberately in your deployment configuration rather than inheriting whatever the release ships with. Your target should be a number you can explain, test under load and revisit after each upgrade, because default values can change between releases.
Confirm the exact configuration key names and their defaults in the Airflow 3.1.0 and 3.2.0 release notes before you write any configuration, since the key that controls worker count and the options that appear with gunicorn support in 3.2.0 are the first things to verify. Check them against the version you actually run, using the Apache Airflow release notes, and record the version in your change ticket. If your platform team builds images or Helm values from templates, confirm that the rendered configuration reflects the intended worker count in a staging environment before promoting it.
Gunicorn and zero-downtime recycling in Airflow 3.2
Airflow 3.2.0 added gunicorn support for the API server. Before that release, the API server was the component serving the UI and REST API that you scaled by running more processes or replicas behind a load balancer. With gunicorn in place, the API server runs as a pool of worker processes managed by a process manager, which is the arrangement most Python web services use in production. The release notes also reduced nothing in this area, but Airflow 3.1.0 changed the default to a single API server worker, so teams that depended on the older default need to set the worker count explicitly after upgrading to 3.2.
- Gunicorn support: the API server can run under gunicorn instead of only the built-in development style server.
- Worker count: the number of API server workers is a setting you control, and the default you inherit depends on whether you are coming from 3.0 or 3.1 or later.
- Process model: each worker handles requests independently, so memory use grows with the worker count.
- Release scope: the same release notes list the scheduler change covered in another tab.
Confirm the exact gunicorn-related option names and their defaults in the release notes for the version you run, and test them in staging before changing production.
Zero-downtime worker recycling means gunicorn can replace API server worker processes without dropping the requests they are serving. Workers are restarted over time, for example to release memory or pick up a new configuration, and the process manager starts replacements before the old workers exit. For a rolling restart, this reduces the window in which the UI or API returns errors during a change. It does not remove the need for a load balancer health check or for more than one API server replica if your users cannot tolerate any interruption.
- Replacement before exit: new workers take traffic before old ones are removed, which is the behavior that makes a restart invisible to most users.
- Single replica limits: with one API server instance, recycling still happens inside that instance, so a host or container restart remains an outage.
- Long requests: slow API calls or large log downloads may be cut off if they outlast the drain period, so review the timeouts your gateway uses.
- Load balancer checks: health checks should point at the API server health endpoint so unhealthy replicas leave rotation.
- Clock and secret consistency: replicas must share the same API secret key and use synchronized clocks, or requests can fail with forbidden errors.
Before relying on rolling restarts, rehearse a restart under realistic UI load and confirm the recycling and timeout behavior with your release notes and load balancer documentation.
The release notes for Airflow 3.2.0 also added a --only-idle flag to the scheduler CLI. The flag is relevant to the API server topic because scheduler and API server restarts are usually planned together during the same maintenance window. Its purpose is to let an operator restrict an action to idle scheduler state rather than acting while the scheduler is busy. Read the release notes for the exact command it applies to and the precise meaning of idle in that version, because the behavior determines whether it is safe to use in automated rollouts.
- Maintenance coordination: use the flag to avoid interrupting scheduling work during a planned change, if the command you run supports it.
- Scope check: confirm which scheduler command accepts the flag in your version, since it applies to the scheduler CLI only.
- Automation impact: a rollout script that waits for idle state needs a timeout and an alert, so a stuck scheduler does not block the pipeline silently.
- Multiple schedulers: if you run more than one scheduler, confirm how the flag behaves per instance.
- Change records: note the flag and version in your change ticket so auditors and operators can reproduce the sequence.
Confirm the flag's exact behavior and supported commands in the Airflow 3.2.0 release notes and test it in a non-production environment before adding it to runbooks.
Combining these features into a restart plan is the part most teams get wrong on their first attempt. The order matters: the database migration, the API server image, the DAG processor, and the scheduler should be sequenced according to your upgrade notes, and the API server recycling behavior should be verified after each step. Plan the rollout so that at any moment at least one healthy API server replica is in the load balancer pool, and confirm that UI sessions and API clients tolerate a brief reconnect. Document the expected behavior so the on-call team knows what a normal restart looks like.
- Pre-change check: record the current worker count, replica count, and health check settings.
- Staged rollout: restart one replica at a time and wait for health checks to pass before the next.
- Verification: load the UI, run a representative API call, and check the API server logs after each replica returns.
- Rollback path: keep the prior image and configuration available, and test that you can return to them.
- Operator notes: record which restarts are expected to be invisible and which are not.
Before the first production rollout, run the full sequence in staging with production-like traffic and confirm each step with your own load balancer and monitoring setup.
Verification is what turns these features into dependable operations. Confirm that the release you run actually provides the gunicorn support and scheduler flag described in its notes, because a patch release on the same minor line can differ from your assumptions. Check the API server worker count and memory use after the change, and watch for errors in the logs during recycling. Set alerts on API server health and latency so a regression appears before users report it. Ask your implementation partner or internal team to document the exact settings, the test results, and the rollback steps as part of the change record.
- Version check: confirm the installed Airflow version and the release notes that match it.
- Setting check: review the effective API server worker and timeout settings in every environment.
- Recycling test: observe that replacement workers serve traffic before old ones exit.
- Alerting: set health check and error rate alerts on the API server and load balancer.
- Evidence: keep the tested runbook, results, and rollback steps with the change record.
Confirm the exact option names, defaults, and any known limitations for your version in the Airflow release notes before you finalize the runbook.
Which scaling option fits which load pattern
This table compares the four ways to add capacity to the Airflow 3 API server: the single-worker default, gunicorn workers with recycling, horizontal replicas behind a load balancer, and vertical sizing. Read each row against your own UI and API traffic pattern, then confirm the version-specific behavior in your environment before choosing. The main takeaway is that worker settings and replica count solve different problems, and most teams need to measure before adding either.
| Option | When it fits | Trade-off | What to confirm |
|---|---|---|---|
| Single worker default (3.1.0 and later) | Small teams, light UI use, or test environments where a few concurrent users open the UI | Lowest configuration effort, but one worker handles all requests, so slow API calls delay other users | Your exact Airflow 3.x version, and whether the default worker count is enough for your observed request latency |
| Gunicorn workers with recycling (3.2.0 and later) | Steady multi-user UI and API load on one host where requests queue behind a single worker | Adds worker processes on the same host, which uses more memory; recycling avoids downtime during restarts | The worker count and recycling settings for your version, and memory headroom on the host after adding workers |
| Horizontal API server replicas behind a load balancer | Sustained high concurrency, or a need to remove the API server as a single point of failure | Every replica must share the same secret key and reach the same metadata database; load balancer and session behavior add operational work | Shared secret key setup per the Helm chart production guide, clock synchronization across nodes, and load balancer health checks and timeouts |
| Vertical sizing of CPU and memory | Single-host deployments where the API server is slowed by CPU or memory pressure rather than request queuing | Simple to apply, but has a ceiling on one host and does not remove a single point of failure | Current CPU and memory use under load, and how the host is shared with the scheduler, DAG processor and database |
| Shared baseline for all options | Every Airflow deployment, whatever API server option is chosen | The prerequisites recommend at least 4 GB of RAM for Airflow overall, which is a minimum, not an API server sizing figure | Measured API server latency and memory use in your own environment before and after each change |
How to set up and scale the API server step by step
This sequence takes an Airflow 3 API server from version check to a load-balanced, multi-replica setup with consistent secrets and clocks. Each step closes with a check so you know it is safe to move on.
Confirm the Airflow version and 3.1 and 3.2 changes
Your platform engineer confirms the exact Airflow 3 patch release in use and reads the release notes for the API server changes. Airflow 3.1.0 reduced the default number of API server workers to 1, and Airflow 3.2.0 added gunicorn support with zero-downtime worker recycling. Before you size anything, confirm which of these behaviors your version has, because the worker defaults and tuning options depend on it. Check: you can state the running version and the worker default it ships with, recorded in your change ticket.
Establish a PostgreSQL or MySQL metadata database
The database team provisions a supported PostgreSQL or MySQL metadata database that every Airflow component will share. The Airflow documentation says SQLite is for testing only and that MariaDB is not supported as a metadata database. Confirm that your chosen database version appears in the Airflow 3 prerequisites for your release. Check: the Airflow components can connect and run migrations against the new database from a test host, and the connection string is stored in your secrets store, not in a config file.
Deploy the API server separately from the DAG processor
The platform team runs the API server as its own process or pod, started with the airflow api-server command, and runs the DAG processor as a separate component started with airflow dag-processor. In Airflow 3 the webserver is replaced by the API server, and the DAG processor must not be folded into it. Scaling the UI and API traffic then stays independent of DAG parsing. Check: both components start, the API server responds on its health endpoint, and the DAG processor reports parsed DAGs in its logs.
Set the worker count and gunicorn options
Set the number of API server workers explicitly rather than relying on the default, since the default changed to 1 in Airflow 3.1.0. On Airflow 3.2.0 or later, you can run the API server under gunicorn, which supports zero-downtime worker recycling. The right worker count depends on your CPU allocation and observed UI and API load, so load test with your own DAG counts and user numbers before fixing a value. Check: under a test load, response times and memory stay within the limits you agreed, and a worker recycle completes without failed requests.
Add API server replicas behind the same database
Add replicas only after one instance behaves correctly, and point every replica at the same metadata database and the same secret key. Each replica must run with identical Airflow configuration so that any request can be served by any instance. Confirm the replicas are stateless from the user's point of view by reviewing where session and log data is written. Check: each replica answers the health endpoint individually, and a log fetch succeeds through each one.
Set a static API secret key per instance
When you deploy with the official Helm chart, create a Kubernetes Secret that holds a static API secret key and reference it from the chart. The Helm chart production guide says the key should be different for each Airflow instance and kept the same across all API server replicas within that instance. Store the value in your secrets manager and restrict who can read the Secret. Check: all replicas start with the same key, and a second instance does not accept tokens issued by the first.
Synchronize clocks across all nodes
The infrastructure team confirms that every API server, scheduler, worker and DAG processor node synchronizes time from the same trusted source. The Helm chart production guide warns that unsynchronized clocks can cause forbidden errors when task logs are fetched. Record which time service each node uses and set alerts for drift. Check: a task log fetched from the UI through each replica loads without a forbidden error, and the clock offset on every node is within the tolerance your security team has approved.
Place a load balancer in front of the replicas
The network team places a load balancer in front of the API server replicas and terminates TLS there, using certificates from your internal certificate authority. Configure health checks against the API server health endpoint so unhealthy replicas are removed from rotation. Confirm which session and sticky routing behavior your load balancer needs for the UI, and test it under load. Check: traffic reaches all healthy replicas, a stopped replica is dropped from rotation, and a log fetch through the load balancer succeeds for a task that ran on a different node.
Shared secret keys and clock sync across replicas
Once the API server runs as several replicas behind a load balancer, every replica has to agree on the secrets that sign and check requests, and on the time those requests carry. The cards below cover the static API secret key, how it is stored, how rotation is planned, and the clock sync problem that shows up in task log fetches.
Static API secret key across replicas
Replicas that do not share the same secret key cannot validate each other's tokens, so a request served by one replica can fail on another. Set the key once for the whole deployment, not per pod, and confirm in your Helm values and config that every API server replica reads the same value. Test by sending the same authenticated request several times through the load balancer and checking that each response succeeds.
One key per instance
The Helm chart production guide says the static API secret key should be different for each Airflow instance. Two environments, such as staging and production, must not reuse the same key, or tokens issued in one can be accepted in the other. Record which key belongs to which instance in your configuration inventory so that a copied values file does not carry a key into the wrong cluster.
Kubernetes Secret handling
The guide directs you to supply the key through a Kubernetes Secret rather than a plain value in Helm values or a ConfigMap. Confirm who can read that Secret, whether your cluster encrypts Secrets at rest, and whether your GitOps repository holds only a reference to it. Make sure the same Secret name is referenced by every API server and by any component that needs it.
Clock sync and forbidden log fetches
The Helm production guide warns that unsynchronized clocks can cause forbidden errors when task logs are fetched. Requests signed on one node can look expired or future-dated on another, so a log view in the UI may fail even though the task ran. Confirm that NTP or your platform time service is running on every Kubernetes node that hosts Airflow pods, and check for clock skew after node replacement.
Rotation planning
The sources supplied for this topic do not describe a rotation procedure for the static API secret key, so confirm the exact behavior with the Airflow release you run before planning one. Ask what happens to active sessions and in-flight log requests when the key changes, whether replicas must restart together, and how a rolling update avoids a window where old and new keys coexist. Write the steps down and rehearse them in a non-production instance.
Checks before each change
Before changing the key or the time configuration, confirm that all replicas report the same key source, that node clocks agree, and that a sample task log loads from the UI through the load balancer. Repeat these checks after scaling events, because new replicas are the most common place where a mismatched key or an unsynchronized clock appears.
How much effort and outside help does API server scaling take
What drives the effort of scaling the Airflow 3 API server?
The main drivers are how many people use the UI, how many automated clients call the REST API, where the API server runs, and whether you are coming from Airflow 2. Concurrent UI users and programmatic callers such as CI jobs, monitoring tools and data catalogs add load in different ways, so the first step is measuring both. Running on Kubernetes with the official Helm chart or on virtual machines changes how replicas, health checks and secrets are wired. If you are still on Airflow 2, the webserver-to-API-server change and the separate DAG processor come first, and that work usually dominates the schedule. Ask any partner to split their estimate into these four drivers so you can see which one moves the total.
Can our own platform team scale the Airflow API server without a partner?
Yes, if your team already runs Linux services, a reverse proxy and a PostgreSQL or MySQL metadata database, and can read Airflow metrics. The configuration changes are modest: worker count, gunicorn behavior on Airflow 3.2 and later, shared secret keys and load balancer timeouts. The harder part is proving the change under realistic traffic, which needs a load test plan, a staging copy of the environment and someone who can interpret the results. Teams that have not run a load test before often choose a fixed-scope engagement for that step only, then operate the result themselves. Your team should decide this based on whether it has the time and the skills for the verification work.
What access and inputs do we need to provide for an API server scaling project?
Expect to provide access in four areas. First, read access to the current API server metrics, request latency and error rates from your monitoring stack. Second, a staging environment that matches production closely enough to test, plus the deployment configuration for the API server and load balancer. Third, read access to the metadata database for connection counts and query load, and a change window for any database work. Fourth, a source of realistic test traffic, such as recorded UI sessions or sanitized API call patterns, and permission to run it against staging. Also confirm which team owns the shared secret key and log-fetch clock settings, because those affect replicas directly.
How long does an API server scaling project take?
Duration depends on scope, not on a fixed rule. A narrow change, such as setting worker counts and secret handling on an existing single-instance deployment, is mostly configuration and verification. Adding horizontal replicas behind a load balancer, moving to gunicorn recycling, and load testing takes longer because each step needs staging proof. A migration from Airflow 2 with a webserver replacement and a separate DAG processor adds the upgrade path and its testing on top. Ask for a written timeline with separate milestones for staging, load testing, production cutover and handover, and check which items depend on your change approvals or your access requests.
What does an API server scaling engagement cost, and what should a quote include?
DeployOpen does not publish prices. A quote follows from a written scope, and most engagements are fixed-scope with deliverables and timelines agreed before work starts. Cost is driven by the number of environments, the upgrade starting point, the size of the load test, and whether the work includes documentation and handover. Ask any provider for one-time items (assessment, configuration, load testing, cutover support) separated from recurring items (monitoring review, upgrades, retainer support). A quote should name the environments in scope, the acceptance tests, who performs the load test, and what happens if staging results do not match production. Internal cost is mainly engineering time and change approvals.
Do API server scaling changes need a change record?
In most regulated environments, yes. The changes covered here alter a production-facing service: worker counts, gunicorn settings, the shared API secret key, replica count and load balancer timeouts. Each one can affect user sessions, task log retrieval and API clients, so your change process should record the planned change, the rollback step, the test evidence and the approver. Rotating the secret key in particular needs coordination, since existing sessions and components may depend on it. Check with your change advisory board or internal policy owner for the exact record format, because requirements differ between organizations.
What goes wrong when API server scaling is done without load testing?
The most common problems are not visible until real traffic arrives. Too few workers make the UI slow for many users. Replicas with different secret keys cause sessions and log requests to fail. Unsynchronized clocks across hosts can lead to forbidden errors when task logs are fetched. Load balancer idle timeouts shorter than long API calls cut connections mid-request. Without a load test on staging, you find these during an incident. Ask any partner how they will reproduce your traffic pattern and which metrics define a pass, such as latency at peak concurrency and error rate during a worker recycle.
Does DeployOpen handle ongoing API server operations after the scaling work?
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including Airflow. Ongoing support, upgrades and continuous testing are available as retainers, so the operating work after cutover can be handled by your team, by DeployOpen, or split between them. Whichever you choose, agree in writing who watches the API server metrics, who rotates the secret key, who applies Airflow patch releases, and who runs the load test again before the next major change.
Security and operational risks of a scaled API server
Scaling the API server adds more ways for it to be reached, more places where secrets live, and more reasons to act on the Airflow 2 end of life. The cards below cover the controls to set before replicas go behind a load balancer, and the timing constraints that apply to the platform as a whole.
Load balancer exposure
Every replica behind the load balancer is reachable by whoever can reach that address, so the API server is only as protected as the network path in front of it. Decide which networks can reach the UI and REST API, restrict the scheduler and workers to internal routes, and confirm that the health check endpoint does not expose more than it should. Ask your security team to review the listener rules before the first production replica is added.
TLS termination and headers
Most deployments terminate TLS at the load balancer, which means the forwarded headers the API server trusts must come from that proxy and nowhere else. Confirm which proxy headers the API server honors, strip client-supplied copies at the edge, and check that cookies and redirects use the external hostname and scheme. Test sign-in and log viewing through the same path your users take.
Secret key management
The API secret key signs values the components exchange, so it must be set once as a static value through a Kubernetes Secret, and it must differ between Airflow instances. A key shared across environments lets a token from staging be replayed against production. Store it in your secrets manager, restrict who can read the Secret object, and plan a rotation window that restarts every API server replica together.
Log access and audit
Task logs are fetched through the API server, so a user who can read logs can read whatever the task printed, including values that were not masked. Map who can open task logs and configuration views through your auth manager roles, and forward API server access logs to your central log store with the replica name attached. Also check that clocks are synchronized on every node, because a clock skew can produce forbidden errors when logs are fetched.
Upgrade timing and the Airflow 2 end of life
Airflow 2 entered limited maintenance on October 22, 2025, and reached end of life on April 22, 2026. Limited maintenance covers only security and critical bug fixes, and end-of-life versions get no fixes or support. Any API server running Airflow 2 today is exposed to issues that will not be patched, so plan the move to Airflow 3 and check the upgrade path before adding replicas.
Patch cadence for Airflow 3
Airflow 3 is supported as an ongoing line with patch releases, and the current patch release is 3.3.2. Ask who will apply patches to the API server image, how quickly a critical fix reaches production, and whether the replica recycling settings allow a rolling restart without downtime. Record the running version on every replica so that the security review can match the deployed build to published fixes.
How the API server fits with the scheduler and DAG processor
Scaling the API server only changes how the UI and REST API handle requests. In Airflow 3 the webserver is replaced by an API server started with airflow api-server, and the DAG processor must run as a separate component started with airflow dag-processor, as the Airflow 3 upgrade guide describes. The scheduler, the DAG processor and the API server are therefore three processes with different loads, so each needs its own sizing decision. Adding API server replicas does not reduce scheduling delay, and it does not speed up DAG file parsing.
Each component should be sized and tuned on its own evidence. Scheduler throughput is governed by scheduling loops and database load, and parsing time is governed by the DAG files themselves, so those are the areas to review when runs start late or DAGs appear slowly. The API server becomes the bottleneck when operators see slow page loads, slow task log retrieval or timeouts from automation that calls the REST API. Before changing any replica count, confirm which component is slow using the metrics and logs your team already collects, and record the baseline so you can tell whether a change helped.
Components also depend on shared state. All three read and write the metadata database, and task log retrieval passes through the API server, which means the shared secret key and synchronized clocks affect it directly. Plan the database connection budget across all three components together, not one at a time, and confirm that your database and any connection pooler can handle the combined load. Treat the scheduler and DAG processor as fixed-size infrastructure to tune, and treat the API server as the component you scale out once the data shows that user and API requests are the constraint.
How DeployOpen works on this
DeployOpen scopes the Airflow 3 API server as a fixed piece of work inside your own infrastructure, starting from your measured UI and API load, replica count, database and load balancer. The written scope sets out what gets sized, built, tested, documented and handed over, so the decisions behind the final configuration stay traceable to your environment.
How we scope
- Review current UI and API traffic patterns, peak user counts and the automation that calls the API.
- Confirm the Airflow version, deployment target and whether the API server runs on Kubernetes or virtual machines.
- Agree in writing which replica count, gunicorn worker settings, shared secret key handling and load balancer configuration are in scope.
How we build
- Configure the API server with a static secret key supplied through a Kubernetes Secret, unique to each instance.
- Set worker counts and recycling behavior for the Airflow version in use, testing each change against representative load.
- Check that clocks are synchronized across API server, scheduler and workers so task log fetching does not return forbidden errors.
How we hand over
- Document the sizing basis, chosen settings, load balancer health checks and the reasons behind each value.
- Provide a runbook for adding replicas, recycling workers and rolling out API server upgrades without downtime.
- Walk your platform and operations teams through the configuration and the load test results before handover.
How we support
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
- Ongoing support, upgrades and re-sizing reviews are available as a retainer after handover.
- Confirm the support boundaries, response expectations and upgrade path for your Airflow version in the written scope.
