Why PgBouncer connection pooling matters for Apache Airflow
Every Airflow component that talks to the metadata database opens its own connections, so the count climbs as you add workers and schedulers. Apache Airflow is an Apache Software Foundation project released under the Apache License 2.0 (SPDX: Apache-2.0), with no paid edition from the project itself. PgBouncer is a separate open-source connection pooler that your team installs, configures and operates alongside PostgreSQL. Teams usually need it once several Airflow 3 components, such as the API server, DAG processor, scheduler, triggerer and task workers, share one PostgreSQL instance and the connection total starts to crowd out other clients.
The Airflow Helm chart production guide advises an external metadata database and recommends PgBouncer to reduce the number of database connections. Airflow 3 splits work across more processes than Airflow 2 did, and every additional scheduler adds to the total, so the gap between idle and peak connection counts widens as the deployment grows. Before you commit to a design, confirm how many connections each component opens at your planned worker and scheduler counts, and check the maximum connection limit your chosen PostgreSQL version and hosting platform allow. Whether that count is a problem for you depends on those numbers, which your own environment will determine.
The decision is not only about performance. A pooler adds a hop, a set of credentials and a service that can fail, so the team that runs it needs to own its sizing, upgrades and monitoring from day one. Plan the pool size with your scheduler and worker counts in mind, and agree who will respond when pooled connections are exhausted.
Helm chart PgBouncer settings and when each fits
This table maps each PgBouncer-related setting in the Airflow Helm chart to what it limits and the inputs that decide its value. Read it before you set the connection string, because the secrets and pool sizes only work together when the connection path runs through PgBouncer. Confirm exact key names and defaults against the chart version you deploy.
| Setting | What it limits or does | What drives the right value |
|---|---|---|
| Connection string | Points Airflow components at PgBouncer instead of PostgreSQL directly, so all metadata traffic passes through the pooler. | Every component that talks to the metadata database must use it, including API server, scheduler, DAG processor, triggerer and workers. Check that no component is left on a direct connection. |
| maxClientConn | Caps how many client connections PgBouncer accepts from Airflow components at once. | The number of scheduler replicas, workers, API server pods and their process counts. Size it so that a normal scale-out does not hit the cap, and confirm the value against your PgBouncer and PostgreSQL limits. |
| metadataPoolSize | Sets the number of server connections PgBouncer keeps open to PostgreSQL for metadata database work. | Concurrent metadata queries from schedulers and the API server, and the PostgreSQL connection limit you can spare for other clients. Raise it only after monitoring shows client waits. |
| resultBackendPoolSize | Sets the server connection pool used for the result backend, separate from the metadata pool. | Whether your result backend uses the same PostgreSQL instance, and how many tasks report results at peak. Ask your team to confirm the result backend choice before tuning this value. |
| airflow-pgbouncer-config secret | Holds PgBouncer configuration that the chart reads at startup. | Keep it in a Kubernetes Secret rather than values.yaml so credentials and pool settings follow your secret rotation process. |
| airflow-pgbouncer-stats secret | Holds the credentials the chart uses for PgBouncer statistics, which feed the metrics endpoint. | Access controls on the monitoring namespace and the metrics scraper. Restrict who can read it, since it grants access to pooler statistics. |
How to set up PgBouncer for Airflow on Kubernetes
This sequence takes you from an external PostgreSQL metadata database to a Helm-deployed Airflow that routes its connections through PgBouncer, with a check at each stage. Each step produces a verified component so that a failure can be traced to one layer.
Provision an external PostgreSQL metadata database
Your platform or database team creates a PostgreSQL instance outside the Airflow release, in a supported major version listed in the Airflow prerequisites for your Airflow version. The Helm chart's embedded Postgres is suitable for testing only, so do not use it as the metadata database for production. Confirm the database, role and network path from the Kubernetes namespace where Airflow will run. The step is done when a client pod in that namespace can open a session to the database with the intended role and run a simple query.
Create the Kubernetes Secret for database credentials
Create a Kubernetes Secret in the Airflow namespace that holds the metadata database credentials, using a strong password generated outside version control. Keep the connection string itself in the Secret rather than in values.yaml, so the password never appears in Helm values or the rendered manifests. Ask your security team to confirm who can read Secrets in that namespace, because these credentials grant write access to Airflow state. The step is done when the Secret exists, its keys match what your chart values reference, and a test pod can read the value without printing it in logs.
Create the PgBouncer config and stats secrets
Create the airflow-pgbouncer-config secret, which holds the PgBouncer configuration that points the pooler at your PostgreSQL host and sets pool behavior, and the airflow-pgbouncer-stats secret, which holds the credentials the metrics exporter uses to read PgBouncer statistics. Use the secret names the Helm production guide expects, because the chart looks them up by name. The step is done when both secrets exist in the Airflow namespace and a short-lived debug pod can list their keys without errors.
Point the connection string at PgBouncer
Update the Airflow connection configuration so that the metadata connection targets the PgBouncer service in the cluster rather than PostgreSQL directly. The database host changes to the PgBouncer service name, while the database name and role stay the same as the upstream database. Confirm that the PgBouncer pool is configured to authenticate the role you are using. The step is done when the rendered configuration shows the PgBouncer service as the metadata host and no pod still references the PostgreSQL host directly.
Set the pool values in the Helm chart
Set maxClientConn, metadataPoolSize and resultBackendPoolSize in the chart values to match the number of Airflow components and their connection demand. maxClientConn caps how many clients PgBouncer accepts, the pool sizes cap server connections to PostgreSQL, and both must fit within the connection limit your database team has allocated. Record the chosen values and the reasoning, because they will need review after scaling the scheduler, workers or API server. The step is done when a helm template render shows these values in the PgBouncer configuration and the total server-side pool size fits within the database limit.
Deploy the release and confirm each component connects
Deploy or upgrade the Airflow Helm release, then check each component in turn: the migration job, the scheduler, the API server, the DAG processor, the triggerer and the PgBouncer pod. Confirm in the Airflow logs that each one completes its database connection, and confirm in PostgreSQL that sessions originate from the PgBouncer pod rather than from individual Airflow pods. Run one test DAG to confirm that scheduling and task state updates succeed through the pool. The step is done when every component reports a healthy database connection and the test DAG completes.
Verify the pooler metrics and alerts
Scrape the Prometheus metrics that the chart exposes for PgBouncer and confirm that the exporter can read the stats secret. Watch client connections, server connections in use, waiting clients and any reported errors during a normal scheduling window and during a deliberate scale-up. Ask your operations team which thresholds they will alert on, since the right values depend on your pool sizes and workload. The step is done when the metrics appear in your monitoring system, a dashboard shows the pool state, and an alert rule has been tested against a simulated condition.
Pool sizing choices for test, staging and production
A test cluster usually runs the embedded PostgreSQL that the Helm chart provides. The chart presents that database as suitable for testing only, so it is a poor place to measure pool behavior. Connection counts stay low because few components run and workloads are small, which means PgBouncer adds little value here and mostly adds one more moving part to debug.
- Pooling is optional: Keep the setup simple so engineers can learn the chart and the component layout first.
- Embedded database limits: Treat the embedded PostgreSQL as disposable, with no expectation of backups, replication or upgrade testing.
- Light monitoring: Basic pod health and Airflow task outcomes are usually enough at this stage.
- Settings are for learning: Any pool values set here should not be copied forward without a load test.
- Data is not representative: Run history and table sizes will not match production, so cleanup and query timings from this environment are unreliable.
Before you move on, confirm with your platform team which PostgreSQL version and chart version the embedded database uses, and record that the environment is not used for pool sizing decisions.
Staging should use an external PostgreSQL instance that matches production's engine and major version, with the connection string pointed at PgBouncer the same way production will be. This is the first place where pool values can be tested against real Airflow component behavior. Staging rarely matches production in worker count, so the goal is to validate configuration paths and failure modes, not to reproduce peak load.
- Same wiring as production: Use the same secrets layout, connection string format and PgBouncer deployment shape.
- Reduced replica counts: Run one scheduler and a small set of workers so that connection math can be checked on paper against the pool limits.
- Settings under test: Exercise
maxClientConn,metadataPoolSizeandresultBackendPoolSizewith values you plan to carry forward, and record each change. - Monitoring turned on: Enable the PgBouncer Prometheus metrics that the chart ships so that client wait and pool saturation can be seen before production.
- Failure drills: Restart PgBouncer and the database during a run to confirm that Airflow reconnects cleanly.
Before promoting values, confirm with your database team which PostgreSQL version is in use, how the staging instance is backed up, and whether the connection limit on the server allows the planned pool sizes.
Production is where connection counts grow fastest, because each scheduler, API server, DAG processor, triggerer and worker process can open its own connections to the metadata database. Airflow supports more than one scheduler at a time using database row-level locks, so adding schedulers for resiliency or throughput also adds connection demand. Pool sizes in production should come from a written calculation of expected components, replicas and pool limits, then be confirmed with load testing.
- Connection budget first: Count every component that talks to the database, multiply by replicas, and compare the total with the server connection limit before choosing pool sizes.
- Scheduler growth: Recheck the budget each time the scheduler replica count changes, because the connection load scales with it.
- Alerts on saturation: Alert on client wait time and on pools that stay at their configured maximum, using the PgBouncer metrics.
- Database health: Watch active connections, lock waits and replication lag on the PostgreSQL side, since pooling does not remove database work.
- Change control: Treat pool settings as configuration changes with review, rollout notes and a rollback value.
Before go-live, confirm the expected replica counts, the database connection limit and the on-call owner for pool alerts, and write these down in the runbook.
Securing pooled connections and their credentials
A pooler adds another place where database credentials, authentication settings and connection traffic meet. The cards below cover where secrets live, which authentication method PostgreSQL should use, whether traffic between PgBouncer and PostgreSQL is encrypted, and who can read the secrets the Helm chart creates.
Keep credentials in Kubernetes Secrets
Store the metadata connection string and the PgBouncer settings in Kubernetes Secrets that the Airflow Helm chart references, not in values files committed to Git. The production guide describes a setup where the connection string points at PgBouncer and the chart reads the airflow-pgbouncer-stats and airflow-pgbouncer-config secrets. Check that no rendered manifest or CI log exposes the password, and plan how a rotated database password reaches PgBouncer and Airflow together.
Prefer SCRAM over MD5
Set PostgreSQL password authentication to SCRAM for the Airflow and PgBouncer roles. PostgreSQL 18 deprecated MD5 password authentication in favor of SCRAM, so new roles should not be created with MD5 hashes. Confirm the password_encryption setting and the pg_hba.conf rules that PgBouncer uses to log in to the database, because a pooler that authenticates with an older method can block a migration to newer PostgreSQL versions.
Choose the authentication method per path
PostgreSQL documents several client authentication methods, including password, GSSAPI, LDAP, certificate and OAuth. Decide which method each client path uses: Airflow components connecting through PgBouncer, PgBouncer connecting to PostgreSQL, and administrators connecting directly. Record the choice in the design, because each method needs its own identity source, rotation process and failure mode.
Encrypt PgBouncer to PostgreSQL traffic
Confirm whether the connection between PgBouncer and PostgreSQL uses TLS in your environment, and whether the PgBouncer version and the PostgreSQL server configuration both support the chosen TLS mode. Traffic between Airflow pods and PgBouncer usually stays inside the cluster, but it still carries credentials on the first connection. Ask the Helm chart maintainers and your platform team which hops are encrypted and which certificate authority each one trusts.
Limit who reads config and stats secrets
Anyone who can read the PgBouncer config or stats secret in a namespace can often read the database password it contains. Restrict get and list on those secrets with Kubernetes RBAC, keep Airflow workers and task pods on separate service accounts, and avoid giving broad namespace read access to dashboard or monitoring tools. Ask the cluster team which roles currently have read access to the Airflow namespace.
Confirm audit trails for secret access
Confirm that Kubernetes audit logging records reads of the pooler secrets, and that someone reviews those records on a fixed schedule. Also confirm who can change PgBouncer settings and whether those changes leave a record in Git or the cluster audit trail. Ask your security team to confirm the retention period and the owner of the review before the pooler goes to production.
Questions teams ask before adding PgBouncer
Do we need PgBouncer for a small Airflow installation?
Not always. PgBouncer matters most when the number of database connections grows with the number of schedulers, API server replicas, workers and pods, because each of those components opens its own connections to the metadata database. A small installation with one scheduler, one API server and a handful of workers may run well without it. The Helm chart production guide recommends PgBouncer to reduce database connections, so the question for your team is whether your connection count is approaching what your PostgreSQL instance can comfortably serve. Check current connection counts in your environment before deciding.
What drives the effort of adding PgBouncer?
The main drivers are the shape of your deployment and how much of the surrounding process already exists. Effort rises with the number of schedulers, workers and API server replicas, the number of environments, how credentials are stored and rotated, and whether metrics and alerting are already wired to Prometheus. Teams with a Kubernetes Secrets process, an external PostgreSQL instance and existing dashboards have less to design. Teams that still keep the embedded PostgreSQL, store passwords in values files or lack monitoring need more work before pooling is safe to change.
What does the work cost, and what makes the quote go up or down?
DeployOpen does not publish prices. A quote follows from a written scope, and most engagements are fixed-scope. The cost components are design of the pool settings, changes to the Helm values and secrets, testing of failover and connection behavior, monitoring setup and documentation for your operators. Costs are largely one-time, while ongoing tuning, upgrades and any retainer for support recur. Ask for the quote to list each deployment environment, the number of components that connect to the database, the secrets process and the monitoring stack it must integrate with.
How long does a PgBouncer setup take?
Duration depends on scope, and no timeline is given here. A change limited to one cluster with an existing secrets process and dashboards is much shorter than a multi-environment rollout that also covers secret rotation, metrics and runbooks. Ask for a timeline tied to the written scope, and confirm which environments, approvals and change windows the schedule must respect. Production changes to the metadata connection usually need a planned maintenance window, which should be part of the plan from the start.
What must our team provide before the work starts?
Expect to provide access and context rather than only requirements. You will need the current Airflow and PostgreSQL versions, the Helm chart version and values files, the cluster topology, the count of schedulers, workers and API server replicas, the existing secrets management approach, and access to your monitoring stack. You should also name the people who approve database changes and the window in which they can be made. Confirm the supported PostgreSQL versions for your Airflow release in the prerequisites before choosing a database version.
Can our own team set up PgBouncer with the official Helm chart documentation?
Yes, a team with Kubernetes and PostgreSQL experience can follow the official Helm chart production guide. The guide covers the external metadata database, the connection string pointed at PgBouncer, the airflow-pgbouncer-stats and airflow-pgbouncer-config secrets, and the pool settings maxClientConn, metadataPoolSize and resultBackendPoolSize. The harder parts are sizing the pool against your database limits, testing behavior under load and failover, and keeping the setup stable through upgrades. Decide which of these your team can own before choosing between self-delivery and a partner.
Which metrics should we watch after adding PgBouncer?
Watch the connection pool and database metrics together. The Helm chart ships Prometheus metrics for PgBouncer, so start with client connections, server connections, waiting clients and pool saturation, then compare them with PostgreSQL connection counts and scheduler or task errors. Also track query latency and any errors from connection timeouts. Set alerts that fire before the pool is exhausted, and review them after each Airflow or chart upgrade. Confirm the exact metric names in your chart version before building dashboards.
How DeployOpen scopes and supports Airflow connection pooling
DeployOpen sells PgBouncer pooling work as a fixed-scope engagement with the scope, deliverables and timeline agreed in writing before any change is made. For this topic, the written scope should name the Airflow components that will connect through the pooler (API server, scheduler, DAG processor, triggerer and workers), the PostgreSQL version and hosting model, the Helm chart version in use, and the pool values to be set, including maxClientConn, metadataPoolSize and resultBackendPoolSize. Pooling touches every component that talks to the metadata database, so a scope that covers only the scheduler will leave the connection count problem in place. Ask for the scope to list each component and the test that proves its connections reach PgBouncer.
The work happens inside your infrastructure, in your Kubernetes cluster, your PostgreSQL servers and your monitoring stack. DeployOpen does not need to hold database administrator credentials beyond what the agreed change requires, and secrets for the pooled connection string are created in your secrets store rather than in a values file. Before handover, the deliverables should include a record of the final pool settings, the airflow-pgbouncer-stats and airflow-pgbouncer-config secrets with their ownership, the PgBouncer metrics dashboards and alert thresholds, and a load test report that shows connection counts under expected scheduler and worker concurrency. Confirm with your team who owns each of these artifacts after the engagement closes.
Ongoing work is handled through a retainer, which covers support, Helm chart and Airflow upgrades, and periodic review of pool sizing as DAG counts and worker counts change. Retainers are priced from the agreed scope, and DeployOpen does not publish a price list, so ask for the retainer terms in writing with the response times and the upgrade windows they include. DeployOpen prepares systems and evidence for your reviews, but independent auditors and certification bodies issue any formal assessment outcome. Ask any partner, including DeployOpen, to show how they will verify pool behavior against your own environment rather than against a generic example.
