Apache Airflow Edge Executor for remote workers

What the Apache Airflow Edge Executor does for remote workers

The Apache Airflow Edge Executor runs task execution on remote workers that connect back to the central Airflow site over HTTPS, so the central network does not need inbound connections to the machines that read or write data. This suits pipelines that must reach systems in other data centers, clouds or restricted networks where opening inbound ports is not acceptable. EdgeExecutor is one of the executors Airflow ships, and it is delivered through the Edge provider package. The Airflow project is released under the Apache License 2.0 and has no paid edition, so the software cost is zero and the cost lies in the design, build and operation of the remote workers.

Before you commit to this pattern, check the executor documentation for the Airflow version you run. Airflow 3 changed several components, including the webserver, which is replaced by the API server, and the DAG processor, which must run as a separate component. The way remote workers authenticate, fetch work and report state depends on those components, so the configuration you read for Airflow 2 may not apply. Confirm which provider package version matches your Airflow release, and confirm the supported Python and database versions in the prerequisites for that release before you draft an architecture.

Decide early which sites will run workers, which queues each site will serve, and what each remote network may reach outbound. Those answers determine the outbound HTTPS endpoints, the credentials each worker holds and how you will recover a site that loses connectivity mid-run. Ask your team to test one remote site end to end, including a deliberate network interruption, before you extend the pattern to others.

Choosing EdgeExecutor, Celery or Kubernetes for each environment

When every worker can reach the message broker, the metadata database and log storage over your internal network, CeleryExecutor is the usual choice. Celery distributes tasks to a fleet of long-running workers that pull work from a broker, so the network requirement is simple: the scheduler, workers, broker and database must have stable connectivity inside one trust zone. Most of the operational load sits in running the broker and scaling the worker fleet.

Network path: Workers need direct reachability to the broker, the result backend and the metadata database. Confirm the connection settings for your version in the executor documentation.
Broker operations: The broker is a stateful service with its own backups, failover and patching.
Worker capacity: Your team sets worker counts and decides how and when to scale them.
Where it fits: Single-site estates with steady task volume and no existing container platform to run tasks on.

Check that the broker and database are reachable with low latency from every worker, because each task depends on that path.

Configuration options and when each one fits

This table lists the decisions a team makes before building an Edge deployment, with the option each decision usually leads to, its trade-offs and the situations where it fits. Use it to settle the design with your platform and security teams, then confirm exact setting names in the Edge provider documentation for your Airflow version.

Decision	Option	Trade-off	When it fits
Worker authentication	Dedicated credentials per remote site or worker pool	Each credential needs issuance, rotation and revocation; confirm the exact mechanism in the provider docs	Several sites with different owners or trust levels, where one site's compromise must not expose the others
Worker authentication	Shared credential for all remote workers	Simpler to deploy, but one leaked credential reaches every remote worker	Small, single-owner estates where the workers sit in one trust zone
Where the worker process runs	Virtual machine or host inside the remote network	You patch and supervise the host and its service yourself	Sites with no container platform, or where hosts are already managed
Where the worker process runs	Container on an existing remote cluster	Needs a container runtime and image delivery into the restricted network	Sites that already run containers and have an internal image registry
Queue or pool mapping to sites	One queue per site, tasks routed by queue name	DAG authors must choose queues correctly; misrouted tasks wait or fail	Tasks that must run where the data lives, such as in-country or on-premises sources
Queue or pool mapping to sites	Pools to cap concurrency per site	Pools limit parallel tasks but do not decide placement by themselves	Protecting a remote site's capacity or a source system's connection limits
DAG code delivery	Git-based bundle pulled by the remote site	Needs outbound access to Git through an approved path; check the bundle type for your version	Teams that want commits recorded per run and a single source of code
DAG code delivery	Object storage or synced file copy	Sync failures can leave sites on different code; add checks for version drift	Sites where Git access is blocked but a storage service is reachable
Log return	Logs streamed back to the central site over HTTPS	Depends on connectivity and log volume; confirm retention and buffering behavior	Smaller log volumes and central teams that need one place to investigate
Log return	Logs kept at the remote site with a separate retrieval path	Investigation needs access to the remote site; plan retention and access control	Logs containing data that must stay inside the restricted network

Running EdgeExecutor next to Celery or Kubernetes in one deployment

Since Airflow 2.10.0, several executors can run in one deployment, so a team can keep central tasks on CeleryExecutor or KubernetesExecutor and send only selected tasks to EdgeExecutor workers in a remote data center, cloud account or restricted network. The practical gain is that the remote site is the only place that needs a route to the central Airflow site, and that route is outbound HTTPS from the edge workers. Tasks that must run beside a source system, such as a plant historian or a regional database, can stay there, while the scheduler, metadata database and the rest of the pipeline remain where your platform team already operates them. Before committing, confirm in the Airflow executor documentation how your installed version assigns tasks to an executor, whether per task or per DAG, and write that rule down as a design decision rather than leaving it to DAG authors.

The routing rule is the part most likely to fail quietly. A task sent to the wrong executor may still run, but on a worker that lacks the network path, credentials or libraries it needs, and the failure may appear only at the first run that touches the remote system. Plan a routing test set that runs in CI and in staging: one DAG per executor that checks the executor a task was assigned, a run against each worker pool, and a check that central tasks never land on edge hosts. Keep the list of which DAGs use which executor under version control so a change to the routing rule shows up in review.

Running two worker types adds operating cost that recurs for as long as the deployment exists. Celery or Kubernetes workers and edge workers have different patch cycles, monitoring signals and scaling behavior, so the team needs a runbook for each, alarms that show when an edge worker has stopped reporting, and a capacity plan for each pool. Ask your implementation partner to price the routing tests, the second monitoring set and the on-call runbook as separate line items in the written scope, so you can see what the split costs compared with moving all tasks to one executor.

How to set up the central site and remote workers

This sequence takes a team from a supported Airflow 3 baseline to a central site that accepts tasks from remote workers over HTTPS, with a test DAG proving logs return to the UI and a failure test showing how the setup behaves when the link drops. Each step ends with a check so problems surface before the next layer is built.

Confirm the Airflow version and prerequisites
The platform engineering lead confirms the target Airflow 3 patch release against the version life cycle and checks it against the prerequisites for Python, the metadata database and Kubernetes if used. The central site needs PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or an Innovation release, and MariaDB and SQLite are not suitable for production. Done when the target versions are written down, the metadata database is provisioned and reachable, and a dry-run install completes in a non-production environment. Ask DeployOpen or your own team to confirm that the Edge provider release you intend to use lists the chosen Airflow release as compatible.
Deploy the central components as separate services
The platform team deploys the API server, the scheduler and the DAG processor as separate components, because Airflow 3 no longer uses the webserver and requires the DAG processor to run on its own. Each component gets its own process or container, shared configuration and access to the same metadata database. Done when all three components start cleanly, the scheduler reports healthy against the database, and the UI loads through the API server. The scheduler documentation supports running more than one scheduler for resiliency, which you can decide on later once the base deployment is stable.
Install and pin the Edge provider
The Airflow platform engineer installs the Edge provider package on the central site and on each remote worker image, pinned to a version that matches the Airflow release in use. Mismatched provider and core versions are a common source of failures in multi-site setups, so the pins should be kept in the same configuration source as the rest of the Airflow environment. Done when the executor appears as available in the central configuration, a worker image with the matching provider starts, and the versions are recorded in your change log. The exact package names, commands and settings must come from the provider documentation for your release.
Configure the central HTTPS endpoint and certificates
The network and security team publishes the central site's Edge endpoint over HTTPS, issues a certificate from an authority that remote workers trust, and restricts inbound access to the workers' source networks. Because remote workers reach the central site over HTTPS only, the certificate chain, hostname and proxy behavior must be tested from each remote network before any worker is enrolled. Done when a client on each remote network completes a TLS handshake to the endpoint, certificate expiry and renewal are recorded in your operations calendar, and the endpoint rejects connections from outside the approved networks.
Enroll and authenticate each remote worker
The platform team enrolls each remote worker using the provider's enrollment and authentication mechanism, giving each worker its own identity so access can be revoked individually. Record which queue or worker name each site uses, so DAG routing can target it later. Done when every worker appears as registered in the central site, a revoked or unregistered worker cannot pick up tasks, and the credentials are stored in your secrets store rather than in DAG code or plain configuration files. Confirm the authentication options and token lifetimes with the provider documentation, since these settings change between releases.
Deliver DAGs and dependencies to remote sites
The data engineering team sets up how DAG code and Python dependencies reach each remote worker, using a delivery method that matches the site's network rules, such as a pinned container image or a synchronized bundle. The chosen method should ensure that a worker runs the same code version the central scheduler expects, so the team agrees on which version a retry uses. Done when a DAG change published centrally is visible at the remote worker, the dependency set matches the one tested in staging, and the delivery process is documented in the same place as the rest of your Airflow runbooks.
Run a test DAG on one remote worker
The data engineering team writes a small test DAG that routes a single task to one named remote worker and prints a known message, then triggers it from the UI. This confirms the scheduler, the Edge endpoint, the worker and the log path all work together. Done when the task reaches success on the intended worker, its logs appear in the UI, and the run record shows the worker that executed it. If logs are missing, check the log storage configuration on the remote side before changing the executor settings.
Add monitoring and test a dropped link
The operations team adds monitoring for worker heartbeats, scheduler health and endpoint certificate expiry, then runs a controlled failure test by cutting the network path between one remote worker and the central site during a test DAG run. The goal is to observe how tasks are marked, whether retries behave as expected and what operators see in the UI and alerts. Done when the outage is detected by monitoring, the affected tasks end in a state your runbook describes, and tasks resume or retry correctly once the link is restored. Record the observed behavior as the baseline for future changes, and ask the provider documentation or DeployOpen to confirm any failure-handling settings before relying on them in production.

Securing HTTPS-only worker links, credentials and secrets

Remote Edge workers pull work and report results across networks you do not fully control, so every link, identity and secret on that path needs a deliberate owner. The cards below cover the controls that connect the central site to each remote network and the points where a mistake exposes your pipelines.

TLS certificate lifecycle and trust

The central API endpoint is the only door remote workers use, so its certificate chain must be trusted by every remote site without disabling verification. Decide who issues the certificate, how renewal is automated, and how remote hosts receive the trust bundle. Check that an expired or rotated certificate causes workers to stop cleanly and alert, rather than silently losing contact with the central site.

Worker identity and credential rotation

Each remote worker should authenticate as its own identity so that a compromised host can be revoked without touching the others. Plan how credentials are issued at registration, how they rotate on a schedule you set, and how a decommissioned site loses access immediately. Confirm the rotation mechanism and its lifetimes with the version of the Edge provider you deploy.

Least privilege for each remote network

A worker at one site should only see the queues, pools and tasks assigned to that site, not the whole deployment. Map which DAGs and task types each remote network may run, and remove broad administrative permissions from worker identities. Review this mapping whenever a new site or DAG is added, because scope tends to grow quietly.

Secrets delivered without touching DAG code

Remote workers often need database passwords, API tokens and certificates that must never appear in DAG files, logs or the metadata database. Use a secrets backend that the worker can reach from its own network, and keep only references in DAG code. Ask which secrets each site truly needs, because every secret a worker holds is available to any code that runs on it.

Egress rules and proxy handling

Remote sites usually allow outbound traffic only through a corporate proxy or a fixed allowlist. Document every destination a worker must reach, including the central HTTPS endpoint and any data sources, and confirm that proxy settings apply to the Airflow worker process and not only to the operating system. Test the rules from a host in each network before any production DAG runs there.

Audit logging of registration and execution

Record when each worker registers, which identity it uses, which tasks it claims and how it finishes. These events are your evidence when a site is questioned or a run is disputed, so send them to a central log store that the remote site cannot edit. Confirm with your security team how long these records must be kept before you set retention.

What remote Edge workers cost and what they need from your team

What drives the cost of an Edge Executor deployment?

The main cost drivers are the number of remote sites, how long network and firewall approvals take at each one, the certificate and identity work needed for each worker to authenticate to the central site, the testing needed across failure modes, and the ongoing patching of remote hosts. Each site adds its own network path, host image and access checks, so cost usually grows with the count of sites and their differences. Ask for the written scope to list each site, each approval path, each identity integration and each failure case the test plan covers, with a separate line for ongoing patching and support. DeployOpen does not publish a price list, and a quote follows from that written scope.

Can our own team build an Edge Executor setup without outside help?

Yes, a capable platform team can build it, provided it has Airflow operations experience and access to the remote networks. The work involves installing the Edge provider, configuring workers to reach the central API over HTTPS, managing worker credentials, deploying workers to each remote host, and testing task failure, network loss and restart behavior. Teams that have not run distributed executors often underestimate the failure testing and the change control around remote hosts. A partner is most useful when the sites are many, the approvals are slow, or the team cannot staff the testing and patching work. Confirm the Edge provider's current documentation for the supported configuration before you commit to a design.

How long does setup depend on network approvals and environment access?

Setup time is driven mostly by the environment, not by Airflow itself. Firewall change windows, security reviews of outbound HTTPS paths, access to remote hosts, and approval of new identities can each take longer than the installation. Ask for a written timeline that separates the approval steps from the build steps and names who owns each approval. Also ask which steps can run in parallel across sites and which must wait for a single change window. DeployOpen will not give a duration until the sites, approvals and access model are known, because any figure before that would be a guess.

What does the work need from our team before it starts?

The work needs several inputs from your side before it can begin:

Network diagrams showing each remote site, the central Airflow site, and the HTTPS paths between them
Firewall change windows and named owners who can approve outbound rules
Access to the identity system that will issue worker credentials, and to the secrets store where they are kept
A test pipeline and sample DAGs that exercise each remote site, including failure cases
Host access for installing and patching worker machines

Collect these before scoping starts, since missing inputs are the most common reason for delay. Confirm each item in the written scope so responsibilities are clear.

How does Edge Executor compare with a VPN or direct inbound access to workers?

Edge Executor lets remote workers pull work from the central Airflow site over HTTPS, so the central site does not need to open inbound connections into each remote network. A VPN or direct inbound access gives the scheduler a network path to workers, which can work well inside a single trusted network but usually needs more firewall changes and broader network trust. The right choice depends on your network policy, how many sites you run, and whether your security team accepts outbound-only worker connections. Ask your network and security teams to review both models against your policy, and confirm the security properties of the current Edge provider with its documentation rather than assuming them.

What does ongoing support cover after the Edge setup is live?

Ongoing support can cover monitoring worker health, applying Airflow and Edge provider updates, rotating worker credentials, patching remote hosts, reviewing failure alerts, and testing changes before they reach production sites. Support is available as a retainer, and its scope should be written down: which sites it covers, how often patches are applied, who responds to incidents, and what is excluded. Ask for the response process and the change process for each remote site. Also check how Airflow version changes affect the Edge provider, since both need to be upgraded together.

What should we request in a written scope for remote Edge workers?

A useful written scope lists the sites in scope, the central Airflow version and Edge provider version, the executor design (Edge alone or combined with Celery or Kubernetes), the identity and certificate approach, the secrets handling, the test plan with its failure cases, the handover documentation, and the approval steps that belong to your team. It should also state what is one-time work and what recurs, and what changes would require a new quote. Request this before any build starts, so the quote reflects the actual environment instead of assumptions.

Limits to confirm in your environment before you commit

The Edge Executor depends on a chain of versions, networking rules and host conditions that the Airflow core documentation does not settle for your site. Confirm each item below against your target Airflow release before you approve a design, because a mismatch in any one of them changes what the remote workers can do.

Edge provider version match

The Edge provider is versioned separately from Airflow core, so the provider release you install must be one that supports your Airflow 3 release. Ask the provider's maintainers or read the provider changelog for the exact pairing. Record the pairing in your change record, because a core upgrade can require a provider upgrade on the same window.

DAG delivery method support

Remote workers need the DAG code and their dependencies, and the delivery method you choose (bundle, image or shared storage) must work across a network you may not be able to mount shared volumes into. Confirm with the provider whether it supports your chosen approach and how a DAG change reaches a remote site. Treat this as a design gate, not a detail to settle later.

Long tasks and dropped connections

Remote workers reach the central site over HTTPS, so a long task can outlive a network interruption. Ask how the provider reports a task whose worker lost contact, whether it is retried or marked failed, and how much time a worker can run before the central site treats it as lost. Test a forced disconnection mid-task before you rely on any retry policy.

Log retrieval during site outages

Task logs written at a remote site may not reach the central UI while that site is unreachable. Confirm where logs are written, whether they are shipped or fetched on demand, and what an operator sees for a task that ran during an outage. Your incident runbook depends on this answer, so ask for the behavior in writing.

Remote host and Python compatibility

Each remote worker host needs an operating system, a Python version and a set of libraries that match the Airflow 3 prerequisites and the Edge provider's requirements. The Airflow prerequisites list the supported Python versions and database versions for the central site. Confirm the worker Python version against both lists, and plan how hosts that drift from the baseline are rebuilt.

Outbound-only traffic policy

Workers initiate HTTPS connections to the central site, so the design depends on your firewall and egress policy accepting outbound-only traffic from remote networks. Check whether your security team allows those ports, how certificates are validated, and whether a proxy or inspection device interrupts the long-lived connections. Document the exception if the policy still blocks the path.

How DeployOpen scopes and delivers Edge executor deployments

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure. For Apache Airflow Edge Executor for remote workers, we start from a written scope that names each remote site, the central Airflow deployment and the HTTPS paths the remote workers will use.

How we scope

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
The written scope lists each remote network, cloud or restricted site that will run Edge workers, and which queues each site serves.
The scope states which executors coexist with EdgeExecutor, such as Celery or Kubernetes, and which DAGs will route to them.

How we build

We configure the central Airflow 3 deployment, including the metadata database and the separate API server and DAG processor components.
We configure each remote worker to reach the central site over HTTPS only, with outbound connectivity and credentials reviewed by your security team.
We test queue routing, task logs, restarts and failure behavior at each remote site before any production DAG is moved onto it.

How we hand over

You receive architecture notes, configuration references, network flow diagrams and runbooks for adding or removing a remote site.
Handover covers how to confirm a worker is healthy, how to drain it, and how to rotate the credentials it uses.
We walk your platform and operations staff through the runbooks and record open items with owners in the handover pack.

How we support

Ongoing support, upgrades and continuous testing are available as retainers, scoped in writing like the original build.
Upgrades of Airflow and the Edge provider are planned against the version life cycle and tested on a staging copy first.
We can re-test remote sites after network, certificate or firewall changes so that connectivity assumptions stay verified.