Apache Airflow remote task logging to S3, GCS or Elasticsearch

Apache Airflow remote task logging to S3, GCS or Elasticsearch: when you need it

You need remote task logging as soon as task pods or workers are short-lived and their local log files disappear with them. In a Kubernetes deployment or any autoscaling worker pool, a failed run's logs may be gone before an operator opens the Airflow UI. Shipping those logs to a durable store is the fix, and Airflow handles it through the [logging] section, where remote_logging switches the feature on and remote_base_log_folder sets the destination, such as an S3 or GCS path or an Elasticsearch endpoint. The Airflow project is licensed under the Apache License 2.0 and has no paid edition, so the logging backend you choose is a cost and operations decision, not a licensing one.

Airflow splits remote handlers into two families, and the choice shapes your operations. Blob handlers (S3, GCS and WASB) write each task's log as an object after the task runs, and delete_local_logs can then remove the local copy. Streaming handlers (Elasticsearch, AWS CloudWatch and GCP Cloud Logging) send log records to a search or logging service, which suits teams that want to query logs across tasks. Confirm in the task logging documentation which handler matches your Airflow version and provider packages, and check how each one behaves when the destination is unreachable, since a missing bucket or cluster can hide the logs you need during an incident.

Before you pick a backend, confirm three things with your security and platform teams: which storage or log service your organization already operates and retains data in, which people may read task logs, and how long logs must be kept. Logs are not a neutral record. Task output can contain secrets, particularly values passed through XCom or environment variables, so access to the bucket or index needs the same review as access to the Airflow UI. Decide these questions first, because they determine the bucket policies, index permissions and retention settings that the rest of the logging build depends on.

Which logging backend fits your environment

This table compares the blob handlers (S3, GCS, WASB), the Elasticsearch streaming handler and the cloud streaming handlers by storage model, query needs, retention control and fit with infrastructure you control, followed by the local and live-serving settings that affect every backend. Read each row against where your logs must live and who needs to search them, then confirm the handler names and settings against your Airflow version. The main takeaway is that blob storage suits archival and audit retention, while streaming to a search index suits investigation, and the two are often combined.

Option or setting	What it does	When it fits
S3 blob handler	Uploads completed task log files to an S3 bucket or compatible object store	Archival retention under your own bucket policies, with object storage already run in your infrastructure
GCS blob handler	Uploads completed task log files to a Google Cloud Storage bucket	Airflow runs on Google Cloud or your logs must stay in GCS for governance
WASB blob handler	Uploads completed task log files to Azure Blob Storage	Airflow runs on Azure or your retention policy is defined on Azure storage accounts
Elasticsearch handler	Streams task logs into an Elasticsearch index that you query in Kibana or the API	Teams need search across task logs and you can run and secure a search cluster
CloudWatch or GCP Cloud Logging handlers	Stream task logs to the cloud provider's logging service	The workload already uses that provider's logging, and you accept its retention and access model
delete_local_logs	Removes local task log copies after upload to remote storage	Workers are short-lived or disks are small, and remote storage is confirmed as the copy of record
worker_log_server_port 8793	Port where workers serve live task logs to the API server	Live log viewing is needed while tasks run, so network paths and the shared secret key must be planned
Triggerer log port 8794	Port where the triggerer serves its live logs	Deferred tasks run on a triggerer and their logs must be viewable, so the port must be reachable from the API server
Shared [api] secret_key	Authenticates log requests between the API server and workers	Always required for live log serving; treat it as a credential and confirm who can read it

How blob handlers and streaming handlers differ in practice

Blob handlers and streaming handlers differ in when a log becomes searchable and where it lives afterward. A blob handler, such as the S3, GCS or WASB handlers, writes the task log to a file and uploads it to object storage when the task finishes. A streaming handler, such as the Elasticsearch handler, sends individual log records to a search or logging service as they are produced. That timing decides almost every operational consequence below.

With blob handlers, the live view comes from the worker's log server on worker_log_server_port 8793 while the task runs, and the object in the bucket appears only after upload. Search across runs depends on tooling you add yourself, such as querying the bucket or indexing it separately. Retention is a bucket policy you own, and delete_local_logs removes the local copy only after a successful upload, so a failed upload can leave a log that exists only on a pod or worker that is about to disappear. With streaming handlers, records are searchable during the run, retention is governed by the index or log group settings, and a backend outage can delay or drop records depending on how the handler buffers them. Confirm the retry and buffering behavior of your chosen handler version with the Airflow provider and logging documentation before relying on it for audit evidence.

Live log serving does not move to the remote store in either model. Workers serve running task logs, and the triggerer serves its own logs on port 8794, with requests authenticated by the shared [api] secret_key. Plan for that dependency explicitly: network paths from the API server to every worker and triggerer, and a key that is rotated and protected like any other credential. Who may read logs also differs. Blob buckets need object-level access controls and audit logging, while search indexes need role-based access that can restrict which indexes or fields a user sees. Because masking covers only values read through Connections or Variables, neither handler will hide a secret that a task printed or passed through XCom or environment variables.

Setting up remote logging step by step

This sequence moves Airflow task logs off the worker or pod into a storage backend you control, then proves that a finished run's logs stay readable in the UI after its pod is gone.

Choose the logging backend
Your platform team picks one destination before any configuration starts: S3, GCS or WASB for blob storage, or Elasticsearch, CloudWatch or GCP Cloud Logging for streaming. Blob handlers store complete log files per task attempt, while streaming handlers send log records to a search or logging service. Confirm the choice against your retention, search and access requirements, and against which provider package and handler your Airflow version supports. Done when the team has written down the backend, the bucket or cluster name, and the owner of that resource.
Create storage and least-privilege identities
Your cloud or platform team creates the bucket, container or Elasticsearch cluster and two separate identities: one that can write task logs and one that can read them for the API server. Scope each identity to the single log location, and avoid granting bucket administration or broad cluster roles. Confirm object lifecycle and retention rules with your security team before the first run, because they decide how long logs persist. Done when a test write and a test read succeed with each identity and the denied operations fail as intended.
Set remote logging options in the configuration
Under the [logging] section, set remote_logging to True and remote_base_log_folder to the bucket path or cluster URL from the previous step. Deliver these values through your normal configuration path, such as environment variables or a Kubernetes Secret, so that the same settings reach the scheduler, workers and API server. Check that every component reads the same file or variables. Done when a configuration dump on each component shows identical remote settings.
Configure the handler and connection
Install the provider package that matches your backend and set the handler and connection details the provider documents, including the Airflow connection that holds the credentials for the write identity. Use a workload identity or secret reference where your platform supports it, so that no long-lived key lands in a values file. Confirm that the connection is visible only to the components that need it. Done when a worker can start a task and the connection test passes from inside the worker environment.
Set delete local logs
Set delete_local_logs so that local copies are removed after the handler uploads them. This matters on pods and workers with ephemeral disks, where local copies would otherwise hold log content that your retention policy does not cover. Confirm that the upload finishes before deletion on a failed task as well as a successful one. Done when a completed task leaves no local log file on the worker, and the object or record exists in the remote backend.
Confirm the shared secret key across components
Check that the [api] secret_key is identical on the API server and every worker, and that the triggerer uses the same value. Workers serve live logs on port 8793 and the triggerer on 8794, and the API server uses the shared key to authenticate those log requests. Decide how the key is stored, rotated and restricted, and confirm that your network rules allow the API server to reach the worker log ports. Done when a log request from the API server to a running worker succeeds and an unauthenticated request is refused.
Run a test DAG and check remote logs
Run a small test DAG whose tasks print known, non-sensitive text, then wait for the pod or worker to terminate. Open the task log in the UI and confirm the content loads from remote storage, not from a local file. Check the bucket or Elasticsearch index directly to confirm the object or documents exist with the expected run and task identifiers. Done when the log is readable after the pod is gone, and the reader confirms that the access rights match your policy.
Review who can read logs
Task logs can contain values that Airflow does not mask, such as values passed through XCom or as environment variables to KubernetesPodOperator, so treat log access as sensitive access. Confirm which roles can open task logs in the UI and which identities can read the bucket or index directly, and record the result as evidence for your access review. Ask your security team whether the log location needs encryption, network restriction or a separate audit trail. Done when the access list is signed off and any gaps have an owner.

Who can read task logs that contain secrets

Remote task logs can hold values that Airflow never masked, so they need the same access discipline as the systems they came from. The cards below cover masking limits, who can reach the log store, the shared key used for live log requests, how long logs stay, and how reads get recorded.

Masking covers only Connections and Variables

Airflow masks values that come through Connections or Variables. Values passed through XCom or as environment variables, such as those given to KubernetesPodOperator, are not masked, so a password placed there can appear in task output and therefore in the stored log. Call mask_secret before the task produces any output, and test a DAG that echoes its environment to confirm nothing leaks. Treat any secret passed by these routes as readable by anyone who can read the log.

Restrict the bucket or index

Whoever can list or read the S3 bucket, GCS bucket or Elasticsearch index can read every task log written there. Grant read access by role to the teams that own each DAG, deny public access on the bucket, and keep write access limited to the Airflow worker identity. For Elasticsearch, define index-level roles so log reads do not inherit cluster-wide access. Confirm these rules with your cloud or search administrators before the first run.

Protect the shared secret key

Workers serve live logs on port 8793, and the API server and workers authenticate those requests with the shared [api] secret_key. Anyone holding that key can sign log requests against reachable workers, so store it in a secrets manager, rotate it on a schedule, and keep the worker log ports off public networks. Ask your platform team who holds the key and how a rotation would reach every component at once.

Set log retention by data sensitivity

Remote storage keeps logs long after pods and workers are gone, so retention becomes a data decision. Set lifecycle rules on the bucket or index policy in Elasticsearch that match the sensitivity of the DAGs, and delete logs when the business need ends. Enable delete_local_logs so worker disks do not keep duplicate copies. Ask your compliance lead to approve the retention period in writing before go-live.

Audit who reads logs

Airflow records task runs, but reads of the log store happen in S3, GCS or Elasticsearch, and those services log access differently. Turn on object access logging or the Elasticsearch audit log, send it to a central system, and review who opened logs for sensitive DAGs. Confirm with your security team which log sources satisfy your audit evidence requirements, because the answer depends on your own controls.

Limit access to the Airflow UI

Users who can open a task log in the Airflow UI or API can read the same content as the bucket, so UI roles matter as much as storage roles. Map Airflow roles to team membership, separate operators from DAG authors, and check that the auth manager you chose enforces per-DAG permissions. Test the mapping with a non-admin account before granting access to production logs.

Choosing between object storage and Elasticsearch

Object storage such as S3, GCS or WASB suits teams that mainly need a durable record of each task attempt and open it when a run fails or an auditor asks for it. Airflow writes logs through a blob handler, so retrieval follows the object key, which is the DAG, task, run and try. You keep the storage cost low and avoid running a search cluster that needs its own capacity planning.

Retrieval model: You find logs by known identifiers, not by free-text search across all runs.
Retention: Lifecycle rules on the bucket control how long archives stay, and this is usually the main cost lever.
Operations: Fewer moving parts than a search cluster, with no index to tune or shards to rebalance.
Access control: Read permissions are set on the bucket, so confirm who can list and download log objects.
Limits: Searching for an error string across hundreds of runs means downloading or scripting against many objects.

Before you commit, confirm the bucket's encryption, versioning and retention settings with your security and records teams.

Limits to confirm in your environment

Remote logging touches several products with their own version lines, license tiers and network paths. The cards below list what to confirm for your Airflow 3 version and your Elasticsearch deployment before you commit to a handler or a topology.

Handler support for your Airflow version

Confirm that the remote logging handler you plan to use (S3, GCS, WASB, Elasticsearch, CloudWatch or GCP Cloud Logging) is supported in the exact Airflow 3 patch line you will run, and that its provider package is pinned in the same constraints file as Airflow. Handler names, settings and defaults can shift between releases, so read the task logging page for your version rather than copying settings from an older tutorial. Test one failed and one successful task run end to end before rollout.

Elasticsearch tier and security features

Single sign-on, field- and document-level security and some anomaly-based alerting sit in paid Elastic subscription tiers, while TLS and native users were made free in the Basic tier in 2019. Check which tier your organization holds or would buy, and whether it covers the authentication and access controls your log readers require. If your logs can contain secrets, field-level restriction is a question to settle with the vendor before the cluster holds production data.

Platinum tier availability

Elastic's subscriptions page states that the Platinum tier is no longer available to new customers and points them to Elastic Cloud instead. If a design assumes Platinum features, such as certain anomaly-based alerting rules, ask Elastic which tier is available to you today and whether a managed option changes your data residency or control requirements. Get the answer in writing before the design is approved.

Node roles and the HA minimum

A highly available Elasticsearch cluster needs at least three master-eligible nodes, and data, ingest and coordinating roles should be sized separately from them. Confirm how many nodes your platform team can provide, which roles each one holds, and whether hot, warm and cold data tiers are needed for log retention. A two-node cluster can fail to elect a master, which would stop log writes from Airflow workers.

Log serving ports behind network policies

Workers serve live task logs on port 8793 and the triggerer on port 8794, and the API server must reach them with the shared [api] secret_key. Clusters with default-deny network policies, separate node pools or firewalled VM subnets may block these paths, so the UI shows logs only for finished tasks or fails for running ones. Test connectivity from the API server to every worker and triggerer in your actual topology, and confirm that the port numbers can be changed in your environment.

Open-source licensing of the log store

Elasticsearch's default distribution is under the Elastic License 2.0, which is not an OSI-approved open source license, and AGPLv3 is offered as an OSI-approved option in the same source code. Decide which license your organization accepts before selecting a distribution, and confirm the support lifecycle for the release line you plan to run, since the 8.x line reaches end of maintenance earlier than 9.x. OpenSearch is an alternative to evaluate, but its wire compatibility is with Elasticsearch 7.10 and it needs its own check against the handler you choose.

Questions teams ask before logging to S3, GCS or Elasticsearch

How much does remote task logging cost to run?

The cost depends mostly on the backend you choose, how much log volume you keep, how long you retain it, and the Elastic licensing tier if you pick Elasticsearch. Object storage (S3 or GCS) is usually billed by stored volume and requests, and it recurs every month. Elasticsearch adds cluster compute, storage, and memory that also recur, plus a subscription decision if you need features beyond the free tier. Setup labor is a one-time cost. Ask any partner for a quote that separates the one-time build from recurring infrastructure and licensing, and that states the retention period and expected daily log volume it assumes.

How long does it take to configure remote logging?

Configuring the Airflow side, meaning the remote_logging and remote_base_log_folder settings, the handler choice, and the delete_local_logs behavior, is a small part of the work. Most of the elapsed time goes to waiting on storage accounts, IAM roles, network paths, and, for Elasticsearch, cluster sizing and licensing decisions. Your timeline depends on how fast your cloud, security, and procurement teams can approve those items. Ask for a written plan that lists each external dependency and who owns it.

What do we need to provide before the work starts?

You will need to provide several items before any configuration begins:

A bucket or container in S3, GCS, or Azure Blob storage (WASB), with a retention and lifecycle policy you have approved
An identity for the Airflow components with write access to the log path and read access for the people who need it, using least privilege
Network paths from Airflow workers, the API server, and the triggerer to the storage or Elasticsearch endpoint, including any private endpoints or proxies
For Elasticsearch, a licensing decision for each deployment, since security features such as single sign-on and field-level security sit in paid tiers, and a node layout with at least three master-eligible nodes for high availability
A named owner for the shared [api] secret_key used for live log serving

Can our internal team set up remote logging alone?

Often yes, if your team already runs Airflow and object storage or Elasticsearch in production. The Airflow change is contained, but the risks sit in details that engineers often miss: live log serving on worker_log_server_port 8793 and triggerer port 8794, the shared secret key, and masking limits for values passed through XCom or environment variables. Teams without prior Elasticsearch operations experience usually need outside help for cluster design and licensing. Assess your team on those specific skills before you decide.

Who should be allowed to read task logs?

Restrict log access to the people who need it for operating and debugging pipelines, and treat logs as sensitive. Task logs can contain secrets or personal data, especially when values are passed through XCom or as environment variables, because Airflow masking does not cover those paths. Map read access to your existing identity groups, apply least privilege on the storage bucket and Elasticsearch indices, and keep separate permissions for administrators who can change log settings. Record who can view logs and review that list on a regular schedule.

Should we choose Elasticsearch or object storage for logs?

Choose object storage if you mainly need to keep logs for later retrieval and audit, and you do not need search across many runs. Choose Elasticsearch if your teams need full-text search and dashboards across task logs. Elasticsearch carries more operating effort and licensing decisions. Elastic's source code is available under the Elastic License 2.0, SSPL 1.0, and AGPLv3, and Elastic states that ELv2 and SSPL are not OSI-approved open source licenses, so confirm which license and distribution fits your policy. OpenSearch is a fork of open source Elasticsearch 7.10 and is an alternative to evaluate with your team.

Does remote logging keep logs safe after pods are deleted?

Remote logging keeps task logs after pods and workers disappear, but only if the upload succeeds and the storage stays available. Enable delete_local_logs only after you have confirmed that uploads work in your environment, because it removes local copies. Test a failed upload, a restarted worker, and a deleted pod before you rely on the setup. Ask any partner to show the test results as part of the handover.

Keeping logs reliable after go-live

Logs stop being reliable the moment nobody owns the bucket, index and upload path. Treat log storage as a production service from day one: set retention and lifecycle rules on the S3, GCS or Elasticsearch side before the first long-running DAG writes history, because delete_local_logs only removes worker copies after a successful upload and does nothing about how long remote logs are kept. Decide retention with your compliance owner, then confirm that the bucket lifecycle policy or the Elasticsearch index lifecycle policy matches it. Check the expiry behavior in a test bucket or index first, since deleting logs early can break audit requests for runs that are still in scope.

Upload failures are often silent. A task can finish, the local copy can remain, and the first sign of trouble is an operator opening a log that is missing or truncated. Alert on worker log upload errors and on growth of local log directories, and make sure the credentials used by workers still have write access after any IAM, service account or Elasticsearch role change. Rotating the [api] secret_key is the other ongoing task that needs a plan: because the API server and workers use it to authenticate log requests, change it in one controlled window across all components that share it, and confirm that live log viewing works for a running task afterward. Confirm the exact rotation steps for your Airflow version with the official documentation before you schedule the change.

Upgrades change the logging path. Review the remote logging settings, the provider package version and the handler class whenever Airflow or the provider is upgraded, and test a task log write and read in staging before production. Then run the restore test that proves the design works: remove a worker pod or VM that ran a recent task, open that task's log from the UI, and confirm it loads from remote storage rather than local disk. Repeat the test after each change to storage, identity or the Airflow version. Record the results as evidence, because they show that log access survives the loss of the worker that produced it.

How DeployOpen works on this

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For Airflow task logging, that means agreeing in writing where logs go, who can read them and how they are kept before any configuration changes.

How we scope

Confirm which remote log destination fits your estate: S3, GCS, Elasticsearch or another supported handler.
Record retention, encryption and access requirements for task logs that may contain sensitive values.
Agree in writing whether log serving ports 8793 and 8794 and the shared API secret key are in or out of scope.

How we build

Set remote_logging and remote_base_log_folder in configuration held under version control.
Test blob and streaming handlers in a non-production environment before choosing one for production.
Verify that delete_local_logs removes local copies only after uploads succeed, and that masking runs before output.

How we hand over

Document the logging configuration, handler choice, storage permissions and the reasons behind each setting.
Provide a runbook for reading logs after pods or workers are gone, including who is granted access.
Walk your platform and security teams through the masking limits for XCom and environment variables.

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, upgrades and continuous testing are available as a retainer.
Confirm with DeployOpen during scoping which Airflow and Elasticsearch versions the support period covers.

Apache Airflow remote task logging to S3, GCS or Elasticsearch