Apache Airflow with AWS: S3, Redshift, EMR and Glue

Apache Airflow with AWS: how the Amazon provider drives S3, Redshift, EMR, Glue and Athena

A data team running self-hosted Airflow that must orchestrate AWS data services has one supported route: the Amazon provider package, release 9.38.0, which requires Airflow 2.11.0 or later. The provider includes operator guides for Amazon S3, Redshift, EMR, Athena, AWS Glue, the Glue Data Catalog, and a wider set of services such as Lambda, Batch, Step Functions and DynamoDB. Each guide documents the operators, sensors and hooks for one service, so the first task is to inventory which AWS services your pipelines touch and check the operator guide for each one against the provider version you plan to pin. The Amazon provider operators documentation is the list to work from, and it should be read against the Airflow 3 version you are deploying, since provider minimums change between releases.

The same provider also ships AWS executors, an AWS auth manager, and the AWS Secrets Manager and SSM Parameter Store secrets backends. This page covers only the data-service operators. Executor choice, identity and secrets are separate decisions with their own consequences, and they should be settled before DAGs are written, because they determine where task code runs and which IAM principal it uses. Confirm which of these components your team actually needs so the provider is installed deliberately rather than pulled in wholesale. The practical question for the data team is who owns the AWS connection and its permissions. Operators inherit the credentials of the Airflow connection they reference, so an overly broad role on one connection becomes access for every DAG that uses it. Decide early whether Airflow runs inside AWS with an instance or task role, or outside AWS with long-lived or federated credentials, and confirm with your AWS account owners how each service's role is scoped before any transfer or job is built.

What you need before connecting Airflow to AWS

Before any AWS operator runs, the Airflow version, metadata database, provider release, AWS account layout, network path and IAM principal need to agree with each other. Each card below is a decision or check that, if skipped, usually surfaces later as a failed connection or an access problem in production.

Airflow 3 version and Python

Start from the Airflow 3.3.2 prerequisites, which list Python 3.10 to 3.14. Pin the Airflow version your team will run in production and confirm that your build image and virtual environments use a Python version inside that range. Airflow 3 also requires that the DAG processor runs as its own component, so plan that process alongside the API server and scheduler from the start. Confirm the exact patch release you will deploy before writing any AWS connection.

Metadata database choice

Connections, variables, DAG runs and task state all live in the metadata database. The production guide recommends PostgreSQL or MySQL, and warns that SQLite can lead to data loss in production. MariaDB is not supported as the metadata database. Decide on the database service, its backup schedule and its high availability design before the AWS connections are stored, because that is where credentials will be kept unless you move them to a secrets backend.

Amazon provider pinned to constraints

The Amazon provider has a minimum Airflow version, and its release must match the Airflow constraints file for your Airflow version. The current Amazon provider release in the supplied facts is 9.38.0, which requires Airflow 2.11.0 or later, so confirm that the release you install is compatible with Airflow 3.3.2 in the provider's own documentation. Install it with the constraints file in your build so transitive dependencies such as boto3 stay stable across rebuilds.

AWS accounts and regions

Decide which AWS accounts Airflow may drive and in which regions. Many teams separate a development account from production and keep S3 buckets, Redshift clusters and EMR or Glue jobs in the same region to limit data transfer and latency. Record the account ID and region for every connection, and confirm that services used by your DAGs, such as Athena query result locations, are available in those regions. Ask your cloud team to confirm any service quotas that your pipelines will need.

Network paths to AWS endpoints

Airflow components must reach AWS service endpoints for S3, Redshift, EMR, Glue and Athena, plus STS if you use role assumption. If Airflow runs in your data center or a private network, confirm the route through a proxy, NAT gateway, VPN, Direct Connect or VPC endpoints, and check DNS resolution for the regional endpoints. Redshift and any database connections also need security group or firewall rules. Test each path from the same host or pod that runs the scheduler and workers, not only from a laptop.

IAM principal type

Choose how Airflow authenticates before creating connections. Options include an IAM role attached to the compute running Airflow, an IAM user with access keys, or a role assumed through STS from another account. Roles avoid long-lived keys and are easier to rotate, while access keys are sometimes the only option for air-gapped or on-premises hosts. Whichever you pick, scope permissions per DAG team or environment, and confirm with your security team how the chosen principal will be audited and reviewed.

How to configure the Amazon provider in five steps

This sequence takes a self-hosted Airflow from an installed Amazon provider to verified access for S3, Redshift, EMR and Glue, with each service added only after the one before it has passed a check.

Install the Amazon provider with constraints
Your platform team installs the apache-airflow-providers-amazon package into the same Python environment or container image as Airflow, pinned with the Airflow constraints file for your Airflow version so the provider and its dependencies resolve against a tested set. Release 9.38.0 of the provider requires Airflow 2.11.0 or later, so confirm the Airflow version in your environment first. The step is done when airflow providers list shows the Amazon provider on every scheduler, worker and API server image, and a DAG that imports its operators parses without errors in the DAG processor.
Create the AWS connection
An Airflow administrator creates an AWS connection (for example aws_default or a named connection such as aws_data_prod) in the metadata database or in your secrets backend, which is the preferred location for credentials. Keep the connection's region and any role-assumption settings in the connection, not in DAG code. The step is done when the connection test in the Airflow UI or a short test task confirms that AWS Security Token Service accepts the identity, and the connection's secret values are not visible in logs or in the rendered task fields.
Grant least-privilege IAM to the connection
Your cloud or security team attaches an IAM role or user to the connection that allows only the actions each pipeline needs, scoped to named buckets, prefixes, workgroups, Redshift clusters, EMR clusters and Glue jobs. Separate roles per environment and per team make audit evidence clearer. Ask the security team to review the policy against the Amazon provider operator guides before the first run. The step is done when the role's policy simulator or access review shows allowed actions for the intended resources and denials for a sample of out-of-scope resources.
Verify access with an S3 list task
A developer adds a small DAG with one task that lists objects under a single test prefix using the Amazon provider's S3 operators or hooks and the connection from the previous step. Run it manually, then confirm it in the task logs. This check proves the network path, the connection, the role and the provider install all work together before any data pipeline depends on them. The step is done when the task reaches success, the expected keys appear in the log output, and the same DAG succeeds from a worker in the environment that will run production tasks.
Add Redshift, EMR and Glue one service at a time
Your data engineers add operators for one service per change, starting with the service your pipelines need first, such as Redshift for SQL loads, EMR for Spark steps or Glue for ETL jobs. Each addition gets its own least-privilege policy statements and a small test DAG that submits a harmless statement or job run, then waits for it to finish. Use the operator guides in the Amazon provider documentation to confirm each operator's parameters for your provider version. The step is done when a submitted job or statement reaches a terminal state in the AWS console and Airflow shows the matching task result, before the service is used in a production schedule.

How to design IAM connections so each DAG reaches only its own data

The first decision is how Airflow proves its identity to AWS, and that choice matters more when Airflow runs outside AWS. With static access keys stored in an Airflow connection, every worker that reads the connection holds a long-lived credential, and rotation becomes a manual project. Assumed roles through AWS STS let Airflow exchange a base identity for short-lived credentials scoped to one role, which is the usual pattern for a self-hosted cluster in a data center or another cloud. Instance or workload identity removes stored secrets entirely, but it only applies when the workers run on AWS compute or a federated identity source that AWS trusts. Confirm which authentication options the Amazon provider supports for your connection type in the Amazon provider package documentation, and confirm how your Airflow version passes role assumption settings through the connection's extra fields.

Blast radius is set by how many DAGs share one identity. A single connection used by every DAG on every environment means a misconfigured task in development can read production S3 prefixes or submit EMR steps against production clusters. A practical starting point is one connection per environment and per team or data domain, with each connection assuming a role whose policy allows only the buckets, Glue databases, Redshift schemas, EMR clusters and Athena workgroups that team owns. Because DAG code can name any connection ID, the restriction has to live in AWS policy, not in DAG review alone. Ask your AWS governance owners to define the permission boundaries and trust policies, since they decide whether roles can be chained, which accounts can assume them, and whether service control policies cap what any role can do.

Before go-live, verify the design rather than trusting the diagram. Run a test DAG under each connection and confirm that it succeeds on its own prefixes and fails on a neighboring team's bucket, a Glue database it should not see, and a Redshift schema outside its grants. Record which role each task session actually assumed, and check that credentials expire on the schedule your security team expects. Plan the rotation of any remaining static keys and the process for revoking a role when a team changes. Keep the connection inventory in version control or a secrets register so audits can trace which identity a pipeline used on a given date.

Which AWS services have Airflow operators and how S3, Redshift, EMR and Glue differ

Amazon S3 is the most common handoff point between Airflow and other AWS data services. The Amazon provider documents operator guides for S3, which cover moving and managing objects, and the provider's sensors can hold a DAG until a key or prefix appears. Most S3 problems come from object size and listing behavior rather than from the operator itself, so the team should review how files are produced and how many objects a prefix can hold before designing any task.

Data volume: Confirm whether transfers move small control files or multi-gigabyte extracts, because large copies run on the worker for the full duration unless the data moves through a service that does the copy.
Wait mode: Confirm whether key sensors poll on a worker slot or run in deferrable mode on the triggerer, since long waits for upstream drops can hold worker capacity.
Cleanup: Confirm who deletes staged objects after a successful load and how failed runs leave partial prefixes behind.
Naming and idempotency: Confirm that output keys are deterministic so a retry overwrites or skips rather than duplicating data.
Access: Confirm the IAM role for each bucket and prefix, and whether encryption keys need extra permissions.

Before building, ask the AWS account owner to confirm bucket layout, retention rules and whether any sensor is expected to wait longer than a normal run window.

How Airflow connections, IAM roles and data locations map to each AWS service

This table shows what each AWS service needs from an Airflow connection, role or data location, and what your team must decide before building DAGs. Read each row as a list of decisions to record per service, then check the exact connection fields against the Amazon provider operator guides for your provider version. The main takeaway is that one shared AWS connection is rarely the right design, because each service needs its own principal, region and location boundaries.

Service	What the team must decide	Where it is set in Airflow	What to confirm before build
Amazon S3	Which IAM principal reads or writes each bucket, and which prefixes each DAG team may touch	AWS connection (role or principal) plus bucket and prefix passed to the operator or sensor	Bucket policy and IAM policy both allow the exact prefixes, with no wildcard across teams
Amazon Redshift	Whether the target is a provisioned cluster or a serverless workgroup, and which database user or IAM identity runs the load	Cluster or workgroup identifier, database name and credentials source, with the AWS connection for API calls	The identity has only the grants needed for the target schema and the required COPY or UNLOAD permissions
Amazon EMR	Whether jobs go to an existing cluster, a transient cluster or an EMR Serverless application, and which execution role the job uses	Cluster or application identifier and job flow or job run configuration in the DAG	The execution role can read its input prefixes and write its output prefixes, and logs go to a location you control
AWS Glue	Job name, the IAM role Glue assumes, and whether the Glue Data Catalog is shared with other tools	Glue job name in the operator, with the role configured on the Glue job itself	The Glue role has S3 and catalog permissions for the job's databases and no broader account access
Amazon Athena	Workgroup per team or environment, and the S3 output location for query results	Workgroup name and output location passed to the Athena operator	Results bucket has retention and encryption settings you approve, and workgroup limits match team use
Region	Which AWS region each service runs in, and whether cross-region reads are allowed	Region set on the connection or on each operator, depending on provider version	Every service in a pipeline is in the same region unless a cross-region transfer is approved
Airflow AWS connection	Whether tasks assume roles through the connection or rely on an instance, pod or workload identity	Airflow connection of the Amazon type, with secrets kept in a secrets backend	Connection grants no more than the union of roles its DAGs need, and secrets are not stored in DAG code

Securing AWS credentials, secrets and network paths from Airflow

Airflow reaches AWS through connections, and every connection is a credential that a DAG author, a worker or a log could expose. The cards below cover where those credentials live, how their permissions are scoped, how data is protected in transit and at rest, and how AWS API activity is recorded. Each one names a decision or check to settle before the first production DAG runs.

Keep keys out of DAG code

Hardcoded access keys in DAG files end up in version control, in task logs and in every copy of the repository. Define AWS connections in the metadata store or a secrets backend, and reference them by connection ID in operators. Review DAG pull requests for literal keys, account IDs and endpoint URLs, and treat any key found in a repository as compromised and rotate it. Confirm who can read connection definitions in the Airflow UI, since that permission controls who sees the stored secret fields.

AWS Secrets Manager backend

The secrets backend list in the Airflow documentation includes the AWS Secrets Manager backend, which the Amazon provider ships. Airflow looks up connections and variables in Secrets Manager by a configured prefix, so the metadata database does not need to hold the credential values. Decide the prefix layout per team or environment before creating secrets, and confirm the exact configuration keys and lookup order against the version of the Amazon provider you deploy. Airflow still needs its own AWS identity to read the secrets, so the design depends on the instance profile or role chosen for the scheduler and workers.

SSM Parameter Store backend

The Amazon provider also includes a backend that reads connections and variables from SSM Parameter Store. It suits teams that already manage configuration values in Parameter Store and want a lower-cost path than Secrets Manager for non-rotating values. Confirm the parameter naming pattern, whether values are stored as SecureString, and whether the KMS key used for decryption is restricted to the Airflow role. Test that a missing parameter fails the task visibly rather than falling back to an unexpected source.

Least-privilege policies per service

Give each Airflow identity only the actions its DAGs need, scoped by resource. A DAG that writes one S3 prefix needs s3:PutObject on that prefix, not on the bucket, and a Redshift loader needs only the S3 read it uses for COPY. Separate roles for EMR job submission, Glue job runs and Athena queries make it possible to see which pipeline made which call. Review policies when a new operator is added, since each Amazon operator calls a different API set, and document the list of actions per DAG team.

Encryption of S3 and Redshift data

Data moved by Airflow tasks is only as protected as the path it takes. Require TLS for connections to S3 and Redshift, and set bucket policies that deny requests that are not sent over TLS. Enable default encryption at rest for S3 buckets and Redshift clusters, and decide whether AWS managed keys or customer managed KMS keys meet your key custody requirements. Check that intermediate staging files and XCom payloads do not hold unencrypted extracts that outlive the run.

Logging AWS API calls

Airflow task logs show what a task did, but they do not replace a record of AWS API activity. Enable AWS CloudTrail for the accounts and regions where Airflow operates, and decide which events feed your central log store. Correlate CloudTrail entries with the role session and the DAG or task identifier, which you can set through tags or naming conventions on jobs. Confirm log retention and access controls with your compliance team before relying on these records as audit evidence.

Why AWS tasks fail under self-hosted Airflow and how to fix them

This table lists the failures teams most often hit when Airflow tasks call S3, Redshift, EMR, Glue or Athena from infrastructure outside or inside AWS. Each row gives the likely cause and the first fix to try, so you can narrow the problem before changing the design. Start with the row that matches the error text in your task logs.

Failure	Likely cause	First fix to try
Wrong region or endpoint	The connection or operator defaults to a region that does not hold the bucket, cluster or job, or a custom endpoint points at the wrong service.	Set the region explicitly on the AWS connection and operator, then confirm the endpoint override is empty unless you use a VPC or compatible endpoint on purpose.
Assumed role trust policy errors	The role's trust policy does not allow the principal Airflow runs as, or the external ID or session tags do not match what the connection sends.	Compare the caller identity in the task log with the role trust policy, then correct the principal and conditions before changing permissions.
Missing S3 permissions	The execution role lacks s3:GetObject, s3:PutObject or s3:ListBucket on the bucket or prefix, or a bucket policy or KMS key policy denies access.	Check the failing API action in the error, grant the minimum action on the exact bucket and prefix, and confirm KMS decrypt rights if the bucket uses a customer key.
Missing Redshift permissions	The database user or IAM role used for the connection cannot run the statement, or the COPY or UNLOAD role is not attached to the cluster.	Confirm the cluster has the role attached for COPY or UNLOAD, then test the same statement with the same user in a SQL client.
EMR or Glue jobs left running after retries	A task retry or timeout submits a new job while the first run is still active, and the operator does not stop or wait on the original run.	Make the job name or run key idempotent, set a clear timeout, and add cleanup or cancel logic on failure. Confirm the operator's wait and termination behavior in the provider docs.
Network path blocked from workers	Workers or pods cannot reach AWS API endpoints, the Redshift port, or the EMR and Glue control plane because of security groups, NAT, proxy or private DNS settings.	Test DNS resolution and TCP reachability from the exact worker or pod that fails, then fix egress rules, VPC endpoints or proxy settings for that network path.
Provider and Airflow version mismatch	The Amazon provider release requires a newer Airflow than the installed core, or operator parameters changed between provider releases, which breaks DAG parsing or task execution.	Check the provider's minimum Airflow version and pin the provider with Airflow constraints. Confirm the installed versions on the scheduler, workers and triggerer match.
Credentials resolved from the wrong source	The task uses an environment credential, instance profile or default profile instead of the intended connection, so permissions differ between local tests and workers.	Log the resolved identity at task start, then remove conflicting environment variables or set the connection explicitly in the operator.

Questions buyers ask before connecting Airflow to AWS

What drives the cost of connecting self-hosted Airflow to AWS?

The cost is driven by how many AWS services the pipelines touch, how many IAM roles and connections must be designed, how the Airflow workers reach AWS, and how much testing the pipelines need before they run on schedule. Each service adds its own operator setup, permissions, and failure cases. Cross-account access, private endpoints, and shared connections across teams add design work. Ask any partner for a written scope that lists each service, each role, each network path, and each test, so the quote can be compared line by line with your own estimate.

How long does an Airflow to AWS integration usually take?

Duration depends on the number of services in scope, the complexity of the IAM model, whether the Airflow environment already exists, and how many pipelines must be migrated or rebuilt. A single service such as S3 with one connection is much smaller than a set of pipelines that move data through S3, Redshift, EMR, and Glue with separate roles. Confirm the timeline in the written scope rather than relying on a general estimate, and ask what dependencies, such as account approvals or network changes, sit outside the partner's control.

What does our team need to provide before the work starts?

You will need to provide the AWS accounts and their account IDs, a named person with authority to create IAM roles and policies, and network access between the Airflow environment and the AWS endpoints in use. Sample DAGs or pipeline descriptions help the design reflect real data flows. You should also provide the Airflow environment details, the secrets store you want to use, and the owners who approve changes. Confirm each item in writing before the scope is signed, because missing access is a common cause of delay.

Can an internal platform team set this up without a partner?

Yes, a team with Airflow operations experience and AWS IAM experience can build this internally. The Amazon provider documents operators for services such as Amazon S3, Redshift, EMR, Athena, AWS Glue, and Glue Data Catalog, and the work is mostly configuration, permission design, and testing. The risk lies in the details: least-privilege policies, separate connections per environment, and tested failure handling. Assess whether your team has these skills and capacity, then compare the internal plan with a written partner scope before deciding.

Which parts of the work are one-time and which recur?

One-time work includes the architecture, provider installation, IAM role and policy design, connection setup, secrets backend configuration, and the first round of pipeline testing. Recurring work includes provider package upgrades, Airflow version upgrades, policy reviews when services change, rotation of credentials, monitoring of failed transfers, and cost checks on AWS usage. Ask whether a quote includes a handover and documentation, and whether ongoing support is needed as a separate retainer.

Do we need to run Airflow inside AWS to use these operators?

No. Airflow can run in your data center, another cloud, or an air-gapped network and still drive AWS workloads through the Amazon provider, as long as the workers can reach the AWS APIs and the credentials are handled securely. The network path, outbound rules, and whether traffic leaves through a proxy or private endpoint must be designed and tested. Confirm the egress path and the credential source early, since these decisions shape the IAM design and the cost of data movement.

How should AWS credentials be handled in an Airflow connection?

Use IAM roles with narrowly scoped policies for each pipeline or team, and avoid storing long-lived access keys in DAG code or environment variables. Store connection details in a secrets backend, such as AWS Secrets Manager or SSM Parameter Store, which the Amazon provider supports, and restrict who can read them. Plan rotation, audit logging, and separate connections for development, staging, and production. Confirm these choices with your security team before the first production connection is created.

What should we test before the AWS pipelines go live?

Test each operator against a non-production AWS account, including success paths, permission denials, retries, and timeouts. Verify that transfers into S3 and Redshift produce the expected row counts and file layouts, that EMR and Glue jobs fail visibly when they should, and that logs and alerts reach the on-call team. Include a rollback plan for schedules that move from cron or another scheduler. Ask any partner to describe their test plan and its exit criteria in the written scope.

How DeployOpen scopes and delivers the Airflow to AWS integration

DeployOpen plans, builds and hands over self-hosted Apache Airflow with the Amazon provider inside your own AWS accounts or on-premises network. Each engagement starts from a written scope that names the AWS services, connections, IAM roles and workloads Airflow will drive, so the boundaries are agreed before any DAG is written.

How we scope

Inventory which S3, Redshift, EMR, Glue and Athena workloads Airflow will orchestrate and who owns each one.
Agree the IAM connection design, including role assumption paths and whether Airflow runs inside or outside AWS.
Confirm the Airflow version, metadata database and executor choice in a fixed-scope written agreement before work starts.

How we build

Configure the Amazon provider and its connections, storing credentials in your chosen secrets backend rather than DAG code.
Build sample DAGs for transfers and job submissions, with sensors and deferrable waiting where your scope requires them.
Test each operator path against non-production AWS resources and record the results for your review.

How we hand over

Deliver runbooks covering connection rotation, IAM policy changes, job failure triage and cost-relevant job settings.
Document the architecture, network paths to AWS endpoints and the list of permissions granted to each connection.
Walk your platform and data teams through the DAG delivery process and the operator patterns agreed in scope.

How we support

Ongoing support, upgrades of the Airflow and Amazon provider versions, and continuous testing are available as a retainer.
Track provider release changes and Airflow version support dates so upgrades are planned before a line goes end of life.
Confirm with your AWS account owners which support terms, service quotas and account limits apply to your environment.

Apache Airflow with AWS: S3, Redshift, EMR, Glue and Athena