Apache Airflow human-in-the-loop approvals

Apache Airflow human-in-the-loop approvals for data releases and change control

When a pipeline publishes data to a downstream consumer or changes a production system, someone accountable often has to sign off before the final step runs. Airflow 3.1 introduced human-in-the-loop operators for this purpose, and they are available in Airflow 3.1 or later only, so a team on an earlier 3.x release or on Airflow 2 needs to upgrade or build a workaround first. Confirm the exact operator names, parameters and approval behavior against the 3.1 documentation before you design around them, because the names and options are the part most likely to differ from what you read in older blog posts or community examples.

Apache Airflow is licensed under the Apache License 2.0 and has no paid edition from the project, so an approval step is not something you buy as a feature. Your team or a partner builds it, and the cost sits in design, integration and operations: deciding who may approve which run, how the decision is recorded, what happens on rejection or timeout, and how the approval interacts with retries and reruns. Those decisions carry more weight than the operator itself, and they should be written down before any DAG is changed.

The main decision is whether an approval gate belongs in the pipeline at all. Gates suit a data release, a change to a shared table or a configuration push that a control owner must review, and they fit poorly on high-frequency loads where a wait would stall the schedule. Ask the control owner which approvers are authorized, how approval is evidenced, and whether an approval is valid for one run or for a set period. Then check that the chosen approver identities map onto the roles your auth manager enforces, since an approval that anyone in the workspace can give is not a control.

Where approval gates fit: release, change control and review steps

A human gate is worth its wait only where a person adds judgment that code cannot supply, such as whether a dataset is fit to publish or whether a change should reach production. The cards below cover situations where that judgment is needed, what the reviewer checks, and what the pipeline does on approve, reject and no response.

Publishing a curated data mart

Before a curated mart feeds dashboards or external reports, a reviewer can confirm that row counts, reconciliation totals and known exceptions look plausible for the period. On approve, the publish task runs and downstream refreshes start. On reject, the run stops before publication and the owner receives the reason. On no response, the pipeline should stay paused with a defined timeout rather than publish by default. Confirm how your team wants timeouts and reminders handled.

Promoting a schema change

A schema change can break consumers that code checks do not cover, so a reviewer decides whether the change is intended and whether consumers have been notified. On approve, the migration and downstream tasks continue. On reject, the pipeline keeps the previous schema and the change returns to the author. On no response, the change should not proceed, so the gate needs an owner who is reachable during the release window.

Releasing a regulatory report

For a regulatory submission, the reviewer checks that the source extracts, reconciliations and sign-off from the business owner are complete for the reporting period. On approve, the submission task runs and the approver's identity is recorded in the run history. On reject, nothing is submitted and the run captures the reason for audit. On no response, the deadline risk should be escalated through a separate alert, because a silent gate can cause a missed filing.

Confirming a backfill before downstream load

A backfill can reprocess months of history and push a large volume of changes into systems that other teams depend on. A reviewer checks the scope, the reprocess behavior and whether downstream owners are ready for the load. On approve, the backfill continues into the publish tasks. On reject, the run ends before any downstream write. On no response, the backfill should remain blocked so that the load does not start unannounced.

Limiting gates to judgment calls

Each gate adds waiting time, and a run that waits for a person can hold workers, delay dependent DAGs and miss its window. Limit gates to steps where a person weighs context that code cannot check, and let automated tests and data quality checks handle the rest. Review gate placement each quarter, and confirm with your team how waiting runs are counted against capacity before you rely on them in production.

How a human approval pauses and resumes a run

A human approval gate pauses a run without freezing the platform around it, and the path it takes depends on Airflow 3's split architecture. The API server replaces the webserver as the component people and clients talk to, so the approver's decision reaches Airflow through that API rather than through a direct database write. The DAG processor runs as its own component, parsing DAG files separately from the scheduler and API server. Task code never opens Airflow database sessions in Airflow 3. It reports state and reads context through the REST API or the Task SDK, so an approval operator that waits on a person has to use those same interfaces to record its wait and pick up the answer.

While the gate is open, the run is in a waiting state that you need to pin down for your own version. Confirm from the 3.1 operator documentation which task state the gate uses, what the approver sees in the UI (the prompt, the options, any form fields and who is allowed to respond), and what happens when nobody answers within a timeout. Also confirm whether a waiting task holds a worker slot or releases it. That answer decides how many concurrent approvals your executor can carry, and it differs between the executors Airflow ships, so treat it as an open question until you test it on the executor you plan to run.

After a decision, the run either continues into downstream tasks or fails, and the handling of rejection, timeout and a retry after either is where pipelines break. Test each path on a staging deployment: approve, reject, let the timeout expire, restart the scheduler or API server while the gate is open, and clear the task to see whether it asks for approval again. Record which outcomes mark the run as failed, which trigger your alerts, and which leave a downstream release blocked. Those results should drive the design of the approval gate, not the operator's default behavior alone.

Approval step options and when each one fits

This table compares the configuration decisions a pipeline designer makes when adding a human approval or input step in Airflow 3.1 or later. Read each row as a question to settle before the DAG is built, and confirm the exact option names and defaults against the 3.1 documentation for your version.

Decision	Options and when each fits	Trade-off	What to confirm
Response type	Approve or reject for release gates, change sign-off and go/no-go checks where the outcome is binary. Free-text or form input for steps that need a reason, a parameter value or a correction before the run continues.	Binary decisions are quick to audit but carry no context. Form input captures more detail but adds validation work and more paths the DAG must handle.	Which response types the 3.1 operators expose and how each response is passed to downstream tasks
Who may respond	Named users for small, fixed approver lists such as a data owner for one dataset. Groups or roles mapped through the auth manager for teams where approvers rotate.	Named users are simple but go stale when people change roles. Role-based access depends on the auth manager you run, so the mapping must be tested before go-live.	Whether the operator restricts responders by user, group or role, and how your auth manager (Simple, FAB, Keycloak or AWS) passes group membership
Response timeout	Long timeouts suit reviews that wait on people, such as weekly data release windows. Short timeouts suit time-boxed operational checks where a stalled run blocks a schedule.	A long wait holds a task slot or deferred state and can push downstream runs late. A short wait can fail a run that a reviewer simply had not seen yet.	Whether the timeout setting exists in 3.1, its default value and whether a waiting task holds worker resources
Default outcome on timeout	Fail the run when the gate protects a release or a change. Use a documented fallback only where the business accepts that outcome, and record it in the run.	A fail default is safe but needs a rerun process. A pass default can release unreviewed data, which is hard to explain to an auditor.	Which outcomes the operator can take on timeout and whether a timeout is logged as a distinct event
Notification channel	Airflow UI alone for teams that already monitor the console. Chat or email notifications through a provider or callback for reviewers who are not Airflow users.	UI-only routing relies on approvers checking the console. External channels add a provider dependency and a risk that messages carry run details out of the controlled environment.	Which notification integrations are available for the 3.1 operators and whether message content can be limited
Decision recorded as run metadata	Record the approver, decision and time for release and change control, where an assessor will ask who approved what. Keep decisions out of run metadata only where no evidence is required.	Recording decisions adds retention and access questions for the metadata database. Skipping records weakens the audit trail for any gate that matters.	Where decision details are stored, how long the metadata database retains them and how they are exported to your evidence store
Gate placement	Before a publish or write task for data releases. Before a production change step for change control. After a review task when the reviewer needs the output in front of them.	Gates placed early delay everything downstream. Gates placed late may run expensive work that the reviewer then rejects.	Whether the gate can be placed inside a task group or dynamic mapping without changing how retries behave
Separation of duties	Separate the approver from the DAG author and the run trigger when the gate supports a control objective. A single approver team may suffice for low-impact internal steps.	Stronger separation adds approver roles and a review process for the role model. Weak separation leaves the gate open to self-approval.	Whether the operator blocks a user from approving a run they triggered, and how your role model enforces that

Setting up an approval gate in an Airflow 3.1 or later DAG

This sequence adds a human approval step in front of a data publication task, so a named approver decides whether the run continues. It ends with tested approve, reject and timeout paths and an audit trail that shows who decided and when.

Confirm the Airflow version and auth manager
Confirm that the target environment runs Airflow 3.1 or later, ideally on the current 3.3 line, since the approval operator depends on that release. Record the auth manager in use, because Airflow 3 allows only one at a time, set by the auth_manager option. Review the 3.1 documentation for the exact operator name and parameters before writing code. Done when the version and auth manager are written down and the team has the documented operator signature.
Build the gate in a test environment
Create the approval DAG in a non-production Airflow instance that matches production on version, executor and auth manager. Keep the DAG in a test bundle or dags folder that production does not read. Check that the DAG parses without import errors and that the approval task appears in the UI as waiting for input. Only move on when the waiting state appears and the upstream tasks have completed as expected.
Define the approval task and its approvers
Set the question text, the allowed responses, and the list of users or roles permitted to respond. Make the context shown to approvers specific, such as the run identifier, the dataset version and the target system. Confirm with the operator documentation whether approver restrictions are enforced by the operator itself or depend on auth manager permissions. Check that a user outside the allowed list cannot submit a response in the UI.
Wire the publication task downstream
Place the publish task downstream of the approval task so that it runs only after an approve decision. Decide how the reject path should appear in the graph, and confirm which task states the operator and trigger rules produce on rejection in your version. Check that a rejected run leaves the publish task in a non-success state and writes nothing to the target system. Record the expected states in the runbook.
Configure auth manager roles for approvers
Map approvers to a dedicated role in the auth manager your environment uses, and keep that role separate from the role that triggers runs. Confirm whether the operator prevents the user who triggered a run from approving it, and configure your own separation rule if it does not. Check with three test accounts: an approver can respond, a read-only viewer can see the task but cannot respond, and a DAG author cannot approve their own run where your design forbids it.
Test the approve, reject and timeout paths
Run three test cases: approve, reject, and let the task time out using the timeout setting you configured. For each, confirm the publication task outcome and check the audit entry, which should show the decision, the decider's identity and the time. Confirm in the documentation what state a timeout produces in your version. Done when all three paths behave as designed and the audit entries can be exported to your log store.
Promote the gate with a change record
Deploy the approved DAG to production through your normal change process, with the approver list and timeout behavior recorded in the change ticket. Run one controlled production execution and confirm that the approval request reaches the intended approvers. Check that the production run's approval decision and audit entry can be linked to the change ticket. The gate is ready when the change record, the approver list and the first production decision all match.

Who can approve, and how decisions are recorded

An approval gate is only as trustworthy as the identity behind each decision and the record left behind. The cards below cover who may approve, how duties are split, what is logged and retained, what the approval view exposes, and how gates behave in restricted networks.

Approver identity and role mapping

Approvers are identified by the auth manager configured for the Airflow deployment. The Simple auth manager is the default and is not a substitute for enterprise identity. Confirm which auth manager is set through the auth_manager option, how users map to roles, and whether a departing approver's access is removed in the identity provider rather than only in Airflow. Plan the connection to your identity provider before any gate goes into a production release path.

Separation of duties

Decide whether the person who wrote or changed a DAG can also approve its run. In many change control designs they cannot, so the gate should require a reviewer with a different identity and role. Check that role assignments in Airflow do not give DAG authors approval rights by default, and test the rule with a deliberate self-approval attempt before relying on it.

Logging and retention of decisions

Record who approved or rejected a step, when, what input they provided, and which run was affected. Decide where those records are stored, how long they are kept, and who can read or alter them. Audit evidence for change control depends on how your team configures logs and retention, so confirm the log destination and retention period with your compliance owner before treating the trail as evidence.

Data shown in the approval view

The approval view displays context so a reviewer can decide, which can include row counts, sample values or report links. Treat anything shown there as exposed to every user with approval access. Keep personal or regulated fields out of the displayed parameters, mask where needed, and check what the task logs and the metadata database retain when a decision is entered.

Gates in air-gapped or restricted networks

A gate that waits on a person still needs the scheduler, API server and database to reach each other inside the restricted network. Confirm that approvers can reach the Airflow UI through your approved access path, that no step calls an external approval service unless that service is inside the boundary, and that a missed or stalled approval does not leave a release half-applied. Test the stalled-gate path during rollout.

Running approvals on Airflow 3.1 and later versus older deployments

Start here if your deployment already runs Airflow 3.1 or a later release. Approval gates depend on the Airflow 3 architecture, so the design work begins with the components that serve the approval step and decide who can answer it. Confirm these points before you write the first gate.

Confirm the exact version: Run airflow version on the scheduler host and compare the result with the version support table. The current patch release is 3.3.2, and staying on a supported patch line keeps fixes available.
Check the API server: In Airflow 3 the webserver is replaced by the API server, started with airflow api-server. Confirm how it sits behind your load balancer and that the DAG processor runs as its own component, because both affect how approvers reach the prompt.
Settle the auth manager: Airflow 3 uses the Simple auth manager by default and allows only one auth manager at a time, set by the auth_manager option. Decide where approvers are authenticated before the gate is built, since changing the auth manager later changes who can respond.
Confirm the metadata database: Use PostgreSQL or MySQL. A waiting gate holds its state in the metadata database, so SQLite is not suitable for any environment where a person may take time to respond.
Write the gate specification: Name the approvers, the allowed responses, the behavior when no one responds, and the record each decision leaves in the logs.

Before you design gates, check each of these items against the prerequisites for your exact version.

Limits to confirm before you design around approvals

Approval gates depend on how Airflow schedules, waits, restarts and versions tasks in your environment. The items below are what to confirm in a test cluster before a release or change-control process depends on them.

Operator support by executor and Kubernetes

Confirm that the Airflow 3.1 human-in-the-loop operators run on the executor you plan to use and on your Kubernetes version. The 3.3.2 prerequisites list Kubernetes 1.30 to 1.35, so a cluster outside that range needs a documented exception or an upgrade first. Check the executor and operator combination in a staging environment, not only in the release notes.

Resources held by waiting tasks

Measure how long a task that waits for approval occupies a worker slot, pool entry or pod. A gate that waits hours or days can starve other work if it holds concurrency. Ask whether the wait releases its slot, how it counts against pool and DAG concurrency limits, and what the cost is on your executor when many approvals are open at once.

Behavior during scheduler and API server restarts

Restart the scheduler and the API server while an approval is pending, then confirm the task still shows as waiting, accepts the decision and resumes. Check the same during a rolling upgrade. The scheduler supports more than one instance using database row-level locks, so confirm that a failover does not duplicate or drop a pending decision.

Timeout and retry interaction

Test what happens when the approval timeout expires and when a retry is configured. Confirm whether a timed-out gate fails, skips or re-enters the waiting state, and whether a retry opens a second approval request. An unclear result here can let a release proceed without sign-off or block it indefinitely.

Passing approval input to downstream tasks

Confirm that the reviewer's input, such as a selected environment, a comment or a rejection reason, can be read by downstream tasks through XCom or the Task SDK, and that the value is recorded with the run. Also confirm who can see that input in the UI and logs, because reviewer comments may contain sensitive context.

Approvals across DAG versions

Decide which DAG version a pending approval belongs to when the code changes mid-wait. Confirm whether a new deployment changes the gate definition for runs already waiting, and whether the approval record names the version that was reviewed. Your DAG bundle and rerun settings determine the answer, so test them together.

How much do approval gates cost to build and run?

What drives the cost of adding approval gates to Airflow pipelines?

The main driver is how many gates you need and how much each one has to connect to your existing controls. A single approval step in a DAG that already runs cleanly is a small piece of work. Costs rise with the number of distinct approval paths, the number of pipelines that need them, and the amount of decision context each approver must see before acting. Other drivers include whether approvals must be recorded for audit, whether approvers sign in through your identity provider, and whether the gate must integrate with a ticketing or change management system. Ask any partner to price the gates individually, with each path named in the written scope, so you can see which items move the total.

How long does a first approval gate take to design, build and test?

The duration depends on the scope agreed in writing, not on a standard number of days. A first gate usually involves four stages: agreeing the decision points and who may approve at each one, building the DAG pattern and the approval handling, connecting identity and any notification channel, and testing rejection, timeout and retry paths alongside the happy path. Teams that already have clear approval rules and a working Airflow 3.1 or later environment move faster, because the open questions are fewer. Ask for a timeline that lists each stage and the client actions each one depends on.

What does our team need to provide for an approval gate project?

Plan to name the people who approve each gate, the change owners who accept responsibility for the outcome, and a backup approver for each path. You will also need an identity setup that maps those people to Airflow roles, since approvers must be authorized to act on a specific task. Expect to supply a test environment, sample DAGs that represent the real pipelines, and a decision on where approval records are kept. Engineers should be available to review the design and attend test sessions. Missing approvers or unclear ownership are the most common causes of delay, so settle them before the build starts.

Can our internal team build human-in-the-loop approvals without outside help?

Yes, a team that already runs Airflow in production can build a simple gate using the operators introduced in Airflow 3.1 and the official documentation. The work becomes harder when gates must follow change control rules, record evidence for auditors, integrate with identity and ticketing systems, and handle timeouts and rejections consistently across many DAGs. Consider whether your engineers have time for design review and testing, and whether they know your approval policy well enough to encode it. A partner is most useful for the design decisions and the test plan, while your team keeps ownership of the DAGs afterward.

What recurring work follows an approval gate after go-live?

Approval gates need ongoing care because they depend on other components. Recurring work includes Airflow upgrades, since the 3.x line changes APIs and plugin behavior between releases, and periodic access reviews to confirm that only current approvers can act. Your change owners will also need to update approver lists when people move roles. Plan for test runs after upgrades and after identity changes, and for reviewing the approval history during your normal control cycle. Ask any partner whether ongoing support, upgrades and periodic testing are included in the initial scope or offered as a separate retainer.

Is the cost of approval gates a one-time or recurring expense?

Both. The design, build, integration and testing work is one-time, and it is priced from the written scope. Recurring costs come from keeping gates working: upgrades, identity maintenance, access reviews, changes to approval rules and any retainer you choose for support. Airflow itself is open source under the Apache License 2.0, so there is no license fee from the project, but infrastructure and staff time still recur. When you compare options, ask for the one-time and recurring portions separately so you can budget each one.

How does DeployOpen price approval gate work?

DeployOpen does not publish a price list. Most engagements are fixed-scope, so a quote follows from a written scope that names the gates, the integrations, the deliverables and the timeline, all agreed before work starts. Ongoing support, upgrades and continuous testing are available as retainers. To compare quotes fairly, ask each provider to list the approval paths, identity and ticketing integrations, test coverage, documentation, handover and any retainer terms in the same format.

How DeployOpen scopes and delivers approval gates

DeployOpen plans, builds and hands over human-in-the-loop approval gates for Apache Airflow inside infrastructure you control. We start from the approvals your change control and data release process already require, then fix the gate design, identity and audit trail in a written scope before any DAG code changes.

How we scope

Map each pipeline step that needs a named human decision, and who may approve it.
Confirm the Airflow version and the human-in-the-loop operators available in your target release before design.
Agree in writing which DAGs, approvers, timeout behavior and evidence records are in scope.
Confirm with your identity team how approver roles reach the Airflow UI and API.

How we build

Implement approval tasks in the DAG patterns you agreed, with clear rejection and timeout paths.
Test approve, reject and no-response paths in a non-production Airflow environment first.
Check that approval actions are logged with approver identity and are visible in run history.
Review DAG code against the Airflow 3 upgrade rules when the target version differs from your current one.

How we hand over

Deliver a runbook covering who approves, how to reject, and how to resume or clear stalled runs.
Document the gate design, role mapping and test evidence so your auditors can trace each release decision.
Walk your operators and approvers through the UI steps in a session recorded in the handover pack.
Agree a list of open items and owners for any follow-on work on identity or secrets.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Retainers can cover Airflow patch reviews and checks of approval paths after each upgrade.
Ask the support scope to name who responds to a blocked production approval and within what window.
Request a quote from the written scope, since DeployOpen does not publish a price list.