Apache Airflow for healthcare and life sciences

Why Apache Airflow fits healthcare pipelines that must stay in your network

Health organizations run scheduled jobs that move electronic protected health information (ePHI) between EHR, claims, laboratory, FHIR and imaging systems every night and many times a day. Apache Airflow 3 can run entirely inside your own cloud accounts, data centers or restricted networks, so the pipelines that touch patient data never need to leave infrastructure you control. Airflow is licensed under the Apache License 2.0, and the Apache Software Foundation project offers no paid edition, so the software cost is the cost of the infrastructure, the people and the support you choose to buy. Airflow 2 reached end of life on April 22, 2026, and gets no fixes, so any new healthcare deployment should start on Airflow 3.

Self-hosting changes what you must control. The scheduler, the DAG delivery path, the secrets backend, and the task logs stop being vendor features and become controls your organization designs and evidences. For example, you need to decide which metadata database holds run history (the Airflow production guide recommends PostgreSQL or MySQL, not the SQLite default), how DAG code is reviewed and promoted, where connection credentials are retrieved from at runtime, and whether task logs that may contain identifiers are stored, masked and retained under your policy. Ask your team to document who can change DAGs, who can read connections, and where each log lands before any ePHI flows through the platform.

Plan the operating model before the first pipeline ships. Confirm that your team can keep up with Airflow's supported versions and the Python, PostgreSQL and Kubernetes prerequisites for the release you choose, and that someone owns security patching during the limited-maintenance window. The scheduler supports more than one instance using database row-level locks, which matters when clinical reporting cannot wait for a restart. A partner or internal team should write these decisions down as a design record, because the HIPAA Security Rule expects documented risk decisions, and an auditor will ask how each Airflow control maps to your obligations.

Which healthcare requirements shape the Airflow build

This table maps each healthcare requirement in scope to its source and to what it means for an Airflow design. Read it as a checklist for the architecture decisions, access model and evidence you need before the first pipeline carries ePHI or clinical data.

Requirement	What it asks for	What it means for the Airflow build
HIPAA Security Rule scope	Applies to covered entities and business associates for all ePHI they create, receive, maintain or transmit (45 CFR 164.302)	Decide early whether your Airflow environment is inside scope as a covered entity or as a business associate, since every DAG, log and metadata record that touches ePHI falls under it
Required and addressable specifications	Appendix A to Subpart C lists 36 implementation specifications: 14 required and 22 addressable (45 CFR 164.306(d) governs addressable ones)	Each addressable control, such as encryption or automatic logoff, needs a documented decision on whether it applies to your Airflow components and, if not, an equivalent measure
Technical safeguards for access	Unique user identification and emergency access procedures are required; automatic logoff and encryption are addressable (45 CFR 164.312)	Give every DAG author and operator an individual identity, plan a break-glass path for the scheduler and metadata database, and set session and encryption settings for the webserver and API server
Business associate contracts	Must require compliance with the Security Rule, flow terms down to subcontractors handling ePHI, and report security incidents (45 CFR 164.314(a)(2)(i))	If a vendor, hosting provider or support partner can reach ePHI pipelines, the contract chain must cover them, and your incident runbook must notify the covered entity on the contracted terms
No HIPAA certification	HHS states no Security Rule standard requires certification, and private certifications do not prevent a later violation finding (HHS FAQ 2003)	Do not accept a vendor or partner claim of HIPAA certification as evidence. Build a control evidence set such as configuration baselines, access reviews and run logs that an assessor or OCR investigator can inspect
OCR enforcement and penalties	OCR enforces through complaints, compliance reviews and audits; the inflation-adjusted penalty runs from $145 to $2,190,294 per violation for penalties assessed on or after January 28, 2026 (45 CFR 102.3)	Logging, change records and access evidence matter as much as the controls themselves, because an investigation reconstructs what happened to ePHI pipelines after the fact
GDPR Articles 32 and 34 (2026 CNIL fine)	Article 32 covers security of processing; Article 34 covers communicating breaches to data subjects. CNIL fined a French hospital EUR 500,000 on 3 September 2026 for external access without VPN or multi-factor authentication and access control ignoring need to know	For EU patient data, enforce multi-factor authentication and VPN or equivalent controls on any remote access path to Airflow, and scope DAG and connection permissions to the care or research team that needs each dataset
NIS2 Annex I health coverage	Lists health as a sector of high criticality, including healthcare providers and medical device manufacturers (Directive (EU) 2022/2555)	If you operate in a covered EU health entity, pipeline availability and incident reporting for clinical or regulatory data may fall under NIS2 obligations, so plan recovery and restart procedures for critical schedules

Choosing a deployment model for clinical data

Hospitals and provider networks usually keep clinical source systems such as the EHR, laboratory systems and the imaging archive in a protected clinical zone. Airflow then runs in a separate data zone and reaches the sources through narrow, logged paths. Identified ePHI should stay in the clinical or integration zone, and the data zone should receive only the fields each pipeline needs.

Pipelines by zone: Run extraction from clinical sources in a dedicated integration zone, run transformations and analytics loads in the data zone, and define the boundary crossing for each DAG.
Metadata database: Place the Airflow metadata database in the same zone as the scheduler and workers that use it. The production guidance recommends PostgreSQL or MySQL, SQLite is for testing only, and MariaDB is not supported as the metadata database.
Kubernetes and images: The Airflow 3.3.2 prerequisites list Kubernetes 1.30 to 1.35. Confirm your cluster version, then deploy from container images with the official Helm chart so the same image moves between environments.
Vendor and business associate access: Vendors that host, support or process ePHI for you need a business associate contract under 45 CFR 164.314(a)(2)(i). Give them access through a monitored jump host with individual accounts.
Scheduler resilience: Airflow supports more than one scheduler at the same time using database row-level locks, which matters when clinical loads must finish before morning reporting.

Confirm which zone each DAG's source connection lives in and who can read the metadata database, since task logs and the database can expose record identifiers. Read the Airflow production deployment guide before finalizing the database and container setup.

Which Airflow 3 components and integrations matter for ePHI

Each card covers one part of a self-hosted Airflow 3 platform that touches electronic protected health information. The choices connect: where the metadata lives, which components run, who can see which DAGs, and how credentials reach tasks. Settle these before the first clinical pipeline goes live.

API server and DAG processor as separate parts

In Airflow 3 the webserver is replaced by an API server, and the DAG processor must run as its own component. Each one is a separate deployment target with its own logs, resource limits and network rules. For ePHI, decide which network segment can reach the API server and confirm that DAG parsing does not need access to patient data stores. Document the component map so auditors can trace where pipeline definitions and run history are served.

Metadata database: PostgreSQL or MySQL

The metadata database stores DAG runs, task states, connections and audit-relevant history. Airflow supports PostgreSQL 14 to 18 and MySQL 8.0, 8.4 and Innovation releases, but not MariaDB, and SQLite is for testing only. Because run history may reference patient cohorts or record counts, the database falls under your encryption, backup and access controls. Confirm encryption at rest, restore testing and retention before go-live.

Multiple schedulers and executor choice

Airflow can run more than one scheduler at once, coordinated through database row-level locks, which supports resiliency for time-bound clinical feeds. Executor choice, such as CeleryExecutor or KubernetesExecutor, determines where task code runs and therefore where ePHI is processed. Hybrid executors are no longer supported in Airflow 3. Map each executor to a network zone, and confirm that worker nodes handling ePHI are the ones you have hardened and logged.

Simple versus FAB auth manager and role mapping

Airflow 3 uses the Simple auth manager by default and allows only one auth manager at a time, set through the auth_manager option. The FAB auth manager comes from a provider package. Clinical operations, research data stewards and integration engineers need different DAG visibility, so plan role mapping before users log in. The Simple manager is generally unsuitable for named, reviewable access to ePHI pipelines, so confirm which manager you will use and how group membership maps to care and research roles.

Secrets for database and FHIR credentials

Pipelines authenticate to EHR exports, FHIR endpoints and imaging archives, so connection credentials are the most sensitive configuration in the platform. Keep them in an enterprise secrets backend you already operate, not in DAG files or plain connection records. Check who can read connections in the UI and API, how credentials rotate, and whether task logs can echo secret values. Treat any credential that unlocks ePHI as in scope for your access reviews.

HAPI FHIR JPA server and imaging connections

Pipelines that load or reconcile FHIR resources often target a HAPI FHIR JPA server. The starter supports PostgreSQL and Microsoft SQL Server for production and does not support MySQL, so the FHIR store and Airflow metadata database may end up on different engines. HAPI FHIR does not provide one security layer, so authorization, consent and audit logging must be designed around it. Confirm imaging system interfaces, such as DICOM endpoints, with their owners before scheduling transfers.

How Airflow logs and access controls map to HIPAA technical safeguards

The technical safeguards in 45 CFR 164.312 state outcomes that you must translate into Airflow settings, identity integration and retention rules, each with a named owner. Unique user identification and the emergency access procedure are required implementation specifications, so every person who can view DAG runs, task logs or connections needs an individual account, and any emergency access needs a written procedure. Automatic logoff and both encryption specifications are addressable. Audit controls, the standard for recording and examining activity in systems that hold ePHI, is where Airflow's scheduler, API server and task logs become evidence. Confirm with your compliance lead which specifications apply to each Airflow component before the design is fixed.

In Airflow 3, the auth manager decides who can sign in and what they can see, and only one can be active at a time, set by the auth_manager option. The Simple auth manager is the default, so it should not be carried into an ePHI environment without a documented review. The FAB auth manager comes from the FAB provider, and the Keycloak and AWS options are covered on the authentication page, so choose the one that ties each person to a named account and to a role matching the care or research team that needs access. For evidence, the metadata database should be PostgreSQL or MySQL, because SQLite is for testing only and MariaDB is not supported. Send logs to a central store with retention set by your policy, since database cleanup and log rotation will otherwise remove the record of who ran what.

The addressable specifications are your decision, and you must document it. For each one, record whether it is reasonable and appropriate for your deployment under 45 CFR 164.306(d). If you decline one, document why and the equivalent measure you use. For example, if automatic logoff is enforced by the identity provider's session timeout, record that choice and test it against the Airflow web session. Airflow 3 also constrains how evidence is produced: task code can no longer use Airflow database sessions or models directly and must use the REST API or Task SDK. Any control that records who moved which ePHI, along with row counts or checksums, should be written through those interfaces. Review that write path during design so an auditor can trace each record back to a specific DAG run.

How a healthcare Airflow rollout runs in phases

The sequence moves from defining which ePHI flows Airflow touches to a controlled cutover, with each phase closing on evidence that the next one can rely on.

Scope ePHI flows and regulatory roles
Start by listing every pipeline that will create, receive, maintain or transmit electronic protected health information, including source systems, destinations, and intermediate storage. For each flow, record whether your organization acts as a covered entity or a business associate, since that determines which Security Rule obligations and business associate contract terms apply (45 CFR 164.302 and 164.314). Your privacy, legal and security leads sign off on the inventory. The step closes when every flow has an owner, a data classification, and a documented regulatory role, and any gap is logged as a decision.
Build a non-production environment inside the network
Deploy a staging Airflow in the same network zone as production, using the same major version, a supported Python and PostgreSQL combination from the prerequisites, and the same container or Kubernetes pattern you plan to run live. Keep it on infrastructure you control, with no ePHI copied in at this stage. Confirm that the metadata database is PostgreSQL or MySQL rather than SQLite, because the production guidance warns that SQLite risks data loss. The step closes when the environment installs from your approved images, the scheduler and API server start, and a sample DAG runs end to end.
Configure identity, secrets and logging
Connect the workforce identity provider through the auth manager you select, map roles to the teams that author, operate and audit DAGs, and remove shared accounts. Move connection credentials into your enterprise secrets store so they are not stored in DAG code or plain environment files. Route task, scheduler and API server logs to your central logging platform with retention set by your records policy, and confirm that access to logs containing patient identifiers is restricted. The step closes when a test user can log in with only the roles assigned to them, a secret rotation is performed without editing DAG code, and a deliberate failed task appears in the central log store.
Validate DAGs and interfaces with test data
Run each DAG against synthetic or de-identified data first, then against test endpoints of the FHIR, imaging or EHR interfaces it calls. Check that the DAG code passes the Airflow 3 upgrade checks if you are migrating from Airflow 2, and that transport uses encrypted connections with certificate validation. Record how the pipeline handles malformed messages, partial loads and retries, so that no failed batch leaves a partially written record set without an alert. The step closes when each DAG has a signed test result covering success, failure and re-run cases, and each interface owner confirms the test endpoint behaves as documented.
Document access, audit and recovery evidence
Write the operating procedures that auditors and your Security Rule risk analysis will ask for: who can change DAGs, how changes are reviewed and promoted, how access is granted and removed, how backups of the metadata database are taken and restored, and how audit logs are reviewed. Map each procedure to the HIPAA safeguard it supports, and note which implementation specifications you treat as addressable and how you documented the decision (45 CFR 164.306(d)). Tell your compliance team that a HIPAA certification does not exist, so the evidence is for your own assessment and any OCR review. The step closes when the document set is approved and a restore test from backup has been completed in staging.
Run a controlled cutover with a rollback plan
Move production schedules in groups rather than all at once, keeping the existing scheduler or manual process available until each group has produced verified output. Agree the cutover window, the named decision owner, and the criteria for a go or no-go decision in writing before the change starts. Write a rollback procedure that re-enables the previous schedule, pauses the new DAGs, and states how any partially processed data is identified and reconciled. Rehearse the rollback in staging first. The step closes when each cutover group has passed its verification check, the rollback procedure has been exercised, and the change record is filed with the security and compliance teams.

What compliance evidence the organization must still produce

A self-hosted Airflow deployment can support HIPAA and related obligations, but no outside body certifies it on your behalf. The evidence has to come from your own risk analysis, your documented controls and the records your pipelines produce, with external assessments added only where your contracts or regulators call for them.

No HIPAA certificate exists

HHS states that no Security Rule standard requires certification and that it does not recognize private certifications. Any statement that a platform or vendor is "HIPAA certified" should be treated as a marketing claim. Your evidence is your own risk analysis, documented policies, configuration records and the results of your reviews of the Airflow deployment, including its scheduler, workers, metadata database and logs. Independent auditors and certification bodies issue reports and attestations. DeployOpen prepares systems, controls and evidence for those assessments but does not issue them.

NIST SP 800-66 Rev. 2 as a mapping aid

NIST SP 800-66 Rev. 2 gives key activities, descriptions and sample questions for each Security Rule standard. HHS lists it as informational rather than legally binding, so use it to structure your mapping from Airflow controls, such as DAG change review, role-based access, secrets handling and audit logging, to the standards they support. Record which Airflow setting or procedure answers each sample question and where the evidence lives, so an auditor can trace each claim to a configuration or log.

Pending Security Rule revision

HHS proposed a major Security Rule revision on January 6, 2025, which would tighten cybersecurity expectations for electronic protected health information. As of October 2026 no final rule has been published, and the Unified Agenda projects final action for July 2027. Build your Airflow evidence so it can absorb stricter requirements, such as documented asset inventories, faster incident reporting and tested recovery, without redesigning the pipelines. Track the rulemaking and budget a review of your control mapping when a final rule is issued.

HITRUST as an optional assessment

HITRUST CSF is a privately maintained control framework that some payers, providers and business partners request in contracts. It is not required by HIPAA, and a HITRUST report is issued by an assessor under HITRUST's process, not by DeployOpen or Airflow. If a partner requires it, confirm the CSF version, the assessment type and the scope, and confirm whether the Airflow hosts, the metadata database, the secrets store and the log pipeline fall inside that scope. Ask early, because scope decisions affect how you segment the deployment.

GDPR and European Health Data Space context

If your organization processes personal data of people in the EU, GDPR Article 32 requires security measures suited to the risk, and the CNIL's 2026 fine against a French hospital cited missing multi-factor authentication for external access and access rights that ignored the care team's need to know. Regulation (EU) 2025/327 on the European Health Data Space adds requirements for health data exchange. For Airflow, document who may trigger, edit and read each DAG, how cross-border transfers are controlled, and where logs containing personal data are retained.

21 CFR Part 11 for regulated research records

Where pipelines produce records used in FDA-regulated research or submissions, 21 CFR Part 11 applies to electronic records and electronic signatures. Airflow's run history, task logs and DAG version history can contribute to that evidence, but you must validate the system for its intended use, control who can change pipelines, and confirm that audit trails and signature practices meet the rule. Confirm these requirements with your quality function before you scope the deployment, since not every research pipeline falls under Part 11.

What healthcare teams ask before scoping Airflow

What drives the cost of a healthcare Airflow project?

The written scope sets the price, and DeployOpen does not publish a price list. The main drivers are the number of source and target interfaces (EHR feeds, FHIR endpoints, imaging systems, claims data), how much ePHI the pipelines touch, how many environments hold that data, the identity integration needed for SSO and role mapping, and the volume of audit evidence to produce. Build work, hardening, testing and documentation are one-time costs. Infrastructure you run, staff time, upgrades and any ongoing support retainer recur. Ask for a quote that itemizes cost per interface and per environment, and confirm whether testing and evidence packs are included.

How long does a healthcare Airflow implementation take?

No researched duration applies to this scope, so the timeline has to come from the written scope for your environment. The factors that move it most are the number of interfaces, how quickly architecture and privacy reviews are approved, how fast your teams grant access to source systems, how long environments take to provision, and whether migration from an existing scheduler is included. Most engagements are fixed-scope, so deliverables and dates are agreed in writing before work starts. Ask for milestones tied to the approvals and access you must provide, so delays on your side are visible early.

What must our organization provide before implementation starts?

Expect to supply four things before build work begins:

Architecture and data flow approvals from your security and privacy officers, covering where ePHI is stored, processed and logged.
Access to interfaces, including test endpoints, service accounts and sample messages with identifiers removed.
Security staff time for access reviews, network path decisions and sign-off on hardening choices.
A named product owner who can decide scope changes and accept deliverables.

The written scope should list each of these with an owner and a date, because the schedule depends on them as much as on the build.

Can our own team deploy Airflow for ePHI workloads?

A capable internal team can do it, provided it has Airflow operations experience, container or virtual machine operations skills, and security staff available for design decisions. Airflow 3 changed the architecture: the webserver is replaced by an API server, the DAG processor runs as a separate component, and task code must use the REST API or Task SDK instead of database sessions. Each HIPAA addressable implementation specification also needs a documented decision on whether it is reasonable and appropriate. Many organizations bring in a partner for the design and hardening phase and keep operations in-house. Decide which part your team owns before you start.

Is Apache Airflow itself HIPAA certified, and does that matter?

No. Apache Airflow is open-source software released under the Apache License 2.0, and no HIPAA certification exists for it or for any other product. HHS states that no Security Rule standard requires an entity to certify compliance and that it does not recognize private certifications. HIPAA compliance is an obligation of the covered entity or business associate that operates the system. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies do that work, so ask any vendor who claims otherwise which assessor issued the claim.

How are Airflow 2 upgrades handled for healthcare pipelines?

Airflow 2 entered limited maintenance on October 22, 2025, and reached end of life on April 22, 2026, so it no longer receives fixes. The upgrade guide requires being on Airflow 2.7 or later, recommends moving to the latest 2.x first, and suggests checking DAG code with Ruff AIR301 and AIR302 rules. Airflow 3 also removes SubDAGs in favor of TaskGroups and changes how tasks reach the database. A healthcare upgrade should run in a validated test environment, with change records, rollback steps and evidence that pipelines produced the same results. Confirm your target Airflow 3 release with the current version support table before scoping.

Does HIPAA apply to pipelines that move ePHI through Airflow?

It applies when your organization is a covered entity or a business associate and the pipelines create, receive, maintain or transmit electronic protected health information. The Security Rule covers that ePHI wherever it sits, including task logs, connection configuration and the metadata database, so confirm which of these may contain identifiers in your design. If a partner accesses ePHI during the work, a business associate contract must require Security Rule compliance, flow the same terms to subcontractors, and require incident reporting under 45 CFR 164.314. Have your legal and privacy teams confirm the contract terms before any access is granted.

Which Airflow deployment choices matter most for protected health data?

Four choices shape the security posture:

Use PostgreSQL or MySQL as the metadata database. The production guide warns that the default SQLite backend can lead to data loss, and MariaDB is not supported.
Choose one auth manager deliberately, since Airflow 3 allows only one at a time and the Simple auth manager is the default.
Plan executor placement so that workers handling ePHI run where your network controls apply.
Set retention and access for logs, since they can contain identifiers.

Write each choice down with its reason, because auditors and HHS reviewers will ask how the decision was made.

How DeployOpen scopes and delivers healthcare Airflow work

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own infrastructure, including Apache Airflow for healthcare and life sciences. Work starts from a written scope that sets out which pipelines, data sources and controls are in play before any build begins.

How we scope

Inventory which pipelines touch ePHI, which source systems feed them and where each dataset is stored.
Map each pipeline to the HIPAA Security Rule safeguards it affects, and agree which controls are in the written scope.
Confirm the network zones, hosting location and identity source the Airflow deployment must sit within.
Agree the FHIR, imaging or other interfaces in scope, and name the owner who signs off each data flow.

How we build

Deploy Airflow 3 on PostgreSQL or MySQL with the executor chosen for your workload and network placement.
Set up DAG delivery, secrets storage and SSO so that credentials and clinical data stay inside your environment.
Configure audit logging, access roles and task isolation so that pipeline activity can be traced for each user and run.
Test restore procedures and failure paths against a staging copy before any production pipeline is switched over.

How we hand over

Document the architecture, the access model, the configuration baseline and the recovery steps your team will run.
Provide runbooks for upgrades, including the Airflow 2 end-of-life date and the DAG checks needed before moving to 3.x.
Deliver evidence artifacts that map each control to the implementation, for your compliance lead and auditors to review.
Walk your platform and data engineering staff through operations, incident handling and change control.

How we support

Ongoing support, upgrades and continuous testing are available as retainers, scoped in writing alongside the build.
Retainers can cover patching, Airflow version upgrades, backup checks and periodic security testing of the deployment.
Ask for the support boundaries, response expectations and escalation path in the written scope before signing.