Apache Airflow for energy and utilities: why self-hosting fits grid and utility data
A grid operator or utility usually has meter readings, substation telemetry and asset records arriving from hundreds of sites, regulatory reports due on fixed dates, and control networks that must stay isolated from everything else. Self-hosting Apache Airflow 3 inside the operator's own infrastructure keeps the scheduler, the metadata database and the pipeline code within the boundary the operator already governs, so data movement between corporate systems and operational data can be designed, documented and audited by the team that owns the network. Apache Airflow is released under the Apache License 2.0 (Apache-2.0), and the project itself offers no paid edition, so the software cost is the infrastructure, staff time and any partner work the operator chooses to buy.
Moving to Airflow 3 changes how the team builds and runs pipelines. The webserver is replaced by an API server started with airflow api-server, and the DAG processor must run as a separate component, which means the design needs at least one more process to deploy, monitor and patch. Task code can no longer use Airflow database sessions or models directly and must go through the REST API or the Task SDK, so any pipeline that once queried the metadata database for state will need rewriting before it can run. Operators migrating from Airflow 2 should first confirm the current upgrade requirements in the Upgrading to Airflow 3 guide, since Airflow 2 reached end of life on April 22, 2026.
Before committing, the team should decide where the scheduler, workers and metadata database will sit relative to the operational network, and who is responsible for each component. Confirm with the internal security and compliance team which data sets may leave the corporate network, and whether the design needs a separate Airflow deployment for regulated reporting. Write these decisions down early, because they determine which executors, DAG delivery paths and secrets integrations are allowed later in the project.
Which energy sector requirements shape the Airflow build
This table maps each energy-sector requirement in scope to the decision it forces on a self-hosted Airflow build. Read each row as a question to settle with your compliance team before design starts. Scope depends on your entity registration, impact rating and jurisdiction, so confirm every row with the operator's compliance team before relying on it.
| Requirement | What it means for the Airflow build | Evidence to plan for |
|---|---|---|
| NERC CIP-002 impact classification | Classify each system that sends or receives data for a pipeline as high, medium, low or outside scope before placing Airflow components. Pipelines that touch BES Cyber System data inherit that rating. | Documented impact rating for each source, with the reasoning behind it |
| NERC CIP-005 Electronic Security Perimeter | Keep Airflow schedulers, workers and metadata databases that handle high or medium impact data inside the defined perimeter, and route remote access through an Intermediate System with encryption and multi-factor authentication. | Perimeter diagram, firewall rules and remote access logs |
| NERC CIP-013 supply chain expectations | Record the origin, version and integrity checks for Airflow, its providers, the Helm chart and container images, and set vendor notification and vulnerability disclosure terms for any outside support. | Software bill of materials, verification records and vendor contract clauses |
| NIS2 Annex I energy scope | Electricity, district heating and cooling, oil, gas and hydrogen operators fall in the high-criticality energy sector. Confirm with national transposition law whether your entity is essential or important. | Entity classification and the national authority's registration record |
| NIS2 Articles 21 and 23 duties | Article 21 requires risk-management measures and Article 23 requires incident reporting. Scheduler outages, failed regulatory feeds and access changes should be logged so they can be reported on time. | Risk register entries, incident runbooks and alert history for pipeline failures |
| Commission Delegated Regulation (EU) 2024/1366 | Applies to cross-border electricity flow data. If pipelines exchange cross-border data, confirm which sector-specific cybersecurity rules apply to the data path and its operators. | Scope analysis for each cross-border data flow |
| Enforceable dates for CIP-003-9, CIP-012-2 and CIP-015-1 | CIP-003-9 applies from 1 April 2026 and CIP-012-2 from 1 July 2026. CIP-015-1 on internal network monitoring applies from 1 October 2028, so plan monitoring hooks now. Confirm dates against the NERC CIP standards page. | Dated compliance plan tied to each standard's enforcement date |
How to separate meter, SCADA-adjacent and corporate data flows
The corporate zone is the most natural home for the Airflow control plane. The scheduler, the API server, the metadata database and the DAG repository can run here, close to the analytics platform, identity services and the change management tooling your teams already operate. Most reporting pipelines, such as regulatory submissions, market data and finance loads, start and end in this zone, so keeping the orchestrator here keeps its dependencies simple.
- Control plane placement: The scheduler, DAG processor and API server run here, with the metadata database on a PostgreSQL or MySQL service you operate. SQLite is not an option for production.
- DAG code: Store DAG code in your internal Git service and promote it through review and CI before the scheduler parses it.
- Secrets: Source connection credentials from your enterprise vault rather than from DAG files or the metadata database.
- Boundary flows: Pull meter and SCADA-derived files that have already passed through an approved transfer path, and never open inbound sessions from the corporate zone into the operational zone.
- Workers: Place workers here only for tasks that touch corporate systems. Tasks that reach into operational networks belong on workers located in that zone.
Confirm with your CIP scoping team which corporate systems, if any, can influence BES Cyber Systems, because that determines whether the corporate zone itself needs stricter controls.
The operational zone sits outside the Electronic Security Perimeter. It is where workers that read data from historians, meter data management systems and substation gateways are most likely to need to run. Placing the scheduler or the metadata database here is a larger decision than placing workers, and it should not be made by default. Airflow should not be placed inside an Electronic Security Perimeter without a documented decision, and any placement must be checked against the operator's own CIP scoping, including whether the asset is a BES Cyber System, an Electronic Access Point or a supporting asset.
- Scheduler and metadata database: Keep them outside the perimeter, in the corporate or a separate support zone, unless a documented decision says otherwise.
- Workers: Run only the workers that need local access, with outbound-only connections to the scheduler where your network rules allow it.
- Data pulls: Read from historians or staging databases through a one-way or tightly controlled path, not from live control systems.
- Crossings: Route every flow through an approved path, such as a data diode, a jump host or a defined interface, and record each crossing in your network inventory.
- Patching: Plan Airflow and provider upgrades around outage windows that operational teams accept.
Verify with your compliance and operations leads that each worker and crossing has been assessed against the relevant CIP and ISA/IEC 62443 requirements before any DAG runs in this zone.
Remote sites such as substations, generation plants and field depots often have limited connectivity and little local staff. Airflow's EdgeExecutor is designed for workers that run away from the central scheduler, so a remote site can execute tasks locally while the control plane stays in a controlled zone. Whether this suits a site depends on its connectivity, its physical security and whether the site's assets fall within an Electronic Security Perimeter.
- Control plane: Keep the scheduler and metadata database in the corporate or support zone, not at the remote site.
- Edge workers: Use edge workers only where the site's network policy allows outbound connections to the central API over encrypted channels, and confirm the executor's behavior during link loss before relying on it.
- Local data: Buffer telemetry at the site, then transfer it through the approved crossing rather than exposing site systems directly.
- Credentials: Give each site worker its own identity and limit it to the datasets that site needs.
- Remote access: Any interactive access to site workers must follow the operator's interactive remote access rules, including the required intermediate system and multi-factor authentication.
Confirm with your architecture team which remote assets are BES Cyber Systems or Protected Cyber Assets before deciding which sites may host workers.
Every flow between zones should be listed, justified and approved before any DAG uses it. The crossing is where most design risk accumulates, because a single convenience connection can bring a corporate dependency into the operational zone or expose control data to a broader network. Writing each crossing down as a named flow, with its source, destination, direction, protocol and owner, gives auditors and engineers the same view of the boundary.
- Direction: Prefer flows that start from the less trusted side toward a staging point, rather than inbound connections into the operational zone.
- Staging: Use a controlled staging store on the approved side of the boundary, with retention set for its purpose.
- Change control: Treat a new crossing or a changed DAG that reads from a new source as a change requiring review.
- Logging: Log transfers at both ends and keep the records with your other operational evidence.
- Supply chain: Include Airflow, its providers and the Helm chart or images in your supplier risk process, since CIP-013 expects coverage of software integrity.
Check each crossing with your CIP and NIS2 advisers before go-live, because the required controls depend on the impact rating of the systems involved.
Airflow settings and integrations that matter for utilities
Utility deployments of Airflow usually fail on a few foundational choices made early: the metadata database, scheduler resilience, where tasks run, how credentials are held, who can sign in, and how DAG code reaches production. Each card below is a decision or control that a grid operator or utility should settle before the first meter or SCADA-derived pipeline goes live.
Metadata database choice
Airflow keeps scheduling state, task history and DAG run records in its metadata database. Use PostgreSQL or MySQL for any environment that holds real operational data. The production guide warns that the default SQLite backend is for testing only and can lose data in production. MariaDB is not supported as the metadata database, so confirm this before an existing MariaDB estate is reused. Plan backups and restore tests for this database as part of the utility's recovery expectations.
Scheduler high availability
Airflow can run more than one scheduler at the same time, coordinating through database row-level locks. For regulatory reporting and meter data loads, this removes a single scheduler as a point of failure, but it does not replace a tested database failover plan. Ask the team to demonstrate what happens to in-flight runs when one scheduler stops, and confirm how the DAG processor, which must run as a separate component in Airflow 3, is also made resilient.
Executor choice for remote sites
Airflow ships several executors, and since Airflow 2.10.0 more than one can run in a single deployment. EdgeExecutor suits substations and generation sites where workers sit behind constrained networks and reach the central platform over HTTPS. KubernetesExecutor suits central clusters running compute-heavy reporting jobs. The hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported in Airflow 3, so do not design around them. Confirm the network path, worker isolation and task log handling for each site before choosing.
Secrets backends for utility credentials
Meter head-end systems, historian connections, market operator endpoints and cloud storage keys should never sit in DAG code or plain connection records. Choose a secrets backend that your utility already audits, such as an enterprise vault, and confirm that worker processes can read only the secrets their DAGs need. Rotation procedures should be tested with a real scheduled pipeline, not only documented. Ask the implementer which backend integrations are in scope and which are custom work.
Auth manager and role mapping
Airflow 3 uses the Simple auth manager by default, and only one auth manager can be active at a time, set by the auth_manager option. Simple suits a lab or pilot, not an operator with audited access. The FAB auth manager comes from the FAB provider, and Keycloak is another route for federated sign-in. Whichever you choose, map identity provider groups to Airflow roles for operations, engineering and read-only auditors, and keep the role model reviewable by compliance staff.
DAG delivery from a controlled repository
Pipelines that feed regulatory reports or operational dashboards should reach production only through reviewed changes. Keep DAG code in a Git repository with protected branches, required reviews and signed commits or an equivalent integrity control, then sync to the scheduler and workers through a defined process. This supports the change management and software integrity expectations that NERC CIP-013 places on vendor-supplied software. Confirm who approves a change and how a rollback is performed.
How a utility rollout is phased
The sequence moves from scope and boundary decisions to a tested, handed-over Airflow deployment, with each phase closing on evidence the utility can review before the next one starts. Regulated scheduling stays under the utility's control at every step.
Scope impact levels and boundaries
Compliance, OT engineering and the Airflow team agree which pipelines touch BES Cyber Systems, which sit outside the CIP scope, and where the Electronic Security Perimeter and any Intermediate System would sit. The utility provides its asset inventory, impact classifications under CIP-002, network diagrams and the list of regulatory reports to be scheduled. The step is done when the boundary decisions and the in-scope pipeline list are signed off in writing.
Design the environment and network paths
The team designs the hosting environment, the network flows between the Airflow control plane, the metadata database and any edge workers, and the firewall and proxy rules that permit only the required connections. The utility provides hosting capacity, identity sources, change-management requirements and approval contacts for firewall changes. The step closes with an approved architecture document and a rule set that the network team has reviewed.
Install Airflow 3 with a fixed version
The team installs a pinned Airflow 3 release with PostgreSQL as the metadata database, runs the API server and DAG processor as separate components, and records the exact versions of Python, PostgreSQL and the container or package sources used. The utility provides a PostgreSQL service that meets its backup and high availability standards, and a secrets location agreed with security. The step is done when a clean install passes a health check and the version baseline is stored in the change record.
Pilot one non-critical regulatory report
The team builds one regulatory report pipeline that is important but not operationally critical, then runs it in parallel with the existing process so outputs can be compared. The utility provides the source extracts, the report specification, the business owner who validates results and the reconciliation criteria. The step closes when parallel runs produce matching outputs over the agreed comparison window and the owner accepts the results.
Onboard edge workers at remote sites
The team brings remote-site workers online, configures their connection to the central scheduler over the approved path, and tests task execution, log retrieval and failure behavior when a site link drops. The utility provides site lists, connectivity details, local operator contacts and the data each site may send outward. The step is done when each site runs a test task end to end and its outage behavior matches the design.
Test scheduler failover
The team runs more than one scheduler against the shared metadata database, then stops the active instance during scheduled work and confirms that another instance resumes scheduling without duplicate runs. The utility provides an outage window agreed with operations, the test scenarios and a rollback plan. The step closes with recorded test results, including any runs that were missed or delayed, and a decision on whether the design is accepted.
Cut over from existing schedulers
The team moves each in-scope job to Airflow in an order agreed with operations, disables the matching job in the old scheduler only after the new run has been verified, and keeps the old scheduler available for rollback during the transition. The utility provides the job inventory, dependency maps and sign-off for each cutover. The step is done when every in-scope job runs from Airflow and the old schedules are retired under change control.
Hand over runbooks and evidence
The team delivers runbooks for upgrades, restores, scheduler failover, edge site recovery and access reviews, along with configuration records and test evidence. Operations staff run at least one supervised recovery exercise before the handover is accepted. The utility provides named owners for each runbook and the internal audit or compliance contacts who will receive the evidence. The step closes when the owners confirm they can operate the platform without the delivery team.
Keeping regulatory reports on time when a scheduler or site fails
A regulatory report with a fixed filing deadline is only as dependable as the scheduler that releases its tasks, so the design question is what happens when a scheduler, worker or site is unavailable on deadline day. Airflow supports running more than one scheduler at the same time, coordinating through database row-level locks, so a standby scheduler can take over scheduling work without a manual rebuild. Confirm that your chosen metadata database and deployment pattern run multiple schedulers as designed, and test a scheduler kill during a live reporting window rather than assuming failover works. Set retries and alerts per critical task, route failure notifications to a team that is on duty for the filing date, and define in advance who can clear and rerun a failed report task.
Backfill needs its own control. An unplanned backfill can flood the workers, compete with the live report run and write duplicate or partial outputs to a regulated destination. Restrict who may trigger backfills for report DAGs, set concurrency limits so catch-up runs cannot starve the current run, and document the rerun procedure with the reporting owner. The metadata database is the record of what ran, so back it up on a schedule, restore it into a separate environment on a regular cadence, and time those restores against your recovery objectives. PostgreSQL or MySQL is the supported production choice; the default SQLite backend is for testing only and can lose data. Document the failover runbook, including site-level failover for remote edge workers, because an examiner will ask how a missed filing was detected and recovered.
Lifecycle planning belongs in change control. The Airflow 2 line reached end of life on April 22, 2026, and end-of-life versions receive no fixes, so new builds should start on Airflow 3, currently at patch 3.3.2. Existing Airflow 2 deployments should move to the latest 2.x first and check DAG code with the Ruff AIR301 and AIR302 rules before upgrading, since Airflow 3 changes the webserver, DAG processor and task access to the database. Record the target version, the patch cadence, the upgrade window and the rollback plan in your change register, and confirm with your utility's compliance team how that register maps to the controls your regulator expects.
Compliance evidence a utility should expect to produce
Auditors and regulators do not certify an Airflow deployment, so the evidence has to be produced by your own processes and kept in a form reviewers can follow. The cards below cover the records a utility should expect to show for its scheduled pipelines, and DeployOpen prepares systems and evidence for that review without issuing audit reports or certifications.
Change management for DAG code
Every DAG that touches meter, SCADA-derived or regulatory report data should move through version control, peer review and a staged promotion path with a recorded approver. Reviewers will ask who changed a pipeline, what was tested and when it reached production. Check that the pipeline repository, its CI results and the deployment record can be matched to one another for any given run. Be aware that the NERC CIP standards do not issue a certificate, so this evidence is what your auditor will examine.
Access control and role separation
Map who can author DAGs, trigger runs, edit connections and read variables, and keep those roles separate. In Airflow 3 the default auth manager is Simple, and only one auth manager can be active at a time, set by the auth_manager option, so confirm which one you run and how its roles are exported for review. Access reviews should be recorded on a fixed schedule, and any shared accounts should be removed before the first review.
Logging and audit trails
Collect scheduler, API server, worker and DAG processor logs in a central store that operators cannot silently edit. Auditors will want to see task history, manual trigger events and configuration changes tied to named users. Confirm retention periods with your compliance function, since the Airflow documentation does not set a retention period for your records, and test that logs from remote sites with edge workers reach the central store.
Supply chain records for Airflow and providers
CIP-013 expects a supply chain risk management plan that covers software integrity and vendor vulnerability disclosure. For Airflow, keep a bill of materials for the core package, every provider package, the Python version and the container image, along with the checksums or signatures you verified at install. Airflow is licensed under the Apache License 2.0 with no paid edition from the project, so your records should name the upstream project and your own patch and approval process for each version you run.
Recovery testing
Keep a dated record of restoring the metadata database and DAG storage into a clean environment, and of resuming scheduling afterward. The production guide warns that SQLite is for testing only and can lead to data loss, so the restore test must use PostgreSQL or MySQL. Record the recovery time you observed, the data gaps found and the actions taken, since these are the records a reviewer will compare against your stated objectives.
Network boundary documentation
Document which Airflow components sit inside the electronic security perimeter, which cross it, and how each crossing is authenticated and encrypted. Keep diagrams that show the edge workers, the API server and the database, and list the firewall rules that permit traffic between them. Because CIP-005-7 governs routable connections and interactive remote access, the boundary record should show that remote access to Airflow passes through the approved intermediate path.
Questions utilities ask before hiring an Airflow partner
What drives the cost of an Airflow deployment for a utility?
Scope is the main driver: how many network zones the design must separate, how many sites and edge workers are involved, how many source systems and destinations the pipelines touch, and how much compliance evidence the work must produce. Each of these adds design, build, testing and documentation effort. Cost splits into a one-time build (architecture, installation, integrations, hardening, runbooks) and recurring costs (infrastructure you run, staff time, upgrades and any support retainer). Airflow itself is licensed under the Apache License 2.0 with no paid edition from the project, so the spend is in people, infrastructure and support. Ask any partner for a fixed-scope quote that lists each zone, site, integration and evidence item as a separate line.
Can our own team build and run Airflow without a partner?
Often yes, if you already run Linux, Kubernetes or database platforms and have engineers who know Python and your OT data sources. The harder parts for utilities are usually boundary design, where pipelines cross from corporate to operational networks, and producing evidence that auditors can follow. Check whether your team has time for a full build, a test environment that mirrors production, and an upgrade path, since Airflow 2 reached end of life on April 22, 2026 and Airflow 3 requires moving the webserver to the API server and running the DAG processor separately. A partner is most useful for a fixed design review, a hardened first deployment and a handover that your staff can operate.
What do we need to provide during the engagement?
Plan for three groups of people. Network owners need to describe current zones, firewalls, routing and any existing data diodes or jump hosts, and to approve the flows that cross each boundary. Compliance staff need to confirm which NERC CIP standards and which NIS2 national rules apply to your entity, and to name the evidence they must retain. OT engineers need to explain the data sources, protocols, polling rates and what must never be written back to control systems. Your team should also provide test access to a non-production environment, a named owner for secrets and certificates, and someone who can approve change windows. Confirm these responsibilities in the written scope before work starts.
How long does an Airflow project take, and what sets the duration?
No duration figure is given here because it depends on your environment. The main factors are the number of network zones and approved flows, how many sites use edge workers, the number of integrations, whether a formal change process must be followed, and how quickly approvals come from network and compliance teams. Projects with a single control-network boundary and a small set of data sources move faster than programs that cover several sites and regulated reports. Ask for a timeline that separates design review, non-production build, testing, cutover and handover, and identifies which milestones depend on your staff rather than the partner.
Does NERC CIP apply to Airflow itself?
NERC CIP applies to the BES Cyber Systems and the assets that your entity has classified, not to a software product as such. Whether Airflow falls inside scope depends on how you categorize the systems it touches under CIP-002-5.1a. If a pipeline can reach high or medium impact BES Cyber Systems, the design must respect the Electronic Security Perimeter rules in CIP-005-7 and the supply chain obligations in CIP-013-2, which cover vendor notification, software integrity and remote access. Many utilities keep Airflow on the corporate side and move only approved, one-way data through the boundary. Confirm the classification with your compliance lead and your NERC Regional Entity before designing the architecture.
What does ongoing support cover after go-live?
Ongoing support is a separate agreement from the build. It typically covers monitoring the platform, applying security patches within supported Airflow versions, planning and testing upgrades, restoring from backups, and updating integrations when source systems change. Airflow's support policy gives limited-maintenance versions security and critical bug fixes only, and end-of-life versions no fixes, so a retainer should name the Airflow line you run and the upgrade cadence. DeployOpen offers ongoing support, upgrades and continuous testing as retainers. Ask what is included in response times, which environments are covered, and how change records and evidence are delivered to your compliance team.
How DeployOpen scopes and delivers Airflow for energy operators
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own infrastructure, including Apache Airflow for energy and utilities. Engagements are fixed-scope, with the scope, deliverables and timelines agreed in writing before work starts.
How we scope
- Written scope lists the pipelines, data sources, OT boundary zones and regulatory reports in the first deployment.
- Scope records which networks, sites and edge workers are in or out of the Airflow environment.
- Scope names the compliance inputs we prepare, such as evidence and control mappings, which auditors then assess.
How we build
- We deploy Airflow 3 components, including the API server and a separate DAG processor, on infrastructure you control.
- We configure the metadata database, executors, DAG delivery, secrets and single sign-on as agreed in the scope.
- We test DAG code, failover of scheduler replicas and restricted-network paths before any production cutover.
How we hand over
- Handover includes architecture, network flow and access documentation written for your operations and compliance teams.
- Runbooks cover backups, restores, upgrades, scheduler failover and the steps for responding to failed regulatory jobs.
- Evidence packs record configuration, change history and test results so your team can give them to an assessor.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Retainer scope should state which Airflow patch lines are covered and how security fixes are applied in your environment.
- Ask for the support hours, response channels and change approval process in the written quote before signing.
