Apache Airflow multi-team deployments and their isolation limits

Review team assignments on a fixed schedule and after every organizational change, because multi-team mode only isolates what it is configured to isolate. Export the current mapping of users, roles and teams, then compare it with the ownership records in your identity provider and HR system. Remove access for people who changed teams or left. Confirm with your chosen auth manager how team membership is granted, since that determines whether the review can be driven from your identity system or must be done by hand in the UI. Check that DAG authors cannot also edit another team's connections or variables. Record each review with a date, reviewer and outcome so the evidence can be shown to an auditor. Use the Airflow security model as the reference for what teams still share, and review against it too. DeployOpen can plan the review procedure and evidence template as part of a hardening engagement.

Probably, but confirm the scope in your own environment before you rely on it. Airflow records actions taken through the UI, the REST API and the CLI, along with some system events, and permission to view those records is separate from ordinary DAG access. For a shared instance, check that each entry identifies the acting user and the team that owns the object changed, because a log that names a user but not a team makes review slow. Confirm retention, since database cleanup can remove old event rows. Plan to ship the events to your SIEM so they survive cleanup and sit outside the reach of the people they describe. Test with a deliberate change by a user in each team, then confirm the record appears where your reviewers will look. The Airflow security model explains the trust boundaries the log must support.

Team scoping limits who sees a Variable or Connection in the UI and REST API. It does not change the fact that teams share the Execution API, the signing keys and the metadata database, so any secret stored there is reachable by anything that holds those keys or that database access. Keep the Fernet key under the same controls as a database password, and rotate it using the documented procedure. For higher-sensitivity credentials, move them to a secrets backend so the metadata database holds no stored values for them. Give each team its own backend path or prefix, with a naming convention that prevents collisions. Verify that values are masked in logs and the UI, and remember that XCom and environment variables need separate controls. Confirm each choice against the security model and your own threat model before go-live.

Expect to upgrade more often than a stable feature would require, and treat each upgrade as a change to a shared system. Multi-team mode is experimental, so its behavior can change between releases, and the Airflow release notes are the first place to check what changed. Stay on a supported minor line: under the Airflow support policy, a version in limited maintenance receives only security and critical fixes, and an end-of-life version receives none. Stage each upgrade in a production-like copy with the full team configuration, rerun your RBAC checks, and confirm that the API server, scheduler and DAG File Processor behave as expected. Keep a rollback path that includes a metadata database backup taken immediately before the change. DeployOpen can plan and carry out upgrades under a written scope, or run them under a retainer.

Back up the metadata database on a schedule, and prove the schedule works by restoring from it. The Airflow production deployment guide recommends PostgreSQL or MySQL for this role and warns that the default SQLite backend is for testing only and can lose data in production. Use point-in-time recovery if your database platform offers it, store backups outside the cluster running Airflow, and encrypt them. Test a restore into an isolated environment and time it, because the restore duration determines how long a shared instance stays down. The database holds every team's encrypted connections and variables, so backup copies carry the same sensitivity as the live database. Document who may request a restore, since a restore can roll back team configuration. Validate recovery with all schedulers running, because they coordinate through database row-level locks.

Monitor each shared component separately, because a responsive web interface does not show that scheduling, parsing or triggering is keeping up. Teams share the API server, the scheduler, the DAG File Processor and the Triggerer. Per the security model, the DAG File Processor and Triggerer run as one instance by default, so a slow parse or a stuck trigger affects every team. Track scheduler heartbeats, DAG parse time and import errors, queued and running task counts, triggerer health, database connection use and metadata database growth. The Airflow scheduler documentation describes how multiple schedulers coordinate, which helps you set alert thresholds. Alert on stalled parsing or scheduling, and route alerts to owners by team where your tooling allows it. Confirm which metrics your Airflow version exposes and where to collect them.

Yes, within a written scope. DeployOpen plans, implements, hardens, backs up, upgrades, documents and supports self-hosted open-source platforms in your own infrastructure, so the access reviews, audit log routing, secrets handling, upgrade procedure, recovery testing and monitoring can be designed and built together. Most engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts. Recurring reviews, upgrades and monitoring support are available as retainers. Cost depends on the number of teams, the components in scope, the current state of the instance and how much evidence collection you want prepared. DeployOpen does not publish a price list, so ask for a quote based on the written scope. We prepare controls and evidence for compliance assessments, but independent auditors and certification bodies issue the reports and certifications.

Not yet, by the project's own label. The [core] multi_team setting is experimental in Airflow 3.2, and it adds team-scoped RBAC in the UI and REST API plus team-scoped Variables and Connections. It does not add task-level isolation. Teams still share the Execution API, the signing keys and, by default, one DAG File Processor and one Triggerer. Before you rely on it for regulated data, confirm the current status in the Airflow security model and the release notes for the version you plan to run, and decide whether the shared components are acceptable for your risk profile.

The main drivers are the number of teams, how far their DAGs and credentials must be separated, and how many shared components you must split out. Effort rises when each team needs its own connections, secrets backend paths, role mappings from your identity provider, and a clear rule for who can edit shared Variables. It also rises when you add a separate DAG File Processor or Triggerer per team, or when you must prove separation to an auditor. Ask any partner for a quote that lists each team, each shared component, the identity integration, and the test cases for cross-team access, so you can compare scopes rather than headline numbers.

You need a named owner for each team's DAGs, a list of the systems each team connects to, and a decision on identity. Your identity provider must map groups to Airflow roles, and the auth manager you choose must support the team-scoped model you design. You also need a secrets strategy, a repository and deployment process that keeps one team's code from reaching another team's DAG folder, and an environment for testing. Gathering these inputs early usually shortens the design phase more than any other step.

Yes, if your team already runs Airflow 3 in production and has people who can work through the security model, auth manager configuration and secrets backend setup. The harder work is the testing: proving that a user in one team cannot see or change another team's DAGs, Variables, Connections, logs or runs, through both the UI and the REST API. Plan for that test suite and for keeping it running through upgrades, since an experimental feature may change between releases. A partner can take on any part of this, from design review to testing, under a written scope.

Choose separate instances when teams need task-level isolation, when they run code that must not share workers, Execution API signing keys or a DAG File Processor, or when their data classifications differ. Separate instances also suit teams that need different Airflow versions or different auth managers. The trade-off is more infrastructure, more upgrades and more monitoring to run. If you use a managed service, note that Amazon MWAA does not support multi-team mode, so separate environments are the only option there. Check the MWAA versions page for its current limits.

A partner should start with a written inventory of teams, data sources, DAG authors, identity groups and compliance obligations, then decide whether multi-team mode fits or whether separate instances are the better design. The scope should name the deliverables: the architecture decision record, configuration for RBAC and secrets, the repository and deployment controls, an isolation test plan, documentation and handover. DeployOpen works on fixed scopes agreed in writing before work starts, and it can run ongoing upgrades and testing under a retainer. Ask for each deliverable and its acceptance test in the quote.