Airflow and Spark integration: what the operators do and where they run
A team that already schedules pipelines in Airflow and needs Spark jobs in the same dependency graph can keep one scheduler as the system of record for ordering, retries and alerts. The integration runs through the Spark provider operators named for this topic: SparkSubmitOperator, SparkSqlOperator, SparkJDBCOperator and PySparkOperator. In this model Airflow orchestrates and monitors while Spark does the compute. Airflow decides when a job starts, passes it parameters, watches its state and marks the task as succeeded or failed. The data processing itself runs on the Spark cluster, so Airflow workers should stay small and should not hold large datasets in memory. The topic covers Spark Connect as the connection path, which means the operators reach a Spark endpoint through a connection definition rather than by launching a local driver on the worker.
Before any DAG is written, the team must confirm the details that define how these operators behave in their environment. Check the provider package version, the exact operator names and their import paths, and whether the operators support Spark Connect, against the Apache Airflow providers documentation for the version they plan to run. Those details are not part of the facts this page relies on, so they should not be assumed from older examples. Confirm them alongside the Airflow release line itself, since Airflow 3 changes how task code reaches the platform.
Where each operator runs is a decision with consequences for security and capacity. Ask which network path connects Airflow workers, the scheduler and the Spark endpoint, which service account or credential the connection uses, and whether Spark driver logs will be visible in Airflow task logs. Write these answers down before the first run, because they determine who can submit compute, where credentials are stored and how much load a single failed DAG can place on the cluster.
What the Airflow Spark integration needs before the first DAG run
Before any Spark operator runs, the Airflow side, the Spark side and the network between them each need to be in place. The cards below cover what to confirm on each side and where the connection between them can fail.
A supported Airflow 3 release
Start from a current Airflow 3 patch release rather than Airflow 2, which reached end of life on April 22, 2026. Airflow 3 changed how task code reaches the platform, so confirm the Spark provider version you plan to pin supports the exact Airflow release you run. Check the metadata database and executor you already operate against the Airflow prerequisites, since those choices carry over unchanged to Spark jobs.
Spark provider package on every worker
The Spark operators come from the Spark provider package, which must be installed in the image or environment that runs task code on every worker, including any Kubernetes pods that execute tasks. Installing it only on the scheduler or API server leaves tasks failing at import or at execution. Pin the provider version and confirm its compatibility with both Airflow and Spark in the provider documentation before the first run.
Spark cluster version and endpoint
Decide which endpoint the operators will use: a Spark Connect endpoint for remote sessions, or a submit endpoint for spark-submit style jobs. Record the Spark version the cluster runs, because Spark Connect and submit behavior differ between releases and must match the provider's expectations. Ask the Spark platform owner for the host, port and any authentication the endpoint enforces.
Java and Spark client on workers
Workers that submit Spark work need the client libraries and a Java runtime compatible with the Spark version in use. A worker without them can reach the cluster and still fail every job. Confirm the required Java version against the Spark release notes for your cluster, and bake the client into the worker image so every task runs with the same versions.
Network paths from workers
Every worker that runs a Spark task must reach the Spark endpoint over the port it uses, and the Spark driver and executors must be able to reach back where the design requires it. In segmented or air-gapped networks, request these firewall rules early because they are often the slowest item to approve. Test the path from an actual worker, not from a laptop.
Certificates for TLS endpoints
If the Spark endpoint uses TLS, the workers must trust the certificate chain, including any internal certificate authority. Plan renewal alongside the Airflow connection, because an expired certificate stops every Spark task at once. Store the trust material where your team manages other certificates, and confirm the connection settings accept the chain in use.
Choosing between SparkSubmitOperator, SparkSqlOperator, SparkJDBCOperator and PySparkOperator
Use SparkSubmitOperator when the unit of work is a compiled or packaged Spark application, such as a JAR or a Python artifact that a data engineering team builds and versions outside the DAG. The DAG passes the artifact location, entry point, arguments and Spark configuration, and Spark runs the application. Airflow mainly controls scheduling, retries and dependencies, while the application logic stays with the team that releases it.
- Typical job: a batch application that a team releases on its own cycle and that Airflow only triggers.
- Connection: a Spark connection in Airflow that points at the cluster or Spark Connect endpoint, plus any credentials the submit path needs. Confirm which connection fields the provider reads for your deployment mode.
- Task output: the run result and logs describe the application execution. Confirm what the operator pushes to XCom in your provider version before passing values downstream.
- Main question: does the application already exist as a versioned artifact with a stable entry point? If not, the build pipeline needs to exist before the DAG does.
Confirm that the artifact location, dependency packaging and cluster access are owned by a team that can change them without editing DAG code.
Use SparkSqlOperator when the transformation is expressed as SQL statements that run on Spark against tables the cluster can already see. The DAG holds the SQL text or a reference to it, and Spark executes the statements in order. This fits teams whose analysts review and maintain the logic as SQL, with Airflow handling the schedule and dependencies around it.
- Typical job: an aggregation or table build that reads and writes catalog tables on the cluster.
- Connection: a Spark connection that defines the cluster endpoint. Confirm which catalog and metastore settings the statements inherit from that connection and from the cluster configuration.
- Task output: confirm what the operator returns or pushes to XCom. Large result sets should stay in tables, not in task output.
- Dependencies: statements that write tables should run in an order the DAG controls, so check how a failed middle statement affects the tables already written.
- Main question: is the logic naturally set-based SQL that reviewers can read, or does it need procedural steps that belong in a script?
Check which team reviews SQL changes and whether that review happens in the same repository and change process as the DAG.
Use SparkJDBCOperator when the main work is moving data between a JDBC source or target and Spark. The operator reads a table or query from a relational database into Spark, or writes Spark output back to one. The database side of the job, including its load, locking and credentials, becomes part of the design along with the Spark side.
- Typical job: extracting an operational table into Spark for downstream processing, or writing curated results back to a reporting database.
- Connection: a JDBC connection for the database and a Spark connection for the cluster. Confirm where the JDBC driver must be available to Spark and who installs it.
- Transfer modes: the provider documents the transfer modes it supports. Pick the mode that matches your load pattern, such as appending, overwriting or reading a full table, and confirm how each handles a partial failure.
- Task output: confirm what status or counts the operator reports to Airflow in your provider version.
- Main question: how much data moves in each run, and can the source database tolerate the read or write load during the window you plan to use?
Confirm who owns database credential rotation and whether the database team has approved the query pattern before the first production run.
Use PySparkOperator when the Spark work is a PySpark script or job that data engineers write and iterate on alongside the DAG. The script usually lives in the same repository or release process as the pipeline, so changes to transformation logic and orchestration can be reviewed together. This is the closest option to writing Spark code directly in the pipeline's own codebase.
- Typical job: a Python transformation with conditional logic, joins and validation steps that are easier to express in code than in SQL.
- Connection: a Spark connection that defines where the script runs. Confirm how Python dependencies reach the Spark environment, since this is a common source of failed runs.
- Task output: confirm what the operator returns and what reaches XCom in your provider version, and keep large data out of XCom.
- Versioning: decide whether the script is pinned to a DAG release or deployed on its own schedule, and make sure the operator references that version explicitly.
- Main question: does the team want the script versioned with the DAG code, or does it need its own release cycle shared with other pipelines?
Check that the dependency set, the script version and the Spark environment are recorded together so a failed run can be reproduced.
Configuring the Spark connection and running the first Airflow Spark task
This sequence takes a self-hosted Airflow 3 environment from a provider package installed on every worker to a verified Spark submission, then adds the JDBC connection that transfer operators need. Each step ends with a check so the team knows whether to move on.
Confirm the Airflow and Spark versions in scope
The platform team records the Airflow 3 patch release it runs, the Spark version and build the cluster exposes, and the Python version on the workers. Airflow 3.3.2 lists Python 3.10 to 3.14 as supported, so the worker interpreter must fall inside that range. Confirm the Spark provider package version against the Airflow version in use before building images. The check is a written compatibility note that the Airflow and Spark owners both sign off on before any image is changed.
Install the Spark provider on every worker image
The platform team adds the Apache Spark provider package to the worker image build, pinned to the version confirmed in the previous step, and installs it with the same constraints file used for the rest of the Airflow environment. Every executor type that runs Spark tasks needs the package, including scheduler-side parsing of DAGs that import the operators. The check is a shell inside a running worker container that imports the Spark operator module without error, plus the same check on the scheduler image.
Define the Spark connection outside DAG code
The Airflow administrator creates a Spark connection that holds the master or Spark Connect endpoint and any authentication parameters the cluster requires. Store the connection in the secrets backend your organization chose, so that passwords and tokens never appear in DAG files, environment variables baked into images, or the metadata database. DAG code should reference only the connection ID. The check is that the connection resolves from the secrets backend when a worker reads it, and that the value does not appear in the connection list shown in the UI.
Test network reachability from a worker
The network team runs a connectivity test from inside a worker pod or VM, not from a laptop, to the Spark endpoint on the port the connection specifies. Firewall rules, DNS, and proxy settings often differ between the scheduler, workers, and the Spark cluster, so this test isolates network problems before any DAG runs. Where the connection uses TLS, confirm that the worker trusts the certificate chain. The check is a successful TCP and TLS handshake from the worker, recorded in the change ticket.
Run a minimal DAG with one Spark operator
The data engineering team deploys a small DAG that uses one Spark operator, such as SparkSubmitOperator, to submit a trivial application like a count over a generated range. The DAG reads its connection ID from configuration and keeps the application code in a version-controlled location your teams can review. Trigger the run manually and keep the default retry behavior so that transient failures are visible in logs. The check is a successful task run in the Airflow UI and a matching application entry in the Spark UI with the expected name and state.
Verify the Spark UI and task logs match
The operations team compares the Airflow task log with the Spark application record to confirm that the application ID, submit arguments, and final state line up. Check that driver and executor logs are reachable through the locations your log policy allows, and that no credential appears in the captured output. If the Spark UI shows no job, the submission most likely reached a different endpoint than the one tested in the network step, so return to the connection definition. The check is a documented run where Airflow and Spark report the same outcome.
Add the JDBC connection for transfer tasks
The data engineering team defines a separate JDBC connection for the source or target database that SparkJDBCOperator will read or write, again stored in the secrets backend. Confirm which transfer mode the pipeline needs, and check the JDBC driver jar is present on the Spark side and referenced by the operator configuration. Confirm the database host and port are reachable from the Spark executors, which can sit on different network segments from the workers. The check is a test read of a small table through the JDBC connection, run as a Spark application and confirmed in the Spark UI.
Record the working baseline before scaling out
The platform team saves the image digest, provider version, connection IDs, secrets backend paths, and the verified DAG and Spark application as the baseline for this environment. Promote this baseline through staging before adding more DAGs, so that later failures can be traced to a known state. Ask your implementation partner or internal team to document rollback steps for the images and connection changes. The check is a baseline record that another engineer can use to reproduce the first successful run.
How the JDBC transfer modes move data between databases and Spark
SparkJDBCOperator moves data through a JDBC connection, and the transfer mode decides how rows are read or written and how existing target data is handled. Before designing the DAG, decide three things for each transfer: the direction (database to Spark, or Spark to database), the expected size of the data relative to the memory available on the Airflow worker and the Spark cluster, and how the target table should be treated when it already contains rows. Those answers determine which mode fits. Confirm the exact mode names, the accepted parameter values and what each mode does to an existing table in the current provider documentation for Airflow and the Spark provider before you write any DAG code. Do not rely on mode names from older examples or blog posts, because the behavior of a mode is what matters for data safety, and names alone do not describe it.
Size is the decision most teams underestimate. A JDBC read can put a large share of a table into Spark memory if partitioning is not configured, and a write pushes every row through the database driver, so throughput depends on the database, the network path and the number of parallel connections. Ask the database owner for the row count, the table width and any load windows, and ask the Spark platform team how many executors and how much memory a job can take without starving other workloads. Keep the Airflow worker out of the data path as much as possible. The worker should submit and monitor the job, not hold the data, so a transfer that needs worker memory is a sign the design should move to the Spark cluster.
Retries need a planned answer before the first run. A failed transfer that is retried in an append-style mode can insert the same rows twice, while an overwrite-style mode can remove data that a partial run already replaced. Decide whether the target is written from a staging table that is swapped in only after a successful load, whether the job writes a run identifier or watermark that makes a rerun idempotent, and whether the database will tolerate a partial load being visible to readers. Test the failure path in a non-production database by killing a transfer midway and checking row counts, then record the result as part of the DAG's acceptance evidence.
Mapping Airflow connection fields to Spark job settings
This table maps each Airflow Spark connection field to the operator parameter or Spark setting it controls and names the team that should own the value. Read it when deciding which settings belong in the connection, which belong in DAG code, and which need security review before go-live. Field names vary by provider version, so confirm them against the installed Spark provider before writing any connection.
| Connection field or setting | Controls | Value owner |
|---|---|---|
| Connection ID and type | The conn_id a DAG passes to each operator, and the Spark connection type the operator expects | Data team, agreed with platform team |
| Host or endpoint | The Spark master URL or Spark Connect endpoint the job is submitted to, set in the connection host field | Platform team |
| Port | The port on the Spark master or Connect endpoint, kept in the connection rather than in DAG code | Platform team |
| Extra: deploy mode | Whether the driver runs in client or cluster mode, passed through to spark-submit when using SparkSubmitOperator | Platform team, with data team input on driver placement |
| Extra: master or queue | The cluster manager target and resource queue the job is scheduled into, which drives where compute is charged | Platform team |
| Extra: Spark configuration keys | Executor, memory and dependency settings applied to each submission, which drive job cost and stability | Data team for job-level tuning, platform team for defaults |
| Login and password | Credentials the connection uses to reach the Spark endpoint or a JDBC source, stored in a secrets backend | Security team sets policy, platform team issues credentials |
| JDBC URL and transfer settings | Source or target database address and the transfer mode used by SparkJDBCOperator | Data team, reviewed by security for network paths |
| Principal or keytab references | Kerberos or service identity used to authenticate the submission in secured clusters, if your Spark setup requires it | Security team |
Securing Spark credentials, network paths and job code from Airflow
Airflow holds the keys to every Spark submission and JDBC transfer it triggers, so the credentials, network routes, job artifacts and logs need to be designed together. The six decisions below are the ones your team configures itself, and each one narrows what a compromised DAG or worker could reach.
Keep Spark and JDBC credentials out of DAG code
Store Spark Connect endpoints, Spark master credentials and JDBC usernames and passwords as Airflow connections backed by a secrets backend, not as literals in DAG files or plain Airflow Variables. DAG files are readable by everyone who can change the repository, and Variables live in the metadata database in clear text unless a backend is configured. Check that DAG authors can reference a connection ID but cannot read the secret behind it, and that the connection UI and logs mask the values.
Use least-privilege database accounts for JDBC
A SparkJDBCOperator transfer should run under a database account that can only read the source tables or write to the named target tables. Create a separate account per pipeline, grant only the required schemas and privileges, and avoid reusing the account that administers the database. Ask your DBA to confirm which operations the transfer needs in each transfer mode so that a broken or misdirected DAG cannot drop or alter unrelated tables.
Restrict which network paths workers can use
Airflow workers need a route to the Spark master, Spark Connect endpoint or submit gateway and to the JDBC host, and nothing else in the Spark network should be reachable from them. Define firewall or security group rules that allow only those destinations and ports from the worker subnets. Confirm the same rules hold for the triggerer and any Kubernetes pods that submit Spark work, since they may run in different network segments.
Encrypt Spark Connect and submit traffic with TLS
Credentials and job definitions travel from Airflow to Spark, so the Spark Connect or submit endpoint should require TLS with certificates issued by your internal certificate authority. Decide how the Airflow worker trusts that authority, where the CA bundle is stored, and who rotates certificates before they expire. Verify the setting in the Spark documentation for your version, because the exact configuration keys for TLS vary by deployment mode and must be confirmed in your environment.
Control which JARs and scripts can run
SparkSubmitOperator and PySparkOperator execute whatever application JAR or Python file the DAG points to, so the artifact path is an execution boundary. Limit submissions to artifacts stored in an approved location, pin versions by checksum or immutable tag, and make changes to that location go through code review. Decide who may publish new artifacts and whether DAG authors can override the application path at run time.
Log every submission for audit
Record which DAG, task, run and user triggered each Spark application, the artifact and arguments used, the target cluster and the outcome. Airflow task logs and the Spark driver logs should be retained and shipped to a store that your security team controls, with the same retention they apply to other audit evidence. Test the trail by tracing one failed run from the Airflow task to the Spark application ID to confirm that the chain is complete.
Common Spark submission and transfer failures and how to fix them
This table lists the failures teams most often hit when Airflow submits Spark work or runs JDBC transfers, with the likely cause, how to confirm it in the Airflow task log or on the Spark side, and the fix. Read each row against the task that failed, starting with the connection and worker checks, since most failures trace back to one of them.
| Failure | Likely cause and how to confirm | Fix |
|---|---|---|
| Connection not found or wrong connection type | The DAG references a connection ID that does not exist in the metadata database or secrets backend, or points at a connection of the wrong type. The task log shows an AirflowNotFoundException for the connection ID, or the Spark hook rejects the connection type. | Confirm the connection ID in the Airflow UI or CLI and check that its type matches the operator (Spark for submit and PySpark, JDBC for transfers). Correct the ID in the DAG or create the connection in the environment the scheduler and workers read. |
| Workers missing the Spark client or Java runtime | spark-submit is not on the worker PATH, or no Java runtime is installed, so the submit command fails before reaching the cluster. The task log shows a command-not-found error or a Java startup error from spark-submit, while the same command works on a manual login to a worker. | Install a matching Spark client and Java runtime in the worker image or host, and set the Spark home and Java home consistently across all worker types. Rebuild the image and roll it through every worker pool, not only one. |
| Network blocked between worker and cluster | Firewall rules, security groups or a missing route stop the worker from reaching the Spark master, Spark Connect endpoint, or driver callback ports. The task log shows connection timeouts or refused connections, and the Spark side shows no inbound registration from the worker. | Ask the network team to open the required ports in both directions for the chosen deploy mode, then test from a worker pod or host with a simple TCP check before rerunning the DAG. |
| Spark endpoint or certificate mismatch | The connection host, port or scheme does not match the endpoint the cluster presents, or the TLS certificate is issued for a different name or has expired. The task log shows hostname verification or SSL handshake errors, and the Spark or gateway logs show rejected handshakes. | Compare the connection host and port with the endpoint configuration on the cluster, then update the certificate chain or trust store on the workers. Confirm the name in the certificate matches the name in the connection. |
| JDBC driver missing on the worker | The JDBC transfer operator loads a driver class that is not on the worker classpath or is the wrong version for the database. The task log shows a ClassNotFoundException or a driver version error before any rows move. | Place the approved driver JAR on every worker that can run the task, and pin its version in the image build. Check the driver release notes against the database version you run. |
| Credentials rotated but not updated in the secrets backend | A password or key was rotated in the source system, but the connection stored in the secrets backend or metadata database still holds the old value. The task log shows authentication failures, and the database or Spark audit log shows rejected logins from the Airflow service account. | Update the secret at its path in the secrets backend, confirm the connection reads the new version, and rerun a single task before resuming the schedule. Add rotation to the change process so the secret and the consuming systems move together. |
| Retries that duplicate a JDBC load | A transfer task fails after rows are committed or after a partial write, and the retry inserts the same batch again. The task log shows a retry following a timeout or worker loss, and the target table shows duplicate keys or doubled row counts. | Make the load idempotent with a staging table, a delete-then-insert on a known batch key, or a merge on a unique key, and set retries only where the write is safe. Record the batch identifier in the task output so the team can confirm which load ran. |
What an Airflow and Spark integration costs and what it asks of your team
What drives the cost of an Airflow and Spark integration?
The cost follows the size and shape of the work: how many operators and DAGs call Spark, how many JDBC sources and transfer modes are involved, how much security review and network approval is needed, and how deep the testing goes. Each DAG that submits Spark work needs its own connection design, parameters, retry behavior and failure handling, so a small set of reusable patterns usually costs less than many one-off DAGs. A quote should itemize these components separately, state what is one-time (design, build, first-time connection and network setup, initial testing) and what recurs (upgrades, Spark and Airflow version changes, support). Ask the provider who is paid for each part, including any work done by your own platform or security staff.
How long does an Airflow and Spark integration take, and what sets the duration?
The duration depends on the number of operators in use, the number of Spark submission paths, the readiness of the Spark cluster and its network access, and how long approvals take inside your organization. Work that touches a restricted network, a new JDBC source or a new credential store usually takes longer than a set of submissions to a cluster that is already reachable. No duration is fixed here, so ask for a written timeline tied to a scope that lists the DAGs, operators, sources and environments involved, and ask which dependencies sit with your team, such as firewall changes, service accounts and test data.
What does the integration need from our platform and data teams?
The work needs access and decisions from several teams before engineering starts. Platform engineering should confirm the Airflow deployment model, the executor in use and the network path from Airflow workers to the Spark endpoint. Data teams should name the Spark jobs, SQL statements and JDBC tables that will run, along with their owners and expected schedules. Security should review how Spark and database credentials are stored and who can read them, and whether the Spark endpoint is reachable only from approved networks. Ask the provider to list these inputs in the scope so each one has an owner and a date.
Can an internal team build this integration without outside help?
Yes, an internal team with Airflow and Spark experience can build it, and many teams do. The questions to answer honestly are whether the team has run Airflow in production with its own connections and secrets handling, whether it has operated the Spark cluster it will submit to, and whether it has time to test JDBC transfer modes and failure paths. Outside help usually adds the most value on the design decisions, such as choosing between submission paths, and on verification before a cutover. Compare an internal estimate and an external quote against the same written scope.
How should we decide between Spark Connect and other submission paths?
The choice depends on how your Spark environment is exposed, which Spark version and deployment mode you run, what your jobs need from the driver and which operators your team can support. Spark Connect fits cases where a client should talk to a Spark server over a defined endpoint, while other paths suit jobs that depend on specific cluster submission behavior. Confirm the exact Spark Connect support in your Spark version and the operators you plan to use with the Airflow provider documentation before deciding, and record the trade-offs in the design so later teams understand why a path was chosen.
What changes when the Spark cluster or the Airflow version changes?
A change to either side can alter connection settings, operator behavior, provider package versions and the network path, so both need a planned upgrade. Airflow 3 has its own requirements, including a move from Airflow 2.7 or later and checks on DAG code, and provider packages must match the Airflow version in use. Spark upgrades can change the endpoint, the authentication method or the supported parameters. Ask for a written process that covers pinned provider versions, a staging environment, regression tests for each Spark submission and a rollback plan. Confirm these details with the Airflow provider documentation and your Spark distribution before scheduling an upgrade.
What should we ask for in a quote for this integration?
Ask for a written scope that lists each operator in use, each DAG and its schedule, each JDBC source with its transfer mode, the Spark endpoints and environments, and the security and network requirements. The quote should separate design, build, testing, handover and any recurring support, and it should name what your team must provide. Ask how changes to scope are priced and how acceptance is verified. Fixed-scope agreements are the usual format for this kind of work, so a clear scope document is the basis for comparing offers.
How DeployOpen scopes and delivers the Airflow and Spark integration
DeployOpen plans, builds and hands over Apache Airflow and Apache Spark integrations inside infrastructure you control, starting from a written scope. The work covers the Spark connection setup, the operators your pipelines use, and the operating controls around them, with each deliverable agreed before build begins.
How we scope
- Inventory which Spark jobs, SQL steps and JDBC transfers Airflow must trigger, and which stay outside it.
- Confirm the Spark version, cluster manager and Spark Connect endpoint your team operates today.
- Record the Airflow 3 target version, executor choice and where Spark connection credentials will be stored.
How we build
- Configure Spark connections and operators in a non-production Airflow environment before any production DAG changes.
- Test SparkSubmitOperator, SparkSqlOperator, SparkJDBCOperator and PySparkOperator paths against representative jobs and failure cases.
- Validate JDBC transfer modes, source and target permissions, and retry behavior on rerun before promotion.
How we hand over
- Deliver the DAG code, connection definitions and configuration documentation as versioned artifacts in your repository.
- Document the runbook for failed Spark submissions, stuck sessions and rotating connection credentials.
- Walk your platform and data engineering teams through the build, verification evidence and open items before close.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Retainer work can cover Spark or Airflow version upgrades that change operator or connection behavior.
- Ask any provider to confirm the scope, deliverables and timelines in writing before work starts.
