Running Airflow task logic in Java or Go with the Task SDKs
The language Task SDKs for Java and Go let Apache Airflow run task logic written in those languages, a capability added in Airflow 3.3.0, released July 6, 2026. A Python-centric team keeps its DAG definitions in Python, while the Java or Go code runs as the body of individual tasks. Teams usually need this when JVM or Go services already hold the business logic a pipeline must call. Rewriting that logic in Python or wrapping it in a separate job tends to duplicate tested code and split ownership. Before committing, confirm in the release notes which task types and features each SDK covers in this release.
Under the hood, the language SDKs build on the Task Execution API, the same internal interface that Python tasks use to report state, logs and results to the scheduler. That API is authenticated with JWTs. Execution tokens default to a 10-minute lifetime with automatic refresh, but they cannot be revoked and carry no per-DAG authorization. A Java or Go task therefore has the same reach as any task on the worker that runs it. Plan worker isolation, network paths and secrets access around that fact, not around the token. Confirm with the security model how token scope applies to your deployment before you place sensitive pipelines on shared workers.
Keep scheduling, dependencies, retries and asset definitions in Python DAG files, and move only the executed logic into Java or Go. Containerized tasks remain the better fit when the logic ships as an isolated image with its own dependencies, or when it needs a runtime the SDK does not support. Before the first production DAG, have the team confirm supported language versions, how logs and task state appear in the UI, and how they will unit test the code and run it in staging. Those checks decide whether the SDK reduces work or moves it into a new support path.
When a Java or Go SDK task beats a containerized task
The strongest case for an SDK task is code that already exists. If your team has tested Java services, libraries or Go utilities that transform data, call internal APIs or apply business rules, wrapping them in an SDK task keeps that logic in the language where it is maintained today. Moving it into Python or into a separate image adds a second copy that must be kept in sync.
- Library reuse: Shared JVM or Go libraries can be called from the task without porting them to Python or rebuilding them as a separate service.
- Single source of truth: The team that owns the library also owns the task, so fixes reach the pipeline through the same release process.
- Existing test path: Unit tests, static analysis and dependency scanning that already cover the code can run in the pipeline that protects it today.
- Version coupling: A task that depends on a library version must be retested when that library changes, so confirm who approves those upgrades.
Check that the SDK supports the language version and build tooling your code already uses, and confirm that your libraries run in the worker environment you plan to operate.
When a pipeline runs many short tasks, the time spent starting each one can matter as much as the work itself. A containerized task typically pays for image handling, container creation and runtime initialization before it runs any logic, and that cost repeats for every task instance. An SDK task can avoid part of that cost, but only if your worker setup and the SDK's process model allow it, so confirm the details in the release documentation before you assume a benefit.
- Container start cost: Each containerized task may pay for image pull, container creation and process startup, which repeats across every task instance.
- Runtime initialization: JVM warm-up can be significant for short tasks, so the benefit of an SDK task depends on how long the process lives and how often it is reused.
- Worker footprint: Installing the Java or Go runtime on workers adds to every worker's image and patch cycle.
- Measurement: Compare task duration, queue time and worker utilization on a representative DAG before choosing one model for the whole platform.
Ask the SDK documentation how each task process is started and reused, then benchmark a representative set of tasks in your own environment before committing.
The language your engineers can debug at 2 a.m. matters as much as the one that runs fastest. Teams that already build Java or Go services can usually troubleshoot an SDK task more quickly than a Python developer working from a container they did not build. Teams whose pipeline authors work mainly in Python may find that a Java or Go task slows down changes, reviews and incident response.
- Ownership: Name who will write, review and carry the on-call duty for each SDK task, not only who wrote the first version.
- Debugging skill: Check whether operators can read JVM or Go stack traces, logs and runtime symptoms with the tools you run.
- Review capacity: Mixed-language DAG code needs reviewers for both languages, so confirm that the change process covers each one.
- Continuity: A small number of engineers who know the SDK can become a single point of failure during staff changes.
Ask each team which languages their on-call engineers can support in production, and document that answer before you choose an SDK.
A containerized task remains the simpler choice in several cases. When a task calls a native binary, needs a specific operating system package, uses a runtime other than Java or Go, or must ship exactly as a vendor builds it, an image gives that work a self-contained environment. It also keeps those dependencies off the Airflow worker, which helps when different tasks need conflicting versions.
- Native dependencies: Tools that need system libraries, GPUs or specialized binaries fit cleanly in an image that carries them.
- Dependency isolation: Each image holds its own runtime and libraries, so one team's upgrade does not change another team's tasks.
- Vendor tools: Third-party command-line tools often ship as container images, which are easier to pin and scan than a copied binary.
- Mixed-language tasks: A task that combines Python, Java and Go pieces may be easier to package as one image with a single entry point.
Identify which tasks need isolation from the worker, then decide task by task across the platform, and record the image build, registry and scanning steps that each container choice adds.
What stays in Python DAG definitions
Moving task logic to Java or Go does not move the pipeline. The Python DAG file still decides when work runs, what it depends on and how the code reaches the scheduler, so the cards below cover what teams keep in Python and what they should verify before splitting a pipeline across languages.
DAG structure and scheduling
The DAG file defines the task graph, the schedule, retries and default arguments that the scheduler reads when it parses the file. A Java or Go task only runs when a Python DAG tells Airflow it exists and where it sits in the graph. Check that every language-specific task still has a clear owner in the Python definition, and that the schedule settings used by the scheduler stay in one place your team can review.
Task dependencies and TaskGroups
Dependencies between a Python extract step, a Java transform and a Go publish step are declared in Python with the usual bitshift syntax or TaskGroups. Keep groups organized by business stage rather than by language, so operators reading the graph can see the data flow. Confirm how each language task reports success or failure back to its upstream and downstream tasks before relying on that chain for downstream gating.
Asset and schedule definitions
Asset URIs, producer and consumer relationships and asset-triggered schedules are defined in Python, so the scheduler can decide which DAGs to run without starting a language runtime. A Java or Go task can update an asset, but the declaration that links it to consumers stays in the DAG file. Review asset names as shared contracts, since a rename in Python breaks every consumer that depends on it.
Operator choice for external systems
Operators from provider packages still handle connections to warehouses, object stores and queues in Python, even when the business logic inside the task is written in Java or Go. Decide which system clients live in the operator and which live in the task binary, because credentials and connection handling differ between the two. Confirm which connection details the task receives at run time and how they are masked in logs.
DAG bundle delivery
The DAG bundle that the scheduler and DAG processor read must contain the Python definitions, and the language artifacts must reach workers through a path the task can execute. Teams should decide whether compiled Java or Go binaries ship in the same image, a separate artifact store or a bundle version tied to a Git commit. Verify that a rerun uses the artifact version the team expects, because the DAG and binary can drift if they are released separately.
Python-side tests
Tests for the Python DAG cover import errors, graph shape, dependency wiring and schedule behavior, and they run quickly in CI without the Java or Go toolchain. Pair these with contract tests for each language task that check inputs, outputs and exit codes against the same fixtures. Ask who owns each test layer, since a graph that passes Python tests can still fail if the compiled task changes its contract.
Choosing between Python, Java SDK, Go SDK and containerized tasks
This table compares the four ways to run task logic in Airflow 3.3: Python tasks, the Java and Go Task SDKs, and containerized tasks. Read each row as a decision question, and use the last column to see when each option is the better fit for your team and executor.
| Decision factor | Python tasks in DAG files | Java or Go Task SDK | Containerized task |
|---|---|---|---|
| Language fit | Native to Airflow; DAG definitions and most operators are Python | Suits teams whose existing logic is in Java or Go and who want it run as Airflow tasks | Any language that packages into an image; no SDK dependency on Airflow's side |
| Dependency isolation | Shares the Python environment managed for your workers; conflicts need careful pinning | Runs with the language's own dependencies; confirm how each SDK packages and launches task code | Strongest isolation, since each task carries its own image and libraries |
| Operational footprint | Lowest: no extra build or image pipeline for simple tasks | Adds an SDK version to track alongside Airflow; confirm supported runtime versions with the release notes | Requires an image build, registry and pull path, plus cluster capacity for each run |
| Observability | Task logs and metrics through standard Airflow views | Logs and state flow through the Task Execution API; verify what your telemetry setup captures | Container logs and pod events add a second place to look when diagnosing failures |
| Access to Airflow state | Uses the REST API or Task SDK rather than database sessions, as Airflow 3 requires | Uses the same API boundary; secrets and connections reach tasks through the execution token | Typically calls the REST API directly; confirm which credentials the image receives |
| Best when | Logic is short, Python-friendly or orchestration-only | Team owns Java or Go code that must stay in that language and runs on your workers | Tasks need system packages, GPUs or a fixed runtime that a shared worker cannot provide |
How to set up a Java or Go Task SDK on Airflow 3.3
This sequence takes an Airflow 3.3 environment from a version check to a verified run in which a Python DAG hands a task to a Java or Go worker. Each step closes with a check you can record, so the team knows the language runtime, the DAG routing and the delivered artifact work before any production use.
Confirm the Airflow and platform versions
The platform owner confirms that every Airflow component runs the 3.3.2 patch release, the current patch in the version support table, on Python 3.10 to 3.14. If you use PostgreSQL or Kubernetes, confirm PostgreSQL 14 to 18 and Kubernetes 1.30 to 1.35. Use PostgreSQL or MySQL as the metadata database, since SQLite is for testing only. Confirm the scheduler, DAG processor and API server run as separate components. Airflow 2 reached end of life on April 22, 2026, so do not build new work on it. Done when the version output on each component image matches the prerequisites page.
Install the SDK for your language
The Java or Go developer adds the Task SDK for the chosen language to the worker build, using the package coordinates and version named in the 3.3.0 release notes and the apache/airflow repository. Pin the SDK to the same release line as the Airflow core so the worker speaks the Task Execution API version the scheduler expects. Confirm exact artifact names in the repository, since this sequence does not fix them. Done when a clean build of the worker resolves the SDK and the dependency tree lists it.
Route a task to the language runtime
The DAG author writes the DAG in Python and declares a task that dispatches work to the external Java or Go runtime, using the operator or entry point the SDK documentation specifies for that language. Scheduling, dependencies, retries, parameters and tags stay in the Python file, so the operations team reads one pipeline definition. Done when the DAG parses with no import errors and the task appears in the graph view with the expected upstream and downstream links.
Build and deliver the worker artifact
The developer compiles the Java or Go task code into an artifact and packages it into the worker image or the DAG bundle that workers pull. Task code must reach Airflow state through the Task Execution API, not through database sessions or models, so the artifact should contain no direct metadata database access. Record the image digest or bundle version with the change ticket. Done when a worker from that artifact starts, registers with the scheduler and reports as available.
Review execution token and secret handling
The security owner reviews how the worker authenticates. JWTs authenticate both the public REST API and the internal Task Execution API. Execution tokens default to a 10-minute lifetime with automatic refresh, cannot be revoked, and carry no per-DAG authorization. Keep credentials out of the artifact and retrieve them through the secrets backend your team approves. Confirm this with your security team and against the security model page. Done when a test task run cannot read a connection that belongs to another DAG.
Run a test DAG and read the logs
The operator triggers a manual run in a non-production environment and watches the task instance move through its states in the UI. The language runtime writes to standard output, which Airflow captures as task logs. Check that the log lines from the Java or Go process appear under the task instance, and that the task ends in the success state with the expected return value or output. Done when the run record, task state and log lines all match the test case written for this task.
Verify retries and failure handling
The tester forces a controlled failure in a test build, such as a non-zero exit, and confirms that the retry count and delay set in the DAG definition apply. Confirm the failed attempts appear as separate task try records and that the final state reflects the last attempt. Check which DAG bundle version a retry uses against the rerun setting for your bundle type. Done when the retry sequence and final state match the written expectation, and the failure surfaces in your alerting path.
Security and operational considerations for language SDK workers
Language Task SDK workers run task code in a JVM or Go process that sits inside your Airflow trust boundary. The cards below cover the token model, the authorization gaps, the secrets and supply chain exposure, and the version discipline you need before a Java or Go task runs in production.
Execution token lifetime and revocation
Execution tokens for the internal Task Execution API default to a 10-minute lifetime and refresh automatically while a task runs. They cannot be revoked, so a token captured from a compromised worker stays usable until it expires. Ask your team to confirm how the worker process stores the token in memory, how long a single task may run, and what compensating controls you will use to limit exposure, such as network segmentation between workers and the API server.
No per-DAG authorization in tokens
Execution tokens carry no per-DAG authorization, so the token proves a worker is running a task but does not restrict which DAG or connection that task may touch. A Java or Go task that can read its token can call the API for any DAG the token's scope allows. Map which teams own which DAGs, decide whether Java and Go workers run in their own pools, and confirm with your Airflow platform team how the auth manager and API server limit cross-team access.
Worker isolation and secrets in the process
A JVM or Go worker holds connection details, credentials and the execution token in one process address space, and a dependency with a flaw can read all of them. Run language workers in separate containers or pods with minimal service accounts, short-lived credentials and no access to the metadata database. Ask which secrets backend the SDK reads from, how values are masked in logs, and whether task code can print them to stdout.
Dependency and artifact supply chain
Java tasks pull JARs from Maven repositories and Go tasks compile modules from proxies, so every build adds transitive dependencies that your team did not review. Pin versions, generate a software bill of materials for each worker image, scan artifacts before promotion, and sign images. Confirm whether your build pipeline mirrors dependencies internally, because a worker that fetches code at runtime bypasses the review you ran at build time.
Logging and OpenTelemetry metrics
Airflow 3.3.0 added OpenTelemetry timer metrics as histograms, which helps operators compare task timing across SDK and container workers. Task logs from Java or Go code still need the same redaction, retention and access rules as Python logs, because stack traces and debug output often contain payloads. Confirm where worker logs land, who can read them, and that the metrics pipeline does not export task arguments or identifiers you are not permitted to store.
Upgrade alignment between SDK and core
A language SDK speaks the Task Execution API of a specific Airflow release, so an SDK version that lags the core can fail in ways that look like task errors. Plan SDK and core upgrades in the same change window, test a Java or Go task against the target core in staging, and keep a rollback image for the worker. Airflow 2 has reached end of life, so any worker that must stay on a supported line needs an Airflow 3 target on your schedule.
Questions teams ask before adding Java or Go tasks
What does it cost to write Airflow tasks in Java or Go?
Most of the cost is engineering time, because the Airflow project itself is released under the Apache License 2.0 and charges no fee for its SDKs. Your staff or a partner is paid for scoping, building the task packages, writing tests and wiring CI. The one-time work covers the first task, the build pipeline and the runbook. The recurring cost is maintenance: moving the SDK version with each Airflow release and keeping the JVM or Go toolchain current. Effort rises with the number of languages, the number of tasks and the number of existing containerized tasks that must be ported. Ask any quote to separate the one-time build from ongoing support and to price each language on its own line.
How long does a first Java or Go task take to reach production?
No duration is fixed by the SDK, so the timeline depends on the path your change takes to production. The steps are confirming an Airflow 3.3 or later environment, standing up a non-production deployment, building one small task, promoting it through staging and passing change review. Each step has its own owner, and the slowest one is usually approval or access to a test environment rather than the code itself. Ask for a written plan with milestones for the first task, then a separate estimate for the remaining pipelines once the pattern is proven.
What skills and tools must our team provide for Java or Go tasks?
Your team must supply four things. First, developers who can write, package and test a task in the chosen language. Second, a non-production Airflow 3.3 environment where the task can run against real connections and variables without touching production data. Third, CI that builds the task artifacts and runs their tests on every change. Fourth, an owner for the worker runtime where the tasks execute, since that owner decides which JDK or Go version is installed and how it is patched. Check the SDK documentation for the exact runtime each language expects in your environment.
Can our internal team build Java or Go tasks without outside help?
Yes, if the team already runs Airflow 3 and has at least one engineer comfortable in the task language. Internal teams usually succeed when one person owns the first task end to end and the rest follow that pattern. They struggle when nobody on the team can review code in the second language, or when the same people must also run the platform, handle security reviews and keep upgrades on schedule. Be honest about that capacity before committing to a pipeline migration. A useful internal test is whether someone other than the author can debug a failed task run from the logs alone.
When should we bring in a partner for Airflow task SDK work?
Bring in a partner when the scope is defined, the deadline is fixed, several languages or many pipelines are involved, or when your staff cannot cover the security review. DeployOpen plans and implements self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. Most engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts, and ongoing support, upgrades and testing are available as retainers. Ask for a scope that lists which tasks are moving, which languages are involved, how existing containerized tasks will be handled and who signs off on each stage.
Do existing Python DAGs have to change to use Java or Go tasks?
Usually the DAG definition stays in Python. The language SDKs give you a way to write task logic in Java or Go, while the DAG, its schedule, dependencies and the call that runs each task remain Python code. The changes you should expect are a new task package for each migrated task, a new operator call in the DAG and new tests that cover the boundary between the two languages. Also review any Python task that still reaches into Airflow database sessions or models, because Airflow 3 requires task code to use the REST API or Task SDK instead. The release notes list the 3.3.0 changes.
What security checks apply to tasks written in Java or Go?
The same trust model applies as for Python tasks. Tasks reach Airflow through the Task Execution API, which authenticates with JWTs. Execution tokens default to a 10-minute lifetime with automatic refresh, they cannot be revoked, and they carry no per-DAG authorization. That means a task running with a token can act within whatever that token allows, so your controls must limit which code can run on workers, which secrets those workers can read and who can publish task packages. Confirm the token settings for your exact version, then review package signing, dependency scanning and secret access in your CI and worker configuration. The security model explains the token behavior.
How DeployOpen scopes and delivers language Task SDK work
DeployOpen plans, builds, tests and hands over Apache Airflow language Task SDK work for Java and Go inside your own infrastructure. We start from a written scope that names which tasks move to Java or Go, which stay in Python DAG definitions, and how workers reach the Task Execution API. Engagements are fixed-scope, and the handover includes the documentation your engineers need to run the code.
How we scope
- We inventory existing tasks and mark which ones have a real need for Java or Go code.
- The written scope names the Airflow 3.3 or later version, the target SDK, and the worker image each language task runs in.
- We record which DAG definitions stay in Python, and who owns each side of that boundary.
How we build
- We build language tasks against the Task Execution API, with execution tokens handled by the Task SDK rather than custom code.
- Each task gets tests in its own language toolchain, plus a DAG-level test that confirms Python and Java or Go tasks exchange data as agreed.
- We set container images, dependency pinning and build pipelines so workers match the Airflow version in your environment.
How we hand over
- You receive source code, build files, container definitions and a runbook for deploying new task versions.
- Documentation covers which tasks are language-based, how to add a new one, and how to roll back a faulty release.
- We walk your engineers through a live run, including logs, retries and failure paths, before the engagement closes.
How we support
- Ongoing support, upgrades and testing are available as a retainer, including updates when Airflow releases new patch versions.
- A retainer can cover SDK version changes and compatibility checks before you upgrade the Airflow core.
- Any change to the written scope is agreed in writing before work starts, so cost and timeline stay visible.
