Apache Airflow and Apache Kafka integration: what it does and when to use it
The Apache Kafka provider handles producing, consuming and sensing work in Airflow, and asset watchers can start DAGs from Kafka messages. For a team that already runs scheduled pipelines in Airflow and now needs to react to Kafka topics, the supported route is the Apache Kafka provider package, which documents operators, sensors and triggers for message queue work, as described in the Apache Kafka provider package documentation. Event-driven starts are a separate mechanism. An AssetWatcher, added in Airflow 3.0, updates an asset when an event trigger fires, and only triggers that inherit from BaseEventTrigger are compatible. The provider's documentation names Kafka as one of the message broker sources alongside SQS, Pub/Sub and Azure Service Bus, so the Kafka trigger is what connects a topic to an asset update and, from there, to a DAG schedule.
Before choosing this pattern, the team should separate two jobs. Airflow is strong at ordered, dependency-aware batch work with retries, logs and a visible run history. A Kafka cluster is built to hold a continuous stream of events for many consumers, with its own retention and replay behavior. Running high-volume event processing inside Airflow tasks usually means the scheduler and triggerer carry load they were not designed for, so the reader should ask whether each topic needs a DAG run per message or a batch run per window. Confirm the provider version and its minimum Airflow version against the environment before committing, since the provider page states its own minimum requirement. Also confirm which Kafka authentication mechanism the cluster uses, because the connection settings and the trigger configuration both depend on it, and plan the consumer to be idempotent because event-driven scheduling can deliver duplicates after a triggerer restart.
Prerequisites for the Apache Airflow and Apache Kafka integration
The Kafka provider, the Airflow version, the Kafka cluster and the Triggerer all have to be ready before any DAG talks to a broker. Check each of these components first, because a mismatch in one usually surfaces as a confusing failure in another.
Airflow 3 and provider versions
The Kafka provider release 2.1.0 requires Airflow 2.11.0 at minimum, but asset watchers, which let Kafka messages start DAGs, need Airflow 3.0 or later. Plan on Airflow 3 and pin the provider version in your constraints file so workers, the scheduler and the Triggerer all install the same package. Confirm the current patch release of Airflow 3 against the version support table before you choose a target.
Supported Kafka cluster release
Build the integration against a Kafka release on the supported list, which is 4.1.2, 4.2.2 or 4.3.1 at the time of writing. Choose one that your broker team can keep patched, since bugfix releases are made only for supported lines. Confirm the release you choose against the Kafka downloads page before you start.
KRaft-mode cluster
Kafka 4.0 and later run only in KRaft mode, so ZooKeeper-based clusters must migrate before you upgrade. If your cluster predates 3.3.x, the upgrade notes recommend moving to 3.9.x first. Confirm the controller quorum is healthy, because consumer group offsets and ACLs depend on it.
Triggerer capacity and placement
Kafka event triggers run on the Triggerer, not on workers, so its memory and placement determine how many watchers you can run. Size it for the number of asset watchers, run it as a component separate from the DAG processor, and confirm that a restart will not interrupt consumers you depend on. Ask your team to test with the watcher count you expect in production.
Network paths to brokers
Workers running Kafka tasks and the Triggerer both need routes to every broker advertised in the cluster metadata, not only the bootstrap address. Check DNS, firewall rules and any load balancer in front of brokers from each Airflow host. Test these paths from the same nodes and networks Airflow uses, since a route that works from your laptop often does not work from a pod or a worker subnet.
TLS and SASL material
Kafka supports SSL and several SASL mechanisms, including SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Prepare the CA chain, client certificates or SASL credentials, and the exact mechanism your brokers enforce before you write the connection. Store these in a secrets backend, not in DAG code, and assign each Airflow component its own identity so access can be revoked separately.
Metadata database engine
The Airflow metadata database must be PostgreSQL or MySQL for production. SQLite is for testing only and can lose data, and MariaDB is not supported. Confirm the engine version against the prerequisites page before you build, because the integration stores connection and trigger state there.
How to configure the Kafka provider and asset watchers in order
This sequence takes a self-hosted Airflow from an installed Kafka provider to a DAG that starts when a Kafka message arrives. Each step ends with a check so the team knows the layer below works before building the next one.
Install the Kafka provider against constraints
The platform engineer installs the Apache Kafka provider into the same Python environment or image that runs the Airflow 3 components, pinned using the Airflow constraints file for the target Airflow version. The provider's release notes state a minimum Airflow version, so confirm your running Airflow meets it before installing. The step is done when the provider version appears in the installed package list on the scheduler, worker and triggerer images, and the DAG processor imports the Kafka operators without errors.
Create the Kafka connection with security settings
The Airflow administrator creates a Kafka connection in your secrets backend or in the Airflow connection store, holding the bootstrap servers and the security settings your cluster requires. Those settings follow the broker's listener configuration, which may use SSL/TLS or one of the SASL mechanisms such as SCRAM-SHA-512 or OAUTHBEARER. Confirm the exact configuration keys with your Kafka team and the provider documentation rather than copying them from another cluster. The step is done when the connection test or a first client handshake succeeds, and no password appears in DAG code or in task logs.
Prove broker reachability from a worker
Run a small check from a worker pod or host, not from a developer laptop, because that is where production tasks and the triggerer will connect. The check confirms DNS resolution, network routes, TLS trust chain and authentication to every advertised broker address, since a bootstrap server can answer while a later broker address is blocked by a firewall rule. Ask the Kafka team which listener and advertised addresses apply to your network zone. The step is done when metadata for the target topic is returned from the worker environment.
Build a produce and consume test DAG
The data engineer writes a short DAG on a non-production topic that produces one uniquely keyed test message and then consumes from the same topic in a separate task. The consumer task sets auto-commit to disabled and commits offsets only after it has processed the message and recorded its result. Use a dedicated consumer group for the test so it does not interfere with real consumers. The check is passed when the produced test message is read back, its key and payload match, and the committed offset for the consumer group advances exactly once.
Define a BaseEventTrigger subclass
The developer writes a trigger class that inherits from BaseEventTrigger and implements the Kafka polling loop, so the triggerer waits on the topic instead of a worker slot. Keep the trigger lightweight: it should emit an event containing the message metadata, not the full payload, and it should avoid heavy imports. Triggers only run on the triggerer, so confirm that at least one triggerer is deployed and that it can reach the brokers. The check is done when a trigger instance started in a dry run emits an event for a test message and logs a clean shutdown when stopped.
Attach an AssetWatcher to an asset
The developer declares an asset and attaches an AssetWatcher that points to the trigger from the previous step, so an incoming event updates the asset and can schedule downstream DAGs. Name the asset and the watcher clearly, because the asset name is what consuming DAGs reference. Validate that only compatible event triggers are used, since triggers must inherit from the base class to work with watchers. The check is done when a test message published to the topic updates the asset and starts the consuming DAG once, visible in the DAG run history.
Verify idempotency and offset handling end to end
The data engineer tests the full path by restarting the triggerer while messages are in flight, then confirms what the downstream DAG did. Event-driven scheduling can deliver duplicates after a triggerer restart but does not lose events, so the consuming task must write results keyed on a message identifier and skip work it has already completed. Record which offsets were committed and compare them with the processed results. The step is done when a restart produces no missing outputs and any duplicate deliveries are absorbed without double writes.
Confirm consumer settings before production
Before the watcher is enabled for a production topic, the platform owner reviews the consumer configuration with the Kafka administrators, including the consumer group name, the offset policy, and the ack timeout that applies to event handling, which the event-driven scheduling documentation sets at 5 minutes by default. Confirm the Kafka ACLs grant the consumer group read access to the topic and group, because a missing ACL blocks consumption without a clear error in Airflow. The step is done when the change is recorded and the team has a rollback plan that disables the watcher without deleting the asset.
How offsets, commits and duplicate events affect pipeline correctness
The core guarantee to design around is that event-driven scheduling in Airflow can deliver duplicate events after a Triggerer restart, but it never loses one. That makes every subscriber that starts a DAG run from a Kafka message an at-least-once consumer, so the DAG and its downstream writes must be idempotent. Treat each delivery as possibly repeated, and design the first task of the run to check whether the event has already been handled before doing any work. Confirm the exact redelivery behavior of your Airflow version and triggers with the event-driven scheduling documentation before you write the consumer logic.
On the Kafka side, the consumer must disable auto-commit, which the event-driven scheduling docs require for Kafka consumers. Commit the offset only after the downstream write has succeeded, so a failure before the commit leads to redelivery rather than silent loss. Check how the Kafka provider's trigger commits offsets in the version you run (provider release 2.1.0 requires Airflow 2.11.0 or later), and confirm which consumer group each asset watcher uses so two DAGs do not share offsets by accident. Pick a group ID per subscriber and record it in the runbook.
The default ack timeout is 5 minutes. If processing an event takes longer than that, the broker or trigger can treat the message as unacknowledged and redeliver it while the first attempt is still running. Two runs can then process the same event at once. Measure the real processing time of each event type under load, then either shorten the work so it finishes within the window or raise the timeout deliberately and test that change. Key every write on a stable event identifier, such as topic, partition and offset or a business key carried in the message, with a unique constraint or upsert in the target system. Reprocessing then changes nothing, and the same key lets operators replay a window safely.
Sizing the triggerer for Kafka listeners
When a small number of topics feed DAGs, a single triggerer process often carries all the watchers. Each asset watcher runs a long-lived event trigger that keeps a consumer subscribed to its topic, so the triggerer's job is to hold those connections open and hand events to the scheduler. The load comes mostly from how many consumers are open and how fast messages arrive on the topics they read, not from the DAG count itself. Start by confirming that the triggerer is the only component running these listeners and that it is not also serving deferred operators from other DAGs.
- Watcher count: Each watcher adds one open consumer. Record the number of watchers per triggerer before adding any more.
- Message rate: Measure messages per second per topic at peak, since every matching message passes through the trigger.
- Shared work: Deferred operators from other DAGs also use the triggerer, so count them in the same baseline.
- Restart behavior: After a restart, Airflow can deliver duplicate events, so check that the consumer side handles replays before the first production run.
- Logs and memory: Watch triggerer memory and log volume over a full business cycle, not only during a quick test.
Before adding a replica, load test one triggerer with a copy of your topic volumes and confirm that event latency stays acceptable.
When the number of watchers grows, the practical step is to run more than one triggerer and divide the triggers between them. Each watcher is still a long-running trigger, so every replica you add takes on a share of open consumers and their message traffic. Spreading watchers works only if the replicas share the metadata database and the scheduler sees events from all of them. Plan the split on paper first: decide which DAGs and topics belong together so that high-volume topics do not all land on the same replica.
- Balance by volume: Group high-traffic topics across replicas instead of stacking them on one process.
- Keep the metadata database sized: Every replica writes trigger state to the same database, so connection counts rise with replica count.
- Check scheduler intake: Events from all replicas must reach the scheduler at the rate your DAGs expect, so measure end-to-end latency, not only triggerer CPU.
- Plan for uneven load: Topics rarely produce equal traffic, so measure each replica separately.
- Document the assignment: Record which watchers run where so that a failed replica can be diagnosed quickly.
After each change, compare per-replica memory, consumer lag and event latency against your baseline before adding the next replica.
On Kubernetes, the triggerer usually runs as its own deployment, separate from the scheduler, API server and DAG processor. That separation lets you scale the triggerer by replica count and set its resource requests independently. The trade-off is that pod restarts, rescheduling and node pressure now affect listeners directly, so the sizing exercise includes failure behavior as well as steady-state load. Confirm how your Helm chart or manifests set triggerer replicas and resources, and make sure your team can see restarts in its monitoring.
- Resource requests: Set memory and CPU requests from measured values under your own topic load, not from defaults copied from other environments.
- Pod disruption: Restarts cause duplicate events, so confirm that your consumers are idempotent before tuning replica counts.
- Autoscaling: Scaling on CPU alone may miss memory growth from open consumers, so choose metrics that reflect trigger load.
- Node placement: Spread triggerer pods across nodes or zones where the message broker is reachable with low latency.
- Broker connectivity: Network policies and egress rules must allow every replica to reach the Kafka bootstrap servers.
Confirm the chart version and resource settings with your deployment team, then run a restart test and check that no events are lost.
Whatever the deployment model, increasing replicas without measurements tends to hide the real bottleneck. The bottleneck might be the broker's partition layout, the consumer's processing time, the metadata database, or the triggerer's own memory. A short measurement plan tells you which one to fix. Use the same test volumes you expect in production, including bursts, and record results for each configuration so that later changes can be compared.
- Trigger count per replica: Record how many watchers each triggerer holds at steady state and at peak.
- Consumer lag: Track lag per topic and consumer group, since growing lag means events are arriving faster than they are consumed.
- Event latency: Measure the time from message production to DAG run creation.
- Triggerer memory and CPU: Watch these over long runs to catch growth from open consumers.
- Database load: Track connection counts and query latency on the metadata database as replicas are added.
Do not treat any of these numbers as a target until your own load test confirms them for your volumes and hardware.
Mapping Kafka topics, messages and Airflow assets
This table pairs each Kafka element with its Airflow counterpart, so a team can see where a topic, consumer group, message key, offset or broker credential lands in an asset watcher, run or connection. Read it as a design checklist: each row names the decision to make before the first DAG is written.
| Kafka element | Airflow counterpart | Design note |
|---|---|---|
| Topic | Asset name and the trigger's topic setting | Name each asset after its topic, for example kafka-orders-v1, so the mapping is visible in the UI and in code review |
| Consumer group | Watcher consumer identity and offset state | Give each Airflow pipeline its own consumer group so one pipeline cannot move another pipeline's offsets |
| Message key | Run conf or XCom value | Pass the key as a parameter only if it is needed for routing; keep large payloads out of XCom |
| Message payload | Event data read by the triggered run | Parse and validate the payload in the first task so malformed messages fail early with a clear error |
| Partition and offset | Idempotency key recorded by the consumer | Store the topic, partition and offset together in a table keyed by that triple, then skip any event already recorded |
| Auto-commit setting | Consumer configuration in the trigger | Disable auto-commit so offsets advance only after the event has been handed to Airflow, as the event-driven scheduling docs require |
| Broker authentication | Airflow connection holding SASL or SSL settings | Store SASL credentials and certificates in a secrets backend, not in DAG code or plain connection text |
| Duplicate detection field | Partition plus offset, with the event timestamp as a secondary check | Duplicates can follow a Triggerer restart, so a repeated partition and offset should be ignored, not reprocessed |
Securing the Kafka connection from Airflow
An Airflow task that reads or writes Kafka is a client like any other, so it needs encryption, authentication and authorization configured on the cluster side and kept out of DAG code. The cards below cover the decisions that connect the Airflow provider settings to the Kafka listeners and ACLs they depend on.
Encrypt client-to-broker traffic
Kafka security is optional, so a listener can accept plaintext connections without any error. Before wiring the provider, confirm every listener the Airflow workers and triggerers will use is served over SSL/TLS, and that no unsecured listener is left open for convenience. Ask the Kafka team to show the listener configuration and to confirm which broker certificates the Airflow hosts must trust, including any internal certificate authority used for the cluster.
Choose a SASL mechanism deliberately
Kafka documents GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER for SASL authentication. PLAIN sends credentials that must be protected by TLS at every hop, so pair it only with encrypted listeners. Choose the mechanism your identity estate already operates, and confirm with the Kafka team which one the cluster offers to the Airflow principal, since a mechanism the broker does not enable will fail at connection time.
Keep credentials out of DAG code
Passwords, keytabs, client certificates and OAuth client secrets should live in an Airflow connection or a secrets backend, never in DAG files or in the consumer configuration dictionary written into the repository. DAG files are readable by every author and are often copied into logs and CI output. Ask who rotates each credential, how a rotation reaches running triggerers, and whether the secrets backend is reachable from the triggerer, since event-driven consumers run there.
Create a dedicated Airflow principal
Give the Airflow workers and triggerers their own Kafka identity rather than sharing a human or broker account. Separate principals let the Kafka team grant produce access to the topics a DAG writes and read access to the consumer groups and topics it watches, and they make audit lines attributable to Airflow. Agree the naming convention for principals and consumer groups before the first DAG is deployed, so later ACL changes stay narrow.
Plan KRaft ACLs before the first run
In KRaft clusters the StandardAuthorizer stores ACLs in the cluster metadata log. Resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so a new topic may appear to work for an administrator and fail for the Airflow principal. Grant the Airflow principal explicit read, write and describe permissions on its topics and groups, and confirm with the Kafka team that allow.everyone.if.no.acl.found is not enabled as a shortcut.
Verify before production traffic
Test each path with the real Airflow principal: a producer task, a consumer task and an asset watcher trigger against a non-production topic. Confirm that a connection attempt over plaintext is refused, that an unauthorized topic returns an authorization error, and that a rotated credential is picked up by workers and triggerers without a manual edit to DAG code. Record the results as change evidence for the security review.
Common Kafka failures in Airflow and how to fix them
This table lists the failures teams most often hit when Airflow produces to or consumes from Kafka, with the cause and fix for each. Read it as a triage guide: match the symptom you see, check the cause, then apply the fix before changing anything else. The main takeaway is that most failures trace back to offset handling, consumer idempotency or a trigger that does not meet the event-driven contract.
| Failure | Likely cause | Fix |
|---|---|---|
| Duplicate events after a Triggerer restart | Event-driven scheduling can redeliver events after a Triggerer restart. It does not lose events, so the subscriber sees repeats. | Make the downstream DAG idempotent, for example by keying writes on the message offset and partition or a business key, and treat a second delivery as a no-op. |
| Consumer lag growing while tasks run slowly | Slow downstream tasks or a blocked triggerer delay how quickly messages are read, so the consumer falls behind the topic. | Profile the triggerer and the DAG tasks that consume events, reduce per-message work, and size consumer throughput against partition count and message rate. Confirm lag metrics with your Kafka monitoring. |
| Messages lost after a consumer restart | Auto-commit is enabled, so offsets are committed before the event is durably handled. | Disable auto-commit on Kafka consumers and commit offsets only after the event has been processed successfully. |
| Ack timeout exceeded and messages redelivered | Processing an event takes longer than the default ack timeout of 5 minutes, so the message is treated as unacknowledged. | Shorten the work done per event, move long processing into a separate task, or confirm a longer timeout is appropriate for your workload before changing it. |
| ACL denials for the Airflow principal | In KRaft clusters, resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is true. The Airflow client principal has no grant for the topic or consumer group. | Grant the Airflow principal read on the topic and group, and write where it produces. Keep allow.everyone.if.no.acl.found at false in production and verify grants with your cluster administrator. |
| TLS handshake or trust errors | The client trust store does not include the broker's CA, the certificate names do not match the broker address, or a listener mixes secured and unsecured endpoints. | Confirm which listener the Airflow connection targets, load the correct CA into the connection configuration, and check hostname entries in the broker certificate against the bootstrap servers. |
| SASL authentication failures | The mechanism configured in Airflow (for example SCRAM-SHA-512 or OAUTHBEARER) does not match the one enabled on the listener, or credentials are stored in the wrong place. | Match the SASL mechanism on both sides, keep credentials in an Airflow secrets backend rather than DAG code, and test the connection before enabling the watcher. |
| Asset watcher never fires | The trigger does not inherit from BaseEventTrigger, so it is not compatible with AssetWatcher and the asset is never updated. | Rewrite the trigger so it inherits from BaseEventTrigger, or use a Kafka trigger from the provider that already does, and confirm the watcher references that trigger. |
Questions buyers ask before connecting Airflow to Kafka
What does connecting Airflow to Kafka cost, and what drives it up?
The cost is mostly the engineering time to design, build and test the connection, and it is largely one-time. The main drivers are the number of event-driven DAGs (each trigger is a separate piece of design and testing), how much idempotency work the downstream writes need, and the security model the broker already enforces. A quote should itemize the trigger count, the topics and consumer groups in scope, the authentication method (SSL or one of the SASL mechanisms), the staging environment, and the handover documentation. Recurring costs are the infrastructure that runs the Triggerer and any retained Airflow capacity, plus support if you buy a retainer. No fixed figure applies here, so ask for a written scope that lists each of these items before comparing quotes.
How long does a Kafka integration take, and what sets the timeline?
Duration depends on scope more than on the Kafka library itself. A narrow setup with one producer task and one event-driven DAG is much smaller than a set of consumers that must each handle duplicate events safely. The timeline usually moves with access approval for brokers and ACLs, the number of topics, the format of the messages, and how many downstream systems must be tested for duplicates. Ask for milestones tied to a staging Airflow, a test topic and a sign-off on the consumer design. Because Airflow's event-driven scheduling can redeliver events after a Triggerer restart, expect testing of restart behavior to take a meaningful share of the schedule.
What must our team provide before the work starts?
The team needs to provide several things before design begins:
- Broker access and bootstrap server addresses for a staging cluster, plus the authentication method your cluster uses.
- ACLs that allow the Airflow service identity to read or write only the named topics and consumer groups. In KRaft clusters, ACLs live in the cluster metadata, and resources without an ACL are restricted to super users unless your administrators have set the allow-everyone option, so this step is not optional.
- Test topics and sample messages that match production schemas.
- A staging Airflow that runs the same major version as production, with the Kafka provider installed.
Confirm these with your Kafka and security owners early, because approvals are often the longest lead item.
Can our in-house team build and run this without a partner?
Often yes, if the team already runs Airflow 3 and operates Kafka. The work needs three skills: Airflow DAG authoring with deferrable operators and the triggerer, Kafka client configuration (including disabling auto-commit and managing offsets deliberately), and enough security knowledge to set up SSL or SASL and ACLs correctly. Teams that lack one of these usually spend the most time on the gap. A partner makes sense when you want a fixed-scope build with written deliverables, when you need staging and failure testing done before go-live, or when your own engineers are committed elsewhere. Whichever route you take, the handover should include a runbook for restarts, replays and consumer lag.
How many event-driven DAGs can one Triggerer handle?
There is no safe general number, so measure it in your environment. Each event-driven DAG adds a trigger that the Triggerer runs, and each trigger holds a Kafka consumer connection and polling activity. The Triggerer's capacity depends on the message rate per topic, the size of each message, the number of partitions, and the other triggers that share the process, such as deferrable operators. Plan a staging load test that matches your peak message rate, watch the Triggerer's resource use and consumer lag, and then size production with headroom. Ask the implementer to show the load test results and the sizing rule that follows from them.
How do we make consumers safe when Kafka sends duplicate events?
Design every subscriber to be idempotent, because event-driven scheduling can deliver duplicate events after a Triggerer restart but does not lose them. In practice, each consumer records a unique key for the work it has already completed, such as a topic, partition and offset combination or a business key from the message, and skips repeats. Writes to databases should use upserts or keyed merges rather than plain inserts. The default acknowledgement timeout is 5 minutes, so long-running handlers need to finish inside that window or be restructured. Test by forcing a Triggerer restart mid-batch and confirming that no record appears twice in the target system.
Which security model should the Airflow connection use?
Use the authentication that your Kafka cluster already enforces. Kafka supports SSL for client authentication and encryption, and SASL mechanisms including GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Keep the Kafka credentials in the Airflow secrets backend or another secrets store you control, not in DAG code or plain environment variables. Scope the service identity with ACLs to specific topics and consumer groups, and encrypt traffic with SSL/TLS. Record which listener the Airflow connection uses, since a cluster can run secured and unsecured listeners side by side. Confirm the chosen mechanism with your Kafka administrators before design starts.
When is Kafka Connect, Flink or Kafka Streams a better fit than Airflow?
Choose a streaming tool when the data must be processed continuously, with low latency and state held between events. Kafka Connect is a better fit for moving data between Kafka and external systems with little custom code. Kafka Streams suits stateful transformations that run inside a Java application that reads and writes Kafka directly. Flink suits large, stateful stream processing with complex windows and event-time logic. Airflow fits when work is batch-oriented, when a message should start a defined workflow with dependencies, retries and human-readable run history, or when Kafka is one step among many systems in a schedule. Many production designs use both: a streaming tool keeps the topic current, and Airflow runs the scheduled work that depends on it.
When Kafka belongs in a streaming tool instead of Airflow
Airflow is built to run batch-oriented dependency graphs: a set of tasks with defined order, retries, schedules and sensors that wait for a condition before continuing. Kafka integration in Airflow fits that model well when a DAG needs to publish a control message, read a bounded window of events, or wait for a trigger message before a downstream load starts. It fits less well when the workload is a continuous stream: high-throughput fan-out to many consumers, per-event processing with low latency, or stateful windowed aggregations that must survive restarts. Those workloads belong in a streaming tool such as Kafka Streams or a dedicated stream processor, where the processing runs next to the data rather than in Airflow task slots. Before choosing, confirm the message rate, the latency each consumer needs and whether the logic is stateless or keeps state across events.
Several Kafka components are often confused in this decision. Consumer groups are the Kafka mechanism by which consumers share partitions and track offsets, and they are used by any client, including an Airflow task or a triggerer. Kafka Connect moves data between Kafka and external systems as a managed runtime with its own connectors and scaling model. Kafka Streams is a client library for stateful stream processing inside your application. Airflow can coordinate around these components, for example by starting a Connect job or checking a topic's state, but it does not replace them. Ask which of these the team already operates, because each adds its own deployment, upgrade and monitoring work.
Cross-site replication is a separate concern. Kafka geo-replication, documented in the Apache Kafka 4.3 operations guide, is the mechanism for mirroring topics between clusters, and an Airflow DAG should not be the tool that copies event logs across sites. If a pipeline depends on a remote topic, confirm which cluster is the source of truth, how offsets map across clusters and who owns failover. The Kafka geo-replication documentation is the next read for those questions. The right answer is often a mix: Kafka and a streaming tool handle the continuous path, and Airflow schedules the batch steps that consume its output.
How DeployOpen scopes and delivers Airflow and Kafka integrations
DeployOpen plans, builds, hardens and documents Apache Airflow and Apache Kafka integrations inside infrastructure you control, under a written, fixed scope agreed before work starts. Each engagement covers the provider configuration, consumer design, triggerer capacity and the operating runbook your team needs after handover.
How we scope
- We list every producer, consumer and topic the DAGs will touch, with the Kafka and Airflow versions in use.
- We confirm the broker security setup, including TLS and SASL mechanisms, and which Kafka ACLs each client needs.
- We agree in writing which DAGs use operators, sensors or asset watchers, and what replay and duplicate handling is expected.
How we build
- We configure Kafka connections through Airflow's secrets handling and keep credentials out of DAG code.
- We build consumers with auto-commit disabled and idempotent writes, then test restarts of the triggerer and broker outages.
- We size triggerer capacity against the number of watched topics and document the sizing assumptions for your team.
How we hand over
- We deliver the DAG code, connection definitions, ACL requests and a runbook covering offsets, retries and replay.
- We document which workloads belong in Airflow and which should stay in a streaming tool, with the reasons recorded.
- We walk your engineers through monitoring, alerting and the steps to recover a stalled consumer group.
How we support
- Ongoing support, Kafka and Airflow upgrade planning and continuous testing are available as a retainer.
- Upgrade work covers provider package changes and Kafka release moves, checked against the versions your cluster runs.
- Any change to scope, such as a new topic or consumer, is quoted and agreed in writing before it is built.
