When Apache NiFi flows need Wait and Notify coordination
A team often has data that cannot move downstream until a related event has happened: a parent file must land before its child records are loaded, or a set of fragments from one upstream export must all arrive before a combined load can run. In Apache NiFi, the Wait processor holds each FlowFile in a wait relationship until a matching release signal appears, and the Notify processor is what writes that signal. This page explains how the two processors pair, how to count fragments, how timeouts should be handled, and which alternatives fit better when the coordination needs to survive restarts or span longer periods.
Typical use cases include holding child records until their parent header file has been processed, delaying a database load until every fragment of a split export has finished its own processing, and releasing a waiting FlowFile once an external step such as a validation or enrichment branch reports success. In each case, the release key is usually a shared attribute, and both processors must agree on it exactly. Teams should map which flows produce signals, which flows wait, and what happens to a waiting FlowFile that never receives its release, before the flow is built.
The release signal lives in a distributed map cache, not in a durable workflow engine. Its contents depend on the cache service being available and on the cache's own retention, so a cache restart or cleanup can remove signals that a waiting FlowFile still needs. Before relying on this pattern, confirm with your NiFi version's component documentation how the cache is configured and shared across nodes in your cluster, and decide whether the coordination is important enough to need an orchestrator with durable state instead.
How Notify releases waiting FlowFiles through the cache
This sequence shows how a FlowFile parked at Wait is held until a Notify processor writes a matching release signal to a distributed cache, so the flow can continue only when the related event has happened. The result is a waiting FlowFile that moves to the success path on release or to the wait path until its signal arrives.
Agree the release signal key and attributes
Before building the flow, the flow designer decides which FlowFile attribute will identify the related event, such as a batch ID or an order number, and which attributes the Notify side should store with the signal. Both sides must produce the same value for the same event. The step is done when the team has written down the attribute name, its expected format and the upstream source that sets it.
Route the waiting FlowFile into Wait
The waiting FlowFile reaches a Wait processor, which reads the configured attribute and checks the cache for a release signal with the same key. If no matching signal exists, Wait routes the FlowFile to its wait relationship and keeps it queued for later checks. The check is complete when a test FlowFile lands in the wait relationship and remains there while no signal exists.
Store the release signal with Notify
When the related event occurs, a separate flow path sends a FlowFile to a Notify processor. Notify writes a release signal to the distributed cache under the same key, and can include selected FlowFile attributes with that signal. The Notify output is the signal itself, so the designer should verify it by inspecting the cache entry after a test run.
Match the signal in Wait
On its next run, Wait finds the stored signal that matches the key on the waiting FlowFile. Attributes stored with the signal are then available to the released FlowFile, which lets downstream processors use values from the event. The check is that the test FlowFile is routed to the success relationship and carries the expected attributes.
Route released FlowFiles onward
Released FlowFiles leave Wait through its success relationship and continue to the next processors, such as a write to a target system. Wait behavior on failure or timeout is set in the processor configuration, so the team should review which relationship carries each outcome. The check is that released FlowFiles reach the target exactly once in a test run.
Confirm the distributed cache service
Wait and Notify depend on a distributed cache service that holds the signals, so the team must confirm which cache controller service the processors use in their NiFi version. In a cluster, the signals must be visible to every node that can run Wait and Notify. Confirm the service name, its availability, and its behavior after a restart against the NiFi documentation for the installed version.
Test the release path end to end
The team runs a controlled test with a FlowFile that waits, a Notify event with a matching key, and a second FlowFile with a different key that must stay waiting. The test shows that matching keys release, non-matching keys do not, and attributes arrive intact. The step is done when the results are recorded and reviewed against the expected behavior.
Which Wait and Notify settings fit each coordination pattern
This table compares the Wait and Notify settings that decide how a signal is matched, counted, cached, expired and handled when it never arrives. Read each row by asking which coordination pattern you are building, then check the matching failure mode before you deploy. Exact property labels can differ by NiFi version, so confirm them in the component documentation for your release.
| Setting | When it fits | What breaks if it is wrong |
|---|---|---|
| Release Signal Identifier | One signal per business key, such as a batch ID or source file name read from an attribute | Identifiers that are too broad release unrelated FlowFiles, and identifiers that are too narrow never match |
| Target Signal Count with a signal counter name | Fragment counting, where a batch is complete only after several upstream FlowFiles report in | A counter name reused across batches inflates counts and releases early, or never reaches the target |
| Attribute copy into the signal (Notify) | Downstream steps need upstream values such as file path, record count or checksum | Copying too many attributes bloats cache entries, while copying none leaves Wait without the values it needs |
| Distributed cache service | Single-node flows or clusters that share one cache service for Notify and Wait | A cache local to one node in a cluster means Wait on another node never sees the release signal |
| Expiration Duration (Wait) | Signals that normally arrive within a known window, based on measured upstream run times | A duration that is too short sends valid batches to the expired path, and one that is too long holds FlowFiles and grows queues |
| Expired relationship routing | Routing stalled batches to an alert, a review queue or a re-trigger flow | Leaving the expired relationship unrouted or auto-terminated silently drops batches that never completed |
| Timeout for signals that never arrive | Batches where a missing signal is a fault that needs an operator, with a scheduled check or a separate alert | Without a timeout, waiting FlowFiles accumulate and the failure stays invisible until a backlog builds |
| Cache cleanup and signal retention | Long-running flows where stale signals must not match future batches with the same identifier | Stale signals with reused identifiers release new batches before their upstream work has finished |
Counting fragments before a batch is released
A fragment count pattern lets a flow hold every fragment of a batch until the whole batch has arrived. An upstream processor that splits one input into several FlowFiles writes the batch identifier and total count as attributes, such as fragment.identifier and fragment.count. Each fragment then runs through a Notify processor, which records a release signal in the distributed cache. The Wait processor on the other branch routes each FlowFile to its wait relationship until the signal for its batch is found. The count must come from those upstream attributes, not from a tally the flow keeps itself, because only the split step knows how many fragments it created. The release signal identifier must also be unique per batch, usually built from the batch identifier, so one batch never releases another. Confirm the exact property names and how the counter behaves in the Wait component documentation before you wire the flow.
The two failure modes are duplicates and missing fragments. A duplicated fragment that reaches Notify again can push the counter to the target before the batch is complete, so the batch is released with gaps. A lost fragment, for example one sent to a failure path that is never retried, leaves the counter short, and the waiting FlowFiles stay in the wait relationship until the cache entry expires. Deduplication has to sit upstream of Notify, and its scope must cover the whole batch window, or a late copy can slip through after the first has been counted. Ordering matters less for the count itself, since Wait only checks whether the signal exists, but it matters for the batch that is released, because downstream steps may expect the fragments in a particular sequence. Plan what happens to FlowFiles whose signal never arrives, and make sure those FlowFiles are not left waiting indefinitely.
Test this pattern with deliberate faults before production use. Replay a batch with one fragment duplicated, drop one fragment, and deliver fragments out of order across cluster nodes. Restart the cache service during a batch and observe what the cache retains. Confirm that expired FlowFiles reach the path you designed and that alerts fire on them. Check that counts reset cleanly for a retried batch, since a stale signal from an earlier attempt can release a new one. Record each result, because the count logic is only as reliable as the attributes and cache behavior your environment actually produces, and the component documentation does not promise delivery guarantees on your specific topology.
Timeouts, lost signals and stuck FlowFiles
Wait and Notify coordinate through a release signal in a cache, so most failures come down to a signal that never arrives, arrives too early, or is lost. Each risk below has a visible symptom, a mitigation pattern to design in before go-live, and a question to confirm in your own environment.
Signals that never arrive
If the upstream event fails or is skipped, FlowFiles sit in the wait relationship indefinitely. Route Wait's expired or timeout path so that held FlowFiles are released to a review queue after a bound you choose, and log each timeout with the release signal identifier. Confirm the Expiration Duration behavior and the relationship names in your NiFi 2 version's Wait documentation before you build the timeout route.
Cache loss or restart
Release signals live in a distributed cache. If the cache service is restarted, cleared or unreachable, signals already stored may disappear and held FlowFiles wait forever. Decide whether the cache is persistent or in memory, document the recovery step for replaying the upstream event, and test a cache restart in a non-production cluster. Ask your platform team how cache state is preserved across node restarts.
Signals released before Wait is reached
A Notify that runs before the matching Wait sees the FlowFile can leave Wait waiting for a signal that already came and went. This is a race in flow ordering, not a cache fault. Place Notify after the event is confirmed, not at the start of the producing branch, and use a test flow that sends both orders to confirm the held FlowFile is released. Verify this with your own event sequence rather than assuming it.
Duplicate signals
Retries, reprocessed FlowFiles or replayed events can send the same release signal twice. Design the signal key so that a duplicate does not release unrelated FlowFiles, and make sure downstream steps tolerate receiving the same release more than once. Pair this with the deduplication pattern for the incoming events so that a repeated notification is absorbed before it reaches Notify.
Cache scope in a cluster
In a clustered NiFi, a Notify on one node and a Wait on another only coordinate if both use the same cache service and the same key. Confirm which cache implementation your cluster uses and how it shares state between nodes. If the cache is local to a node, release signals stored by one node will not be seen by Wait on another, and FlowFiles will appear stuck with no error. Test the flow across at least two nodes.
FlowFiles accumulating in the wait relationship
A slow leak of held FlowFiles often shows up as queue growth on the connection leaving Wait, not as an error. Alert on the age of the oldest FlowFile in that queue and on the count of waiting FlowFiles, using NiFi's reporting and monitoring for the connection. Set thresholds from your own batch windows, and confirm with operations which person acts on each alert.
Alternatives for batch completion checks in NiFi
Wait and Notify is not the only way to decide that a batch is complete, and for some workloads it is the wrong one. Use it when a single FlowFile must be held until one specific release signal arrives. If the real question is whether all pieces of a batch have landed, a merge-based design is often easier to reason about. MergeRecord and MergeContent combine FlowFiles into one output once bin conditions are met, so completion is expressed as a bin rule rather than as a cache entry. The bin settings that matter here are the minimum and maximum number of entries, the minimum and maximum group size, the maximum bin age, and the correlation attribute that groups fragments together. Before choosing this route, confirm with your own sample data how often a bin will close on age rather than on count, because a timed-out bin is a completeness decision you must handle. The sibling topic on merging small files and batching records covers these settings in depth.
A second option is a database status table checked by a polling flow. Each producer writes a row per batch with an expected count, an arrived count and a status, and a scheduled flow queries the table for batches that have reached their expected count. This keeps the state in a system that your operations team already backs up, monitors and can query during an incident, which a distributed cache entry does not offer. The trade-off is polling load and latency, and the need to make the status update idempotent so that a retried producer does not inflate the count. Ask your database owners how they want the polling query indexed and how long completed rows are retained for audit. Ordering of batches is a separate concern covered in the topic on guaranteeing order and priority.
An external orchestrator such as Apache Airflow is the better choice when completion depends on several systems, schedules or teams, not only on NiFi data. Airflow handles dependencies between tasks, retries at the workflow level, backfills and alerting on missed runs, while NiFi keeps moving the data itself. A practical split is to have NiFi write a completion marker or status row and let the orchestrator decide when to trigger downstream work. Plan who owns that boundary, how the orchestrator learns that a NiFi run failed, and how a stalled batch is escalated. Confirm these responsibilities in writing before the build starts, since an unowned completion check is the most common cause of batches that silently never close.
Setting up Wait and Notify in a NiFi flow
This sequence sets up a release signal, a shared cache and the paired Notify and Wait processors so that held FlowFiles move on only when the matching event arrives. Each step ends with a check so the flow is proven before it carries production data.
Define the signal identifier and attributes
Decide which attribute uniquely identifies the related event, such as a batch ID, order number or source file name, and which FlowFile attributes the release signal should carry for downstream use. The team that owns the source data and the team that owns the flow should agree on this in writing, because the Wait processor matches on the same value. Check: write the identifier on a sample FlowFile from each path and confirm the values match exactly, including case and format.
Configure the cache service on every node
Set up the distributed map cache controller service that Notify and Wait will share, following the component names in the NiFi 2 documentation rather than older names from 1.x. In a cluster, confirm the service is enabled and reachable from every node, since a signal stored on one node must be visible to Wait running on another. Check: enable the service, then confirm the controller service state is enabled on each node before any processor is started.
Add Notify after the producing path
Place Notify on the path that completes the related work, so it runs only after the producing data has been written to its target. Point it at the shared cache, set the signal identifier attribute, and include any attributes that Wait should later read. Check: run a single test FlowFile through the producing path and inspect the Notify output, then confirm a release signal with the expected identifier exists in the cache.
Add Wait with release and timeout routes
Place Wait on the path that must hold until the event arrives, using the same cache service and the same signal identifier expression as Notify. Connect the release relationship to the next processing step and the wait relationship back to a holding loop or a monitored queue. Route any FlowFile that has waited too long to a dedicated failure path that alerts an operator instead of looping forever. Check: confirm every relationship on Wait has a connection or is auto-terminated deliberately, and that no relationship is left unrouted by accident.
Test with a matching signal first
Send one held FlowFile through Wait, then send the matching Notify signal, and confirm the held FlowFile leaves on the release relationship with its attributes intact. Repeat with the signal arriving before the data, if the source can produce that order, to confirm Wait still releases. Check: record the flow file IDs and provenance events for both cases so the result can be compared with later runs.
Test duplicate signals
Send the same release signal twice and observe how many FlowFiles are released. Confirm that duplicate signals do not release unrelated data or create unexpected cache entries, and decide whether the downstream step must tolerate repeated release. Check: compare the count of released FlowFiles with the count of expected events and document any difference as a defect or an accepted behavior.
Test missing and late signals
Hold a FlowFile without sending its signal and confirm that it reaches the timeout route, alerts the operator and does not stay in the wait loop indefinitely. Then send a late signal after the timeout and record what happens to the already-routed FlowFile. Check: verify the alert fires, the timed-out data has a documented recovery path, and the late signal does not silently release the wrong data. Confirm the timeout setting against the Wait component documentation for your NiFi 2 version before you rely on it.
Security and operational considerations for cache-backed signals
Wait and Notify keep their release signals in a distributed cache, so the cache becomes part of your security boundary and your recovery plan. The cards below cover what that cache holds, who can change it, how signal attributes expose data, and how to watch and restore the waiting flows.
What the cache holds
Notify writes a release signal keyed to a value you choose, and it can also store FlowFile attributes with that signal. Before you build the flow, list every attribute that would be copied and confirm the cache service stores it in memory or on disk. Treat the cache as a data store that holds business identifiers, not just a counter. Confirm the storage and expiry behavior of your chosen cache service in the NiFi administration guide.
Who can read and write signals
Anyone who can write to the cache can release a Wait processor early, and anyone who can read it can see what was signaled. Restrict write access to the flows that own the release logic, and keep read access to the same narrow group. Review which users and controller services can reach the cache service in the cluster, not only in the flow where you built it.
Sensitive attributes in signals
Copying attributes such as account numbers, patient identifiers or personal names into a signal spreads that data to every flow and person who can read the cache. Prefer a correlation key and a status value, and keep sensitive content in the FlowFile itself. If a signal must carry a sensitive value, document why, limit who can read it, and confirm how the cache service protects stored values.
Authorization through NiFi policies
Access to the processors and controller services that implement Wait and Notify is governed by NiFi authorization policies, not by the flow design alone. Grant the Notify and Wait components only to the process groups that need them, and separate operators who can start or stop them from those who can change their settings. Confirm the policy model in the NiFi administration guide before rollout.
Monitoring queue age in wait
FlowFiles held in the wait relationship are waiting on an event that may never arrive. Track how long the oldest FlowFile has been queued, and alert when that age passes the limit your business process allows. Without this check, a missing Notify shows up only when a downstream report is late. Set the threshold from your own process, since the outcome depends on your release and expiry design.
Backup and recovery of cache state
Cache contents are runtime state, so a cluster restart or cache loss can strand waiting FlowFiles or drop releases that already happened. Decide up front whether a lost signal means replaying the upstream event, re-running a batch, or manual intervention, and write that into the runbook. Ask your team to confirm how the cache and the flow repositories are backed up, and test a restore before go-live.
Questions teams ask before building Wait and Notify flows
What does a Wait and Notify build cost in effort, and what drives it up?
The effort depends less on the two processors than on how many flows need coordination, how many release signals each one depends on, and how much the team must test failure cases. Wait and Notify themselves are configured in a few properties, so the bulk of the work is in designing the release key, the attribute copying, the timeout routing and the tests that prove a stalled batch is caught. Costs rise when several flows share one cache, when signals come from systems that send duplicates or out of order, and when the target system must be checked for partial loads. A quote should list each flow in scope, the number of release signal types, the cache design, the testing approach and what the client provides. Ask whether the price covers documentation and handover, because those are one-time items that protect later changes. Recurring costs are mostly operations time for monitoring stuck waits.
How long does a pilot take, and what does its length depend on?
The length of a pilot depends on the number of flows that must wait on each other, the number of distinct release signals, and how quickly the team can produce realistic test data. A pilot usually covers one batch pattern end to end: a source that produces data, a Notify step that releases a signal, a Wait step that holds FlowFiles, and a timeout path that proves the stalled case is handled. Timelines grow when the upstream system cannot emit a reliable completion event, when access to the distributed cache must be arranged with the platform team, or when the target database has strict load order. Ask for the pilot scope in writing with named flows, the success checks and the decision point for moving to production. Treat any duration you are given as a figure to confirm against that written scope rather than a fixed promise.
What does the client need to provide for the work to go smoothly?
The client needs to provide three things before design starts: named owners for each flow involved, access to the distributed cache or the cache service the flows will use, and representative test data. Flow owners decide what a release means in business terms, such as a full day of files arriving or every fragment of an order being present. Cache access matters because Notify stores the release signal there, so the platform team must confirm the cache service is running in the target cluster and that the flows are allowed to use it. Test data should include normal batches, late fragments, duplicate signals and a batch that never completes. Without these, the team can only test the happy path, which is where coordination flows most often fail in production.
Can an internal NiFi team build Wait and Notify flows without outside help?
Yes, an internal team with working NiFi experience can build these flows, provided it has someone who understands the business meaning of each release signal and the time it takes data to arrive. The processors are documented in the NiFi components reference, and the hard parts are design decisions: choosing the release key, deciding what attributes Notify should store, and defining the timeout behavior. Teams often struggle with the cache lifecycle, with signals that arrive before the Wait step is ready, and with testing that simulates missing or repeated events. An outside partner can add value by reviewing the design, building test harnesses and writing runbooks, which is a fixed-scope engagement agreed in writing before work starts. The decision depends on your team's capacity and on how many flows will depend on this pattern.
When should a team choose an alternative to Wait and Notify?
Choose an alternative when the coordination logic is simple enough to live in one flow, or when the upstream system can provide the completion state directly. A common case is a single flow that receives all fragments itself, where a merge processor and a count check may be clearer than a cache-based release. Another is a batch that has a known schedule, where a scheduled check of a status table or a file marker can be easier to audit. Use a dedicated workflow or orchestration tool when the dependencies span many systems, need long-running state, or require human approvals. Compare the options on failure behavior, observability and who must maintain them. Document the reason for the choice, since auditors and future maintainers often ask why a pattern was selected.
How do you keep release signals reliable after go-live?
Reliability depends on monitoring the waiting FlowFiles, the cache contents and the timeout path, and on reviewing those signals when sources change. Set alerts on the number of FlowFiles sitting in the wait relationship and on the age of the oldest one, since growth there usually means a missing or misnamed release signal. Check that the cache service has capacity, backup and access controls, because a restart or full cache can silently drop signals. Change control matters too: when a source system adds fragments, changes its key format or starts sending duplicates, the release logic must be updated and retested. Assign an owner for the runbook, run periodic tests that send a known signal and confirm release, and record each change in your normal change process. Ask any partner to define these checks in the handover documentation.
How DeployOpen works on this
DeployOpen is a Chainscore Labs company that plans, implements, customizes, hardens, documents and supports self-hosted open-source platforms in infrastructure you control. For Wait and Notify coordination, we start from the business event that must release held data, then design the cache keys, signals and failure paths around it in your NiFi environment.
How we scope
- Written scope names the upstream flows, release events and downstream targets that depend on Wait and Notify coordination.
- We map each Wait release signal to a Notify source, a cache identifier and the FlowFile attributes it must carry.
- Scope records the expected fragment counts, timeout behavior and who owns the alert when a release never arrives.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before build work starts.
How we build
- We build the Notify and Wait flow pattern in a non-production NiFi instance that matches your target topology.
- Cache service choice follows your clustering design, and we confirm how release signals behave across cluster nodes.
- Test cases cover missing fragments, duplicate signals, late arrivals, timeouts and restarts of the flow.
- Flow definitions are versioned in your Git repository so changes to coordination logic go through review.
How we hand over
- Handover includes the flow documentation, cache and attribute naming conventions and the timeout runbook.
- Operators receive a walkthrough of how to inspect waiting FlowFiles and clear stale release signals safely.
- We document the alternatives considered for batch completion checks so future changes start from recorded decisions.
- Acceptance tests are run in your environment and results are delivered as part of the written deliverables.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, scoped separately from the build.
- Support can cover re-tests of coordination flows after NiFi upgrades, including changes to cache components.
- We can review waiting queue depth and timeout alerts with your operations team on an agreed cadence.
- Retainer scope, response expectations and service levels are confirmed in writing before the retainer starts.
