When the Apache NiFi Stateless execution engine fits your flow
Use the Stateless engine when a team consumes from a transactional source and needs the whole flow to succeed or fail as one unit. A process group on the Stateless Execution Engine behaves more like a single processor: each concurrent task runs the whole flow to completion before it brings in more data. The NiFi User Guide usually prefers Stateless when three conditions hold together: the source is transactional (JMS, Kafka, Amazon SQS, Kinesis or HTTP), the flow has one destination, and the whole flow should commit or roll back as one transaction. A Kafka consumer that writes to a single object store, for example, fits this pattern well. Confirm the source's acknowledgment behavior with your platform team before you design around it, because the engine's transaction boundary has to line up with when the source considers a message delivered.
The traditional engine remains the better default when a flow has several destinations, when it relies on queues that must survive a restart, or when it needs long-running buffering between stages. Stateless does not persist data queued inside the group, so data in process is lost on restart and the source must re-send it. Before you choose, list every queue in the candidate flow and ask whether losing its contents on a restart is acceptable. Also check whether the flow fans out to more than one destination, since that pushes the design back toward the traditional engine. Treat the choice as a per-process-group decision, not a cluster-wide setting.
NiFi Stateless is also published as a separate binary download alongside NiFi Standard and the NiFi Toolkit, so teams can run Stateless flows outside the main NiFi installation. Check the NiFi downloads page for the binary that matches your target release before you plan packaging. Whether you run the binary or embed the group in a standard cluster, your team should confirm which Java runtime and NiFi version its flows are built against, and record that choice with the flow definition so that upgrades and rollbacks are predictable.
Stateless or traditional engine: which option fits each flow
This table compares the Stateless Execution Engine with the traditional engine across the factors that decide the choice: source type, destination count, persistence, transaction scope and failure handling. Read each row against the flow you plan to build, and treat the confirm column as the checklist to close before you commit to an engine.
| Factor | Stateless engine | Traditional engine |
|---|---|---|
| Source type | Transactional sources such as JMS, Kafka, Amazon SQS, Kinesis or HTTP, where the source can redeliver unacknowledged messages | Any source, including file, database and polling sources where the flow controls its own offsets and retries |
| Destination count | One destination, so the whole flow succeeds or fails as a single unit | Multiple destinations, routing and fan-out, where each branch can succeed or fail independently |
| Persistence across restarts | Data in process is not persisted, so the source must re-send anything in flight after a restart | Queued FlowFiles persist in the repositories and resume after a restart |
| Transaction scope | Each concurrent task runs the whole flow to completion before taking more input, which behaves like a single processor | Each processor works on FlowFiles individually, with queues buffering work between processors |
| Failure ports | Confirm which failure relationship the group exposes and how the source acknowledgment is tied to it before relying on it | Failure relationships route individual FlowFiles to retry, dead-letter or error handling paths inside the flow |
| Timeouts | Confirm the timeout behavior for the group and the source, because a slow destination can hold a whole transaction open | Timeouts apply per processor and per operation, so a slow step affects only that processor's queue |
| Merging behavior | Confirm which merge and batching processors are supported inside a Stateless group before designing around them | Merge processors work across queued FlowFiles, with backpressure and provenance on each connection |
| Confirm before choosing | Source redelivery semantics, acceptable replay of in-flight data after restart, and whether the destination is idempotent | Queue sizing, repository storage and how much work a restart may re-process |
Failure ports, timeouts and merging caveats to design for
Stateless groups change how failures, slow runs and merges behave, because the whole flow runs as one unit per task. The cards below cover the decisions that determine whether a single bad run stays contained or stalls the source. Confirm each behavior in the User Guide and by testing your own flow before you rely on it.
Routing failed transactions to failure ports
Within a Stateless group, a failure in any part of the flow fails the transaction as a whole, and the group's failure port is where the affected data is sent instead of completing. Decide which failures should fail the whole transaction, which should be routed to a dead-letter destination inside the group, and which should be handled upstream. Check the exact port behavior in the User Guide and by forcing a failure in a test flow.
Timeouts on long-running runs
A Stateless run that exceeds its configured timeout is treated as a failure, so a slow downstream system can turn healthy throughput into repeated failed transactions. Set the timeout from measured run durations under load, not from a guess, and confirm how the timeout interacts with the source's own acknowledgment or visibility window. Check in a test flow that a timed-out run is redelivered as you expect.
Merging processors inside a group
Merge processors hold FlowFiles until a bin is complete, which conflicts with the Stateless model of running one transaction to completion before taking more input. A merge that waits on data from a later run can stall the group or hold data that is not persisted across restarts. Confirm how your merge settings behave in the User Guide and test a partial batch before moving the design to production.
Sizing work so one run cannot block
Each concurrent task runs the whole flow to completion, so one stuck or poisoned record can hold a task and reduce throughput for the rest of the source. Keep each transaction small enough that a failure affects a bounded amount of data, and set concurrency with downstream capacity in mind. Test with a deliberately failing record to confirm the remaining tasks keep consuming.
Dead-letter handling for poisoned messages
A message that fails every time will be retried indefinitely unless the design stops it. Decide where repeated failures go, such as a separate topic, queue or storage location, and how an operator inspects and replays them. Confirm with your source's redelivery settings that the poisoned message is not returned to the group forever, and test that the dead-letter path keeps the message.
Set up and verify a process group on the Stateless engine
This sequence takes one candidate flow from a design check through conversion, timeout and failure configuration, and a forced failure with a restart, so you can confirm that re-sent data reaches the destination exactly once under your design before the group carries production traffic.
Confirm the source and destination fit
Before touching NiFi, the flow owner and the integration engineer confirm that the source is transactional, such as JMS, Kafka, Amazon SQS, Kinesis or HTTP, that the flow has one destination, and that the whole flow should succeed or fail as one transaction. If the source is a plain directory or any source that cannot re-deliver data, the Stateless engine is the wrong choice. Done means a written note names the source, the destination and the failure behavior the business expects, and the team agrees to proceed.
Build and test the flow on the traditional engine
The integration engineer builds the processors, controller services and record handling in a normal process group on the traditional engine first, which is usually easier to inspect while the logic is still changing. Test with representative sample data and confirm the expected output at the destination. Done means the flow produces correct output on sample data with no unexpected queueing. Once it does, the logic is ready to move into a process group that will run on the Stateless engine.
Convert the process group to Stateless
Create or select the process group that will run the flow, then set its execution mode to Stateless in the group's configuration. Move or copy the verified processors into that group and connect the source processor to the destination. Confirm that the group contains only the flow for this one source and one destination, because a single Stateless group should not mix unrelated paths. Done means the group shows the Stateless execution mode and the flow starts with no validation errors on its components.
Set the timeout and failure handling
Set the group's timeout so a slow destination cannot hold a transaction open indefinitely, and route the failure ports to a defined handling path, such as a retry branch or a dead-letter destination that your team monitors. Use the Stateless engine's documented failure port behavior to decide what happens when any step fails. Confirm the timeout value against the NiFi User Guide and your destination's expected response times. Done means every failure path has a named owner and the timeout value is recorded with the reason it was chosen.
Run a forced failure test
The test engineer causes a controlled failure, for example by making the destination reject requests or by stopping a dependent service during a test run, while the source holds known test messages. Check that the failed transaction is rolled back, that the source keeps the messages available, and that the failure path receives the expected records. Done means a written test record shows the input count, the failure route count and the destination count, with no partial writes at the destination.
Restart and confirm exactly-once delivery
Restart the NiFi node or the group while test messages are in process, then let the source re-send them. Because data queued inside a Stateless group is not persisted, the source must re-deliver everything that was in flight. Compare the source offsets or message IDs with the destination records to confirm that each message arrives once. Note that exactly-once results depend on your destination's idempotency and your design, so confirm the behavior with your source and destination documentation. Done means the reconciliation report shows no missing and no duplicate records, or each duplicate has a documented handling rule.
Document the design and hand over
The owner records the source, destination, timeout, failure routes, restart test result and the reconciliation method in the flow's runbook, and stores the flow version in your change process. The operations team confirms that monitoring covers the failure routes and the destination. Done means the runbook is approved by the flow owner and operations, and the test evidence is filed with the change record for later review.
Security and operational considerations for Stateless flows
A Stateless process group changes where credentials live, what evidence survives a failure, and how operators respond when something stops. The cards below cover the decisions to settle before a Stateless flow reaches production, and they assume the source and destination are transactional.
Credentials and parameters for connections
Source and destination credentials, such as Kafka client credentials, JMS connection factory logins or SQS access settings, should be held in sensitive parameters rather than typed into processor properties. Confirm which parameter context the Stateless group reads at start, because a value changed in the context after deployment may not reach a running group. Ask the team that owns each connection to name who may rotate its secret and how a rotation is rolled into the flow without editing it by hand.
Authorization on the process group
Decide who may start, stop, edit and read the Stateless process group before it is deployed, and keep those rights separate from the rights used to operate the traditional flows around it. Review which users can view component properties, since those views can expose endpoints and usernames even when secrets are masked. Document the group owner so that a change request for a transaction source or destination has a named approver.
Logging and provenance without persisted data
Because data in process is not persisted, a failed transaction cannot be replayed from NiFi's own queues after a restart. Decide what evidence you need instead: correlation identifiers in message headers, failure reports written to a log or audit store, and retention of source offsets or message identifiers in the source system. Confirm with your auditors whether this evidence satisfies your data handling record requirements before go-live.
Monitoring failed transactions and timeouts
Alert on failure-port traffic and on transactions that run past the time the source allows before it redelivers. Track how often the source redelivers the same message, since repeated redelivery can point to a destination outage or a flow that rejects valid records. Set alert thresholds with the team that owns the destination, and test them by forcing a failure in a non-production environment.
Restart and recovery runbooks
A restart of a Stateless group means in-process data is gone, so the runbook must tell operators to leave recovery to the source, then confirm that the source redelivers messages from the last committed position. Write separate steps for a planned stop, a node failure and a destination outage, and state who decides whether to pause the source. Rehearse the runbook with the on-call team, because the first time it is needed should not be the first time it is read.
Destination idempotency and duplicates
A transaction that fails after a destination accepts a write can be redelivered by the source, which can create duplicates. Check whether each destination supports idempotent writes or keyed upserts, and record that decision in the flow design. Ask the destination owner to confirm how they detect duplicates, because NiFi will not do it for them.
Questions teams ask before running Stateless flows
What does the work involve when moving a flow onto the Stateless engine?
The work is a per-flow design and migration exercise: reviewing each candidate process group, confirming its source and destination behavior, rebuilding the group as a Stateless process group, and testing failure handling end to end. Effort rises with the number of flows, the number of sources and destinations per flow, and how much custom code or scripted logic sits inside each group. Flows that already use transactional sources and have one destination are usually the simplest to move, while flows that fan out to several destinations or hold data across restarts need a redesign before they are suitable.
Can our own team run Stateless flows without outside help?
Yes, a team with working NiFi experience can run Stateless flows, but the design decisions need to be made deliberately. The NiFi User Guide describes the Stateless engine as the usual choice when the source is transactional, there is one destination, and the whole flow should succeed or fail as one transaction. Your team will need to judge whether each flow meets those conditions, handle data lost on restart by making sources re-send it, and set timeouts and failure ports correctly. An outside partner can help with that review and with testing, which is often the part teams have the least time for.
What do we need to provide: source owners, test data and change windows?
You need to provide access and people, not only infrastructure. Source owners for each Kafka topic, JMS queue or SQS queue must confirm how acknowledgement, redelivery and retention behave, because those settings decide what a failed transaction means in practice. Test data should include malformed records, duplicates and large messages so failure ports and timeouts can be exercised. Change windows matter because flows are switched from the traditional engine to the Stateless engine in production, and you should agree in advance how in-flight data will be drained or re-sent. Ask your team to confirm which of these owners are available and when.
How long does a Stateless engine project take?
Duration depends mainly on the number of flows, how many of them need redesign rather than a straight move, and how deep the testing must go. A small set of simple, transactional flows with one destination takes far less time than a mixed estate with merging, long-running processors and several destinations. Testing depth also adds time, particularly when failure ports, timeouts and restart behavior must be proven under load. No duration is given here because it should follow from a written scope, and you should ask any provider to show how the schedule maps to the flows and tests in that scope.
What should a fixed-scope quote for this work list?
A fixed-scope quote should list each process group in scope with its source, destination and proposed engine, so it is clear which flows are moved and which stay on the traditional engine. It should name the deliverables, such as redesigned flows, failure port handling, timeout settings, test results and a runbook for operators. It should state what the client provides, including source owner contacts, test data and change windows, and what happens if a flow turns out to need a different design. Ask for exclusions in writing, for example flows with merging logic or custom processors that are not included. Most engagements of this kind are fixed-scope, so the agreed scope should be documented before work starts.
What happens to data in a Stateless flow during a restart?
Data queued inside a Stateless process group is not persisted, so anything in process when the node restarts is lost and must be re-sent by the source. This is why transactional sources matter: Kafka, JMS, Amazon SQS, Kinesis and HTTP sources can redeliver or retry messages whose processing did not complete. Teams should confirm, per source, that redelivery is configured and that duplicates can be handled downstream. If a source cannot re-send, the flow is usually a poor fit for the Stateless engine, and the traditional engine with persisted repositories should be kept.
Which Stateless engine limits should we verify before going live?
Verify the behaviors the NiFi User Guide describes for the Stateless engine rather than assuming the traditional engine works the same way. The main points to confirm are that each concurrent task runs the whole flow to completion before more data is brought in, that failure ports route data as you expect, and how timeouts behave for long-running steps. Also confirm how merging components behave inside a Stateless group, since that is an area where designs often need adjustment. Check the current NiFi User Guide and your own test results, because the behavior of individual components should be confirmed against the version you run.
What changes when you move an existing flow to Stateless
Start by sorting every flow you plan to move into two groups: flows whose source is transactional, such as JMS, Kafka, Amazon SQS, Kinesis or HTTP, and flows that deliver to exactly one destination. The NiFi User Guide points to these two traits as the usual reasons to choose the Stateless engine, and a flow that has neither is a weak candidate. For each candidate, list the processors that fan out to several destinations, the Wait and Notify pattern if it is in use, and any step that depends on FlowFiles waiting in a queue for a long time. Those steps need redesign or should stay on the traditional engine, and the process group is the right unit to decide on.
The change that most affects operations is persistence. Data queued inside a Stateless group is not kept across a restart, so anything in process when the group stops must be re-sent by the source. Review your current runbooks for this. If today's recovery step is to restart NiFi and let queued FlowFiles drain, that step no longer protects your data for a Stateless group. Confirm that each source can replay from the last acknowledged position and that your team knows how long a replay window the source retains. Also confirm how failure ports and timeouts route work that does not complete, because those settings replace some of the queue-based retries you may rely on now.
Run the moved flow in parallel with the original before cutting over. Send a copy of real traffic to the Stateless group, compare destination output against the traditional flow, and watch for duplicates, which a transactional source makes easy to create during a switch. Keep the parallel run long enough to cover a restart and a source outage. Version-specific behavior matters here: NiFi 2 is the current major line, and the Stateless engine's configuration, component names and defaults should be checked against the NiFi migration guidance for the exact release you run before you finalize the cutover plan.
How DeployOpen scopes and delivers Stateless NiFi flows
DeployOpen plans, builds, tests and documents Stateless NiFi process groups inside infrastructure you control. We start with a written scope that decides which flows run on the Stateless engine and which stay on the traditional engine, then deliver the flows with the failure, timeout and restart behavior recorded for your team.
How we scope
- Inventory each candidate flow with its source type, destination count and transaction boundary.
- Mark which flows need restart tolerance, since Stateless data in process is lost on restart.
- Agree in writing which process groups are in scope, with fixed deliverables and timelines.
How we build
- Design each Stateless group around one destination and a single success or failure outcome.
- Wire failure ports, timeouts and upstream acknowledgment so rejected data returns to the source.
- Test restarts, broker redelivery and merging behavior against the sources your flows consume.
How we hand over
- Document each group's engine choice, failure routing, timeout settings and re-send expectations.
- Store flow definitions in version control and walk your operators through promotion steps.
- Run a handover session covering restart procedures so your team can operate the flows alone.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after handover.
- Review engine choices again when sources, destinations or NiFi versions change.
- Ask for a quote built from your written scope, since DeployOpen does not publish prices.
