What happens to Apache NiFi when a queue grows past the swap threshold
When a connection's queue grows past its swap threshold, NiFi starts moving FlowFile objects out of the Java heap and onto disk. Swapping is the mechanism that keeps a backed-up queue from exhausting memory: NiFi writes the FlowFile attributes and metadata for the excess items to swap files, keeps only a small in-memory record of them, and reads them back when the queue drains. The content of each FlowFile lives in the content repository either way, so swapping protects the heap rather than moving data content. Operators need to understand this behavior when a queue is growing, heap usage is climbing, and they must decide whether the problem is the flow design or the sizing of the node. Confirm the default swap threshold for your version in the NiFi System Administrator's Guide before you rely on any figure, and check the threshold setting that applies to your connections in the nifi.properties file.
Swapping has costs that are easy to miss. Swapped FlowFiles require extra disk reads and writes, so throughput on a swapped connection usually falls, and the node spends more effort on I/O during a backlog. Swapping also changes the order in which FlowFiles are delivered. Prioritizers apply to the in-memory queue, so a connection that holds swapped items can deliver them in an order that differs from what the prioritizer would produce across the whole queue. If your flow depends on strict ordering, such as first-in-first-out handling of time-sensitive records, test the prioritizer behavior under a backlog before you go live.
Before you tune any setting, ask two questions. Why is the queue growing: is the downstream processor slower than the upstream rate, or is a destination unavailable? And is the swap threshold being hit regularly, or only during rare spikes? A queue that swaps every day points to a flow design or capacity problem, and tuning the heap alone will not fix it. Plan to watch queue sizes and swap-related status in your monitoring, and treat sustained swapping as a signal to revisit back pressure settings and processor concurrency, as covered in the sibling sections on back pressure and JVM heap tuning.
How queue behavior changes by flow situation
A short burst is the case swapping is designed to absorb. A source delivers more FlowFiles than downstream processors can take for a limited period, the connection passes its swap threshold, and NiFi writes the overflow to swap files in the FlowFile repository. Heap use stays comparatively flat because the swapped FlowFiles are held on disk, and the queue shrinks as the destination catches up. As the queue drains, swapped FlowFiles are loaded back in batches, so normal processing resumes without operator action. Read the swapping and prioritizer sections of the NiFi System Administrator's Guide for the exact mechanics in your version.
- Operator view: The connection's queued count rises during the burst and falls afterward, and the swapped portion is visible in the connection status while swapping is active. Confirm the labels in your version's interface.
- Ordering: Prioritizers may not hold strictly across swapped and in-memory FlowFiles, so do not assume strict ordering through a burst.
- Heap: Memory use stays predictable because the overflow sits on disk rather than in the JVM heap.
- Next step: Usually none beyond watching the drain. Confirm the queue returns to its normal level and that the content repository did not fill during the burst.
Check that the queue returns to its pre-burst level within the window your flow requires, and that prioritizer ordering is acceptable for the data involved.
A steady backlog happens when a downstream system accepts data more slowly than the flow produces it, for example a database under lock contention or an API with rate limits. The queue stays above the swap threshold for long periods, so NiFi keeps writing swap files and reading them back as processors take work. The swapped count stays high instead of falling, and throughput is set by the slowest destination rather than by the queue size. Once the connection reaches its back pressure thresholds, upstream processors pause, which slows the source. The guidance on back pressure and swapping in the NiFi System Administrator's Guide explains how these thresholds interact.
- Operator view: The queued count stays high across status history while the destination step processes less than its input step receives.
- Disk growth: Swap files and the content they point to accumulate on the FlowFile and content repository volumes, so watch free space on those disks.
- Upstream effect: Back pressure eventually pauses the upstream processors, which means the source slows down and the backlog stops growing.
- Next step: Use status history to find the slow step, then fix the destination capacity, its rate limits or the number of concurrent tasks before changing NiFi queue settings.
Confirm whether the destination can scale or whether the flow must accept a lower ingest rate, because no NiFi setting can make a slower system faster.
A queue that never drains usually has one of three causes: the arrival rate permanently exceeds processing capacity, a processor fails in a retry loop, or FlowFiles are penalized or held and never leave the connection. Swapped FlowFiles keep accumulating, so the FlowFile repository, the content repository and the heap all carry a growing load. Attributes that are large or numerous increase heap pressure, and a backlog on one node of a cluster can affect that node alone. Without intervention, the queue eventually reaches back pressure thresholds and stalls the source, or the disks fill. The NiFi System Administrator's Guide is the right reference for confirming how each setting behaves before you change it.
- Operator view: The queued and swapped counts keep rising across status history, and bulletins may show repeated errors from one processor.
- Heap and disk risk: Growth in swapped FlowFiles and in attribute size raises memory and disk pressure, so check these first on the node that holds the queue.
- Stuck versus slow: Separate a permanent capacity mismatch from a processor failing in a retry loop, because the fixes differ.
- Next step: Reduce the arrival rate, fix the failing step, or add expiration with prioritizers so stale FlowFiles leave the queue on purpose.
Before applying a change, confirm its effect on existing queued data with the vendor documentation for your version, and plan a test on a copy of the flow.
Queue settings that decide when swapping and back pressure start
This table compares the per-connection settings that decide when FlowFiles are swapped to disk and when upstream processors stop. Read each row as a question to answer for your flow: what it controls, when you would change it, and what it costs you. Default values differ between NiFi versions, so confirm them in the Administrator's Guide for your release before you change anything.
| Setting | What it controls | When to change it and trade-off |
|---|---|---|
| Swap threshold | Number of FlowFiles a connection keeps in heap before NiFi swaps the rest to disk in batches | Lower it when heap pressure from long queues is the problem; a lower value protects memory but adds disk reads and writes and can slow ordered queues |
| Back pressure object threshold | Number of FlowFiles allowed in a connection before the upstream processor is stopped from starting | Raise it to absorb short bursts; raising it too far lets queues grow past the swap point and makes ordering and recovery harder to reason about |
| Back pressure data size threshold | Total size of content queued in a connection before upstream processors stop | Set it below the content repository headroom for large files; a high value with large FlowFiles can fill the content repository before back pressure engages |
| Prioritizer | Order in which queued FlowFiles are pulled for the next processor | Use one only when order matters; a prioritizer forces NiFi to examine queued items, and ordering across swapped FlowFiles is not guaranteed to match the prioritizer exactly |
| FlowFile expiration | Maximum age of a queued FlowFile before NiFi removes it from the connection | Use it for data that loses value over time; expired FlowFiles are dropped, so confirm with the data owner that loss is acceptable and that events are logged |
| Connection thresholds across the flow | How each connection's limits combine with processor run schedules and destination speed | Check every connection on the path to a slow destination, because the first full queue upstream determines when the whole path slows |
Why prioritizers lose strict ordering once FlowFiles are swapped
A prioritizer on a NiFi connection orders the FlowFiles it holds, such as oldest first or by a priority attribute, but that order is only strictly enforced across the FlowFiles currently in memory. When a connection passes its swap threshold, NiFi writes surplus FlowFiles to disk as swap files and loads them back into the queue in blocks. A FlowFile in a swap file that holds an older timestamp can therefore wait behind newer FlowFiles that were already loaded. Treat prioritizer order as best effort across a swapped queue, not as a global guarantee.
This matters most for flows that depend on oldest-first processing, for SLA-sensitive records and for expiration. If a downstream step assumes records arrive in event-time order, out-of-order blocks can break deduplication windows, sequence checks or "latest state wins" merges. If you use FlowFile Expiration with a prioritizer, a record can sit in a swap file long enough to age past its limit while newer records are processed, so the expiration behavior you designed for may not match what happens under load. Check the swap threshold, the queue depth at which your connections swap, and how often a queue crosses it during peak hours.
Do not rely on documentation alone for this. Build a test on the NiFi version you run: fill a connection past its swap threshold with FlowFiles carrying known timestamps and priority attributes, drain it, and record the order and age of what leaves. Repeat the test after any NiFi upgrade, because the swapping and prioritizer code paths change between releases. Confirm the exact behavior with your NiFi release notes and the Apache NiFi System Administrator's Guide, and where ordering is a business requirement, design the flow to enforce it with timestamps and downstream checks instead of depending on the queue.
How to design flows that keep queues small
This sequence moves a NiFi flow from a queue that grows past its swap threshold to one that drains between bursts. Each step names the measurement or change that keeps FlowFiles out of swap and shows how to confirm the result.
Measure queue depth and destination speed
The flow owner records queue counts and sizes on each connection, then compares them with how fast the downstream processor or external system accepts FlowFiles. Use the status history and connection status in the NiFi UI over a representative period, including a peak. The step is done when you can name the connection that grows fastest and the destination that limits it. Keep the readings as the baseline for every later step.
Fix the slowest step first
Work on the processor or external call that limits throughput before changing anything upstream. Typical causes include a slow database query, a remote API with rate limits, or a processor waiting on a single thread. Confirm the cause with processor performance data or the destination's own logs, not by guessing. The step is done when the limiting step runs at the rate the upstream sources supply, or when you have documented the remaining gap for the client to accept.
Split work across parallel paths
Where the data can be divided safely, add concurrent tasks to the processor or route FlowFiles through parallel branches. In a cluster, load-balanced connections spread FlowFiles across nodes, but the flow must tolerate the resulting loss of ordering. Test that each branch returns the same results as the single path. The check is a test burst that shows the parallel path drains faster without duplicate or missing records.
Set back pressure per connection
Configure object and data size thresholds on each connection so that upstream processors stop scheduling when a queue fills, instead of letting it grow into swap and heap pressure. Choose thresholds that match the destination's capacity and the amount of memory available for the flow. Then verify that upstream processors show back pressure in the UI during a burst rather than the queue continuing to rise. This is confirmed when the queue stops at the threshold and the source resumes once it drains.
Route failures to separate retry queues
Send failure and retry relationships to their own connections with a clear retry policy, instead of looping them back into the main path. This keeps repeated failures from filling the queue that carries new data. Add penalties or scheduled retries where the destination needs recovery time. The check is that a deliberate destination outage builds up only the retry queue, while the main queue keeps draining.
Batch where the processor supports it
Use processors that merge or batch records, such as record-oriented writers or merge processors, when the destination accepts batches. Fewer, larger writes reduce per-FlowFile overhead and the number of FlowFiles waiting in queues. Confirm the batch size and timing against the destination's limits, and check each processor's documentation for supported options. The step is done when the destination write count falls and the queue depth for that connection stays lower under the same input.
Verify queue depth returns to baseline
Run a controlled test burst at the expected peak and record queue depth over time against the baseline from the first step. Confirm that queues drain to their baseline level after the burst, that no connection stays above its swap threshold, and that heap use stays steady. Keep the results with the flow documentation so that the next change can be compared with them. The sequence is complete when the burst test passes and the owner accepts the figures.
Operational risks when queues grow large
Large queues rarely fail in one dramatic step. Heap pressure, repository disk use, expiration and slow alerting build on each other, so a backlog that looks harmless on the canvas can exhaust memory or disk before anyone responds. Each card below names the signal to watch and the first action to take.
Heap exhaustion and OutOfMemoryError
Swapping protects the heap by moving FlowFile attributes out of memory, but large attribute maps, many queued FlowFiles and heavy swap-in activity still consume heap. The first signal is rising heap use that does not fall after queues drain, along with garbage collection pauses in the bootstrap log. Check which connections hold the most FlowFiles and which attributes are unusually large before changing any heap setting. Capture a heap dump only when your heap tuning allows it, since dumps are costly to write and read.
FlowFile repository disk contention
The FlowFile repository stores queue metadata and swap files, and it writes constantly while queues move. When it shares a volume with the content or provenance repositories, a large backlog increases write contention and can slow every flow on the node. Watch disk latency and I/O wait on the volume holding the FlowFile repository, and compare them with queue counts over the same period. The first action is to confirm that the FlowFile repository sits on its own fast volume before adding nodes or raising queue limits.
Content repository growth while data waits
Every FlowFile waiting in a queue keeps its content on disk, so a large backlog of big files can fill the content repository faster than normal traffic would. Growth shows up as rising content repository usage while the queued byte count climbs, and it can reach the point where upstream processors stop accepting data. Check queued size in bytes as well as FlowFile count, and review the archive settings that determine how long content stays. The first action is to find which connection holds the largest content, then decide whether to drain it or reduce its inflow.
Expiration dropping data silently
FlowFile expiration removes data from a queue once it passes a configured age, and the removal does not raise a bulletin or fail a processor. Flows that rely on expiration to clear old data can lose records during a long slowdown without any visible error at the time. Review each connection that has an expiration setting, and check how many FlowFiles expired in the status history for that connection. The first action is to confirm whether expired data must be retained, rerouted to a failure path or replayed from provenance before you change the timeout.
Delayed alerts on queue growth
Back pressure thresholds stop a connection from accepting more data, but the operator often learns about the backlog only after upstream flows slow down or stop. Without an alert on queue count, queued bytes or the age of the oldest FlowFile, a backlog can grow for hours before anyone looks. Configure alerting on the queue and back pressure metrics you already expose, and test that the alert reaches the team that owns the flow. The first action is to set queue depth alerts for each business-critical connection and review them against the time it takes your team to respond.
Questions teams ask about large queues and swapping in NiFi
Does swapping in NiFi mean FlowFiles are lost?
No. Swapping moves FlowFile metadata from heap memory to disk when a connection's queue passes its swap threshold, and NiFi reads those records back as the queue drains. This is what lets queues grow without exhausting the heap. Content for each FlowFile stays in the content repository either way, so swapping does not move the content itself. Data loss in NiFi usually comes from other causes, such as a repository disk that fills, or a flow that expires FlowFiles on purpose. Confirm the swap behavior for your exact NiFi version in the NiFi System Administrator's Guide, and test a restart with a loaded queue in a non-production environment before you rely on it.
How much effort does it take to tune NiFi queues?
The effort depends on how many connections are involved and how well the flow is understood. A single connection with a known backlog can often be diagnosed in a short review. A flow with many process groups, mixed data sizes and several slow destinations needs measurement across each connection, plus changes to prioritizers, back pressure thresholds and sometimes the flow shape itself. The work falls into three parts: finding the queues that grow and why, changing settings or redesigning steps, and proving the change under realistic volume. Ask for a written assessment that lists the connections in scope, the data profile of each, and the tests that will show each queue stays bounded.
Can our own team tune NiFi queues without a partner?
Often yes, if the team already runs NiFi in production and has engineers who read status history and bulletins with confidence. The work needs knowledge of the flow's data sizes, the capacity of each destination and the cluster topology, which in-house staff usually hold. Outside help tends to add the most value where the team has little recent experience with NiFi 2 changes, cannot safely run load tests against production, or wants an independent review before a change window. A practical test is whether your engineers can explain why each queue reaches its threshold and predict the effect of a proposed change. If they cannot, a short assessment before any redesign lowers the risk of tuning the wrong connection.
What does a NiFi queue tuning project need from our team?
Three inputs make the work practical: access to the flows, representative test data and queue metrics over time. Flow access means read access to the canvas, the flow definitions in version control, and the parameter values for each environment. Test data should match real sizes and burst patterns, including the largest attributes and files, because queue behavior follows content size as well as FlowFile count. Queue metrics should come from status history or your monitoring stack and cover queued count, queued size, back pressure events and swap-related bulletins across a normal period and a peak. You also need a named change owner who can approve setting changes, and a staging cluster and window where changes can be tested.
How long does a NiFi queue tuning cycle take?
The cycle length depends on flow count, environment access and change control, and no fixed duration applies to every engagement. Each step adds calendar time: collecting metrics over a representative period, reviewing the flows, testing changes on a non-production cluster and then passing your change request process. Formal approval boards and freeze periods often add the most elapsed time, because testing can finish while a change waits for a window. Ask for a schedule in the written scope with named milestones, the number of flows and environments included, and the approvals your organization must give. Timelines are agreed in writing before work starts, and any scope change should be recorded in writing as well.
What drives the cost of a NiFi queue tuning quote?
Cost follows the size and risk of the scope. The main drivers are the number of flows and connections to review, the number of environments that must be tested, the complexity of data formats and destinations, whether flow redesign is included or only setting changes, how much metric history must be analyzed, and how many change windows the work must fit. One-time items include the initial assessment and any redesign or test harness. Recurring costs appear only if you add ongoing support or periodic reviews under a retainer. Ask that each quote itemize these drivers, state what is excluded, and name who runs the tests and who receives the handover. DeployOpen does not publish a price list, and a quote follows from the written scope.
Does swapping slow down NiFi throughput?
It can. Reading swapped FlowFiles back from disk adds I/O work, and that work competes with content repository reads and writes on the same storage. The effect usually appears as a slower drain of a deep queue and higher disk latency, not as a complete stop. Flows that keep queues below the swap threshold avoid this cost. Common design fixes include controlling burst concurrency, moving slow calls into separate process groups, placing repositories on dedicated disks and setting expirations where the business allows them. Confirm the effect on your version by comparing queue drain time with and without a realistic load.
Does swapping change the order of FlowFiles in a queue?
It can. Prioritizers set the order of FlowFiles in a connection, but once a queue has swapped, NiFi reloads FlowFiles from disk in batches, so strict prioritizer order may not hold across the whole queue. Teams that depend on strict ordering, such as by event time or sequence number, should test this explicitly and should not assume it. Options include keeping queues below the swap threshold for those connections, carrying the ordering key in attributes, and handling out-of-order arrivals downstream. The NiFi System Administrator's Guide describes swapping and prioritizers. Confirm the exact guarantee for your release before you design any ordering-sensitive logic.
How DeployOpen scopes queue and swap tuning
Queue and swap work starts as a written, fixed-scope engagement that names the flows, connections and environments in scope before any change is made. DeployOpen works inside infrastructure you control, whether that is your cloud accounts, your data centers or an isolated network, and the scope document records which NiFi clusters, which process groups and which back pressure and prioritizer settings are reviewed. Deliverables are agreed in writing with the timeline, so you know what you will receive: a queue review, a set of flow redesign recommendations or changes, and a validation report that shows queue depths and swap activity before and after each change.
The engagement needs inputs from your side, and the quality of the review depends on them. Provide the exported flow definitions or access to the canvas, the NiFi status history and bulletin output for the period when queues grew, and any metrics your team already collects from the JVM and operating system. Also provide test windows on a non-production cluster that carries representative data volumes, because swapping behavior under real prioritizer and expiration settings cannot be judged from the canvas alone. Confirm with your platform team who can approve a restart or a configuration change during those windows.
Ongoing tuning is available as a retainer for teams whose flows change often or whose volumes grow over time. A retainer covers periodic queue reviews, re-validation after flow changes and support for new connections, and its scope, response expectations and hours are agreed in the same written form as a fixed-scope engagement. Ask for the retainer terms, the exact deliverables for each review and the test environment that will be used before you compare options, since those items drive most of the recurring effort.
