NiFi content repository archiving and disk usage: what it controls and when you need to tune it
Archive settings decide how long NiFi keeps the content of processed FlowFiles for provenance replay and how much of the content repository disk the archive may use. Back pressure decides when NiFi stops accepting new data because the content repository is nearly full. If a node's content repository disk keeps filling, or a provenance replay returns no content, one of these settings is usually the cause. The archive and back pressure behavior is described in the NiFi System Administrator's Guide, which is the document to check before changing any value on a running node.
Teams usually need to tune these settings in three situations. The first is flows that move large files, because each archived claim can consume disk quickly and the archive reaches its limit sooner than planned. The second is a long replay window, where investigators or auditors expect to open content from events days or weeks old, a requirement the disk must be sized for from the start. The third is a shared disk, where the content repository competes with the FlowFile, provenance and status repositories for space, so an archive limit that looks safe on its own can still lead to back pressure. Replay is only possible while the archived content still exists, so the retention period also determines what replay can show.
Before changing values, confirm three things in your environment: how much disk the content repository has on its own volume, the daily growth of archived content under your actual data rates, and the replay window your operations or compliance team needs. Then set the retention period and archive usage percentage together so the archive is trimmed before the disk reaches the back pressure percentage, and check the result under a sustained load test. Record the values, the NiFi version and the reason for each change, and review the same settings on every node in a cluster. Confirm the exact property names and defaults for your NiFi version in the Administrator's Guide before any production change.
How the content archive, usage limits and back pressure interact
The archive retention period sets how long NiFi keeps content from FlowFiles after they leave their queues and finish processing. It is the time limit on the archive. Archived content inside this window can still be shown in provenance replay, so the period determines how far back an operator can reconstruct what a FlowFile contained.
- Replay horizon: Content older than the window is removed, so a provenance event can remain while its content no longer can be replayed.
- Compliance evidence: If auditors need payloads from a given period, the retention period must cover that period, and archiving must stay enabled for those flows.
- Storage growth: A longer period keeps more archived data on the content repository disk at the same data rate.
- Separate provenance retention: Provenance metadata has its own retention settings, so an event can outlast its archived content. Check both values together.
Confirm the default value and the unit syntax for your version in the NiFi System Administrator's Guide before you set a period.
The archive usage percentage caps how much of the content repository disk archived content may occupy. When archived content passes this share, NiFi removes the oldest archived content first, even if the retention period has not expired. It is the space limit that the time limit cannot enforce on its own.
- Oldest first: Cleanup targets the oldest archived content, so replay coverage shrinks from the far end of the retention window.
- Shared volumes: When the repository shares a volume with other data, other writers consume room that the archive limit does not control. Confirm how usage is measured in your layout.
- Active content is untouched: Cleanup removes archived content only. Content still held by queued FlowFiles is active, is never archived and is not removed by this setting.
- Lowering the value: A lower limit can trigger removal of replay content soon after the change, so schedule the change with the teams that rely on replay.
Confirm the default value and how cleanup is timed in the administration guide for your version, because the exact behavior affects how much replay history you can count on.
The back pressure percentage is the content repository fill level at which NiFi stops accepting new content. When the repository reaches it, processors that would write new content are blocked until space is freed. This protects the disk from filling completely, but it slows the flow whenever the threshold is reached.
- Writes pause, drains continue: New content writes wait for space, while queued FlowFiles can still move through the flow as space is released.
- Archive counts toward the total: Archived content occupies the same disk, so a high archive usage limit can push the repository toward this threshold.
- Visible symptoms: Upstream queues grow and connections display back pressure. Teams often mistake this for a slow downstream system, so check repository fill level first.
- Headroom for in-flight data: Keep the threshold below the point where the disk would fill, with room for content that is already in flight.
Confirm the setting name, its default and whether it is measured against the disk or the repository in the administration guide for your version.
The three controls act in sequence on the same disk. Retention and the archive usage percentage decide how much archived content survives. The back pressure percentage decides when new writes stop, and it counts archived content along with active content. The practical design is to make archive cleanup finish before writes block, so the archive yields space first.
- Ordering: Set the archive usage limit below the back pressure threshold so cleanup can free space before writes are blocked.
- Illustrative example: If the archive limit were set close to the back pressure threshold, archived content could occupy nearly all the headroom, and writes could block while the archive is working as configured.
- Large files: Flows that move large files consume headroom quickly, so the gap between the archive limit and the back pressure threshold must cover the largest expected volume in flight.
- Controlled testing: Change one value at a time in a non-production cluster and watch disk use, back pressure events and provenance replay before applying the change elsewhere.
Before you set any value, confirm the cleanup timing and the comparison rules in the administration guide for your NiFi version, and record the values your flows were tested with.
Content repository settings and when each one fits
This table lists the nifi.properties keys that control content archiving and back pressure on the content repository, with what each one controls and the trade-off of changing it. Read each row alongside the workload it protects, because archive retention decides how far back provenance replay can reach, while the usage thresholds decide when NiFi stops accepting new data.
| Setting | What it controls | Trade-off when raising or lowering it |
|---|---|---|
| nifi.content.repository.archive.enabled | Whether processed content claims are kept in the archive directory after the FlowFiles that used them are gone, which makes provenance replay of content possible. | Turning archiving off frees disk and I/O for busy flows but removes the ability to replay content from provenance events. Turning it on requires disk capacity sized for retained content. |
| nifi.content.repository.archive.maxRetentionPeriod | How long archived content is kept before it is eligible for removal. | A longer period extends replay and investigation windows but grows disk use. A shorter period protects disk but may leave older provenance events without replayable content. |
| nifi.content.repository.archive.maxUsagePercentage | The share of the content repository disk that archived content may occupy before the oldest archive entries are deleted early. | A higher value keeps more history but leaves less headroom for active flow content. A lower value protects active processing at the cost of shorter replay history. |
| nifi.content.repository.archive.backpressure.percentage (back pressure usage percentage) | The content repository disk usage level at which NiFi stops accepting new data and applies back pressure to upstream processing. | A higher value uses more disk before writes block, which risks a full disk. A lower value blocks writes earlier and slows intake, which protects the disk but can reduce throughput. |
| nifi.content.repository.directory.* (content repository directories) | The locations where content claims are written, including whether more than one directory is listed for parallel disk use. | Separate fast volumes spread I/O and isolate growth, but each directory must be sized and monitored. Sharing a disk with other repositories makes a full content disk affect the whole node. |
| Large-file flows (archive sizing against replay needs) | Large objects consume content claims quickly, so retention and usage limits that suit small records can fill the disk within a short time. | Leaving archive settings at defaults can exhaust disk during bursts of large files. Size retention from how far back replay is needed and from the data volume per day, and test a burst before go-live. |
| Defaults and values for the target version | The shipped defaults for each key and their exact names for your NiFi release. | Confirm each key and default in the NiFi System Administrator's Guide for your version before changing production files, since key names and behavior can change between 1.x and 2.x. |
How to set archive retention and disk thresholds safely
This sequence sets archive retention and usage limits from measured data rates and agreed replay needs, then checks that the content repository keeps enough headroom before back pressure blocks writes. Each step ends with a check you can repeat after every change.
Measure data rate and content size per flow
The platform engineer records how much data each flow ingests per hour and the average size of its FlowFile content, using the status history and connection metrics in the NiFi UI. Multiply the rate by the retention period you expect to need to estimate archive growth per flow group. Check that the totals are written down per flow group and per node, so you can compare them against free space on the content repository volume in the next steps.
Agree the replay window with business owners
Work with the owners of each flow to decide how far back provenance replay must work, such as whether an operator needs to re-send yesterday's files or last month's. Record the answer per flow group, including any flows that carry large files and may need a shorter window. Ask owners to confirm in writing whether content older than the window can be lost. The step is done when every flow group has a stated replay window that the platform team can translate into a retention setting.
Place the content repository on a dedicated volume
The platform team mounts the content repository on its own filesystem, separate from the FlowFile, provenance and status repositories and from the operating system disk. This keeps archive growth from starving other repositories. Confirm the layout on each node with a command such as
dfagainst the configured content repository path and check that it reports a separate device. The step is done when the output shows the content path on a volume that holds no other NiFi repository.Set retention and usage limits in nifi.properties
The platform engineer edits
nifi.propertieson each node to set the archive retention period throughnifi.content.repository.archive.max.retention.periodand the archive usage ceiling throughnifi.content.repository.archive.max.usage.percentage. Use the values agreed in the earlier steps, and keep the same values on every node so the cluster behaves consistently. Confirm the exact property names and their allowed formats for your NiFi version in the administration guide before you change them. The step is done when a configuration review shows identical values on all nodes.Leave headroom below the back pressure threshold
The platform engineer compares the archive usage ceiling with the disk level at which NiFi applies back pressure to content writes, and confirms that the archive ceiling sits clearly below it. The gap is there so archive cleanup can run before writes stop. Check the back pressure setting for your version in the administration guide, and write the gap, in percentage points and in gigabytes of free space, into the change record. The step is done when the record shows the archive ceiling, the back pressure point and the free space left between them.
Restart and verify in the UI and disk monitoring
The operator restarts the nodes using the change process your organization already follows, one node at a time where the cluster allows it. After each restart, open the NiFi UI and check the System Diagnostics view for content repository storage usage on every node. Confirm that disk monitoring, whether it is your existing host agent or a dashboard, reports the same volume and shows the usage that the UI reports. The step is done when each node shows the expected settings and the monitoring alert thresholds match the headroom recorded earlier.
Test a provenance replay before go-live
The operator picks a sample FlowFile from each flow group that falls inside the agreed replay window, then replays it from the provenance view and confirms that its content downloads and the replayed data reaches the destination. Repeat the test on a FlowFile just outside the window to confirm that its content is no longer available, as the owners expected. Record the FlowFile identifiers, the results and the owners who signed off. The step is done when every flow group has a passing in-window replay and a documented out-of-window result.
Settings for flows that move large files
Large files fill the content repository faster than the archive can recover because archived content is only reclaimed when the archive limit is reached, and every FlowFile that holds a large payload keeps its content claim alive while it sits in a queue or waits for a slow destination. A single multi-gigabyte file moving through a connection can therefore push the content repository disk past the back pressure threshold on its own, even when the average data rate looks modest. Once the disk crosses that threshold, NiFi stops accepting new content into the affected connections, so the first symptom is usually a stalled upstream processor rather than a full-disk alert. Before you size the volume, confirm the largest single file the flow can receive, how long a typical file waits in queues, and whether any destination can hold files for hours during an outage.
The most effective flow-side levers are to reduce how much content NiFi stores and how long each claim survives. Splitting large files early, for example with SplitRecord for record-oriented formats or SplitContent for byte-delimited data, lets downstream steps release smaller claims as each piece completes instead of holding one large claim until the whole file finishes. Drop or strip content you do not need as early as possible: use a RouteOnAttribute step to discard files that fail validation before they reach a transform, and avoid copying large payloads into FlowFile attributes, which increases heap pressure. For high-volume large-file flows, consider a separate content repository directory or a dedicated volume so that these flows cannot starve the rest of the instance, and confirm the layout with the repository sizing topic in this group.
Connection back pressure and expiration interact directly with content retention, so review them together with the content repository settings. A connection that holds large FlowFiles keeps their content claims until the FlowFiles leave the queue, which means a generous object threshold on a slow connection can hold more archived and active content than the disk was planned for. Expiration with prioritizers can release stale large files, but it also discards data, so agree the expiration policy with the data owner first. The threshold values, prioritizer choices and expiration settings are covered in the back pressure tuning topic, so use that page for the actual values and treat this section as the checklist for what the large-file flow must allow.
Security and operational risks of archived content
Archived content in the NiFi content repository keeps payloads long after the flow has finished with them. The risks below connect data protection, replay access, disk capacity, retention and backup planning, and each one needs an owner before the archive is switched on.
Archives carry the same sensitive data
Content archives hold the full payloads that passed through each connection, including personal, financial or health data that may never have been meant to persist. Treat the archive directory as an extension of the active data store. Confirm who can read the volume at the operating system level, whether the disk is encrypted at rest, and which backup and snapshot jobs copy it. Access controls set in the NiFi UI do not protect files read directly from disk.
Provenance replay exposes archived content
Provenance replay lets users with the right policies download or view the content of a past FlowFile, which in practice reaches into the archive. Review which roles can view provenance data and content, and align those policies with how the flow's data is classified. A user who can see the event list may not need the payload, so separate those permissions where your authorization model allows it. Test replay with a non-administrator account before go-live.
Full disks halt ingestion
When the repository volume fills, NiFi applies back pressure to upstream connections and stops accepting new data, so ingestion stalls even when processors are healthy. The archive usage percentage and the back pressure percentage determine how much headroom exists before that happens. Size the volume for the archive's peak, not its average, and confirm how your team would respond if the threshold is reached at night or on a weekend.
Retention must follow policy
The retention period sets how long archived content and provenance remain available, and it should match the retention schedule your data governance team has approved rather than a default chosen during installation. Shorter retention reduces exposure and disk growth, but it also limits how far back an investigation can replay a FlowFile. Document the chosen period, who approved it, and how deletion is verified on disk.
Monitor disk usage and threshold events
Disk usage on every content repository directory needs continuous monitoring with alerts that fire well before back pressure engages. Track the fill level of each volume, the archive usage trend, and any NiFi bulletins about repository thresholds. Define who receives each alert and what the first response is, such as pausing a high-volume source or shortening retention for a non-critical flow. Check that alerts still fire when a repository sits on a shared disk.
Archives are not backups
Archived content exists to support replay and short-term investigation, not to recover data after corruption, deletion or a failed node. A backup strategy has to name the data it protects, the recovery point it supports and where copies are kept, and the archive rarely meets those requirements on its own. Confirm with your backup owner whether flow definitions, parameter contexts, the flow configuration file and the repositories are in scope, and test a restore rather than assuming one.
Questions teams ask about NiFi content archive sizing
What drives the cost of tuning content archiving in NiFi?
The cost follows the size of the flow inventory and how many of those flows carry large or high-volume content. Each flow needs its data rate, average and largest file sizes, and the replay window the business actually requires before anyone can set a retention period or archive percentage with confidence. Costs recur when data volumes or flow counts change, because the disk and back pressure settings need rechecking after each major change. Ask any partner to split its quote into discovery of flows and volumes, configuration changes, verification under load, and handover notes, so you can see which part you could do in-house.
Can our own team tune NiFi content repository archiving?
Yes, if your team has administrative access to the NiFi nodes, knows the flows well enough to measure data rates, and can test changes outside production. The settings themselves are documented in the NiFi System Administrator's Guide, and the work is mostly measurement and reasoning about trade-offs. Teams usually struggle with three things: reproducing realistic load, deciding how much replay history auditors or operators need, and keeping the content, provenance and flowfile repositories on disks that do not compete with each other. An outside partner can help where these gaps exist, while your staff keep ownership of the decisions and the resulting configuration.
How long does a content archive tuning engagement take?
Duration depends mainly on how many flows and nodes are in scope, how quickly you can supply usable data about them, and whether a test environment exists that can reproduce production load. A small single-node setup with a few flows moves much faster than a clustered deployment with many process groups that handle large files. Scoping is agreed in writing before work starts, so the timeline reflects the flow inventory you provide. Ask the partner which inventory items they need before they will commit to dates, and what happens to the schedule if test data does not match production.
What do we need to provide before tuning content archive settings?
The work needs read access to the flow definitions, the current nifi.properties and bootstrap.conf files for each node, and metrics or status history that show data rates over time. It also needs the disk layout for the repositories, including whether the content repository has its own volume, the filesystem type, and the capacity available. Retention requirements must come from your data owners, risk team and compliance function, because the replay window is a policy decision as well as a technical one. Ask for a sample of the largest files each flow handles and a description of any flows that must not lose lineage.
Can we shorten the replay window to save disk space safely?
A shorter window can be safe when no operational, audit or incident process depends on replaying content older than the new limit. Before you change the retention period, confirm which flows rely on replay, how long your incident response and reprocessing procedures take, and whether provenance data is kept separately with its own limits. Shortening the window reduces how much archived content the repository holds, but it does not remove the need for back pressure settings that protect the disk during peaks. Make the change in a test environment first, watch disk usage and back pressure events for a full business cycle, and record the decision with its owner.
What happens to replay if content archive is disabled?
If archiving is disabled, NiFi stops keeping the content of processed FlowFiles after they leave their queues, so provenance events can still show what happened but the content cannot be replayed from the archive. Replay then depends on re-ingesting data from the original source, if that source still holds it. Teams should check whether any downstream system, audit requirement or reprocessing job assumes the archived copy exists. Confirm the exact behavior for your NiFi version in the System Administrator's Guide before turning archiving off on production nodes.
Which NiFi settings should we check first when disks fill up?
Start with the settings that decide how much content stays on disk and when NiFi stops accepting new data. These are the retention period, the archive usage percentage for the content repository, and the back pressure percentage that blocks writes when the volume fills. Confirm the current values on every node, because clustered nodes can drift apart after manual edits. Then check whether large-file flows share a volume with the provenance or FlowFile repositories, since shared disks fail in ways that are harder to diagnose. A partner can review these values against your flow inventory, but the final numbers should be tested under your own load.
How DeployOpen scopes NiFi content repository work
DeployOpen scopes content repository and disk work for NiFi as a fixed-scope engagement. Before any change is made to archive settings, the written scope names the flows and content types being assessed, the nodes and repository volumes involved, the retention and archive targets agreed with the business, and the deliverables the client receives. Typical written deliverables for this kind of work include a current-state record of the content repository and archive configuration, a settings change list for nifi.properties and any flow-level changes, the back pressure and archive thresholds chosen for each node, and a test record showing how provenance replay behaves after the change. Ask for the deliverables to be listed by name so you can check them at handover.
The work runs inside your own infrastructure. DeployOpen makes changes on your NiFi nodes, in your cloud accounts, data centers or restricted networks, and the content repository, archive directories and monitoring stay under your control. Confirm during scoping who holds administrator access to the nodes, where the change records are kept, and which maintenance windows are available for restarts. Because disk behavior depends on the volumes and hosts you already run, the scope should include a check of the actual disk layout before any target is set, and a note of the assumptions that must be re-verified if storage changes later.
Ongoing support is available as a retainer for teams that want help after handover, such as reviewing disk usage trends, adjusting thresholds as flow volumes change, or testing settings before a NiFi upgrade. DeployOpen does not publish prices. A quote follows from the written scope, so the inputs that move cost are the number of nodes and flows in scope, how many content types and large-file flows need separate settings, the amount of testing needed for provenance replay, and whether the work is one-time or recurring. Ask each bidder to price the deliverables, the testing and the retainer separately so the comparison is like for like.
