NiFi provenance repository sizing and tuning

Sizing the NiFi provenance repository for lineage you actually need

The NiFi provenance repository records an event for every change made to a FlowFile as it moves through a flow, such as creation, routing, modification and delivery. Those events are what let operators trace where data came from, what happened to it and where it went. Because every processed FlowFile can produce several events, the provenance settings decide how much disk and heap the cluster spends on history, and how much of that history is kept long enough to be useful. On a high-volume cluster, the repository can grow faster than the content it describes and compete with the content and FlowFile repositories for the same storage.

Start by deciding which lineage questions the business must answer and for how long. Auditors, incident responders and data owners ask different things: a replay of last Tuesday's batch needs events kept for days, while a flow that only moves telemetry between two systems may need almost none. Write the retention period and the maximum size you can accept, then size the disk and index against your measured event rate in a test environment. Confirm the exact property names, default values and index tuning options for your NiFi version in the NiFi System Administrator's Guide before you change anything in nifi.properties.

The alternatives trade history for resources. The VolatileProvenanceRepository keeps events in memory, so it avoids disk writes but loses all events on restart, which is acceptable only when no one will need to investigate the period before a restart. The NoOpProvenanceRepository disables provenance storage entirely and reduces resource use further, but it removes the ability to trace or replay FlowFiles. Choose either option only for flows where you have documented that lineage is not required, and record that decision with the flow owner, since it is hard to reverse once data has passed through unrecorded.

Provenance repository options and when each fits

This table compares the persistent provenance repository with the volatile and no-op options, then lists the tuning settings that shape retention and disk use. Read the first three rows to choose a repository type, and the remaining rows to decide what to confirm in the administration guide for your version.

Option or setting	What it controls or keeps	Trade-off and what to confirm	Flow profile it suits
Persistent provenance repository	Keeps lineage events on disk so they can be searched and replayed after restarts, within the retention limits set.	Uses disk and index resources that grow with event volume. Confirm the default implementation and its settings in the current administration guide.	Regulated or audited flows where you must trace where data came from, went and changed.
VolatileProvenanceRepository	Keeps provenance events in memory only, so recent lineage is visible while the node runs.	Events are lost on restart, so it cannot support investigations after a restart. Confirm memory limits with the administration guide before use.	High-volume flows where short-lived lineage in the UI is enough and restart loss is acceptable.
NoOpProvenanceRepository	Stores no provenance events, which removes the provenance storage and indexing load.	No lineage, search or replay is available for the flow. Confirm that no downstream audit or troubleshooting depends on provenance.	High-volume, low-risk flows that do not need lineage, such as bulk transfers with independent audit controls.
Retention time	Sets how long provenance events are kept before they age out of the repository.	Longer retention increases disk use. Ask your team to set it from the required investigation window, then confirm the property name and behavior in the guide.	Flows where investigators need to look back over a defined period.
Size limits	Caps the total storage the repository may use, removing older events when the limit is reached.	A low cap can drop events before the retention time expires. Confirm which limit applies first and how removal is ordered.	Shared disks where provenance must not crowd out content or FlowFile repositories.
Index shard size	Controls how the provenance index is divided into shards, which affects indexing and search performance.	Larger shards mean fewer files but heavier individual searches. Confirm the setting and its guidance for your version before changing it.	Flows with heavy event volume where search latency in the UI matters.
Multiple storage directories	Spreads provenance storage across more than one directory or volume to increase write capacity.	Only helps if the directories sit on separate disks. Confirm the property format and how events are distributed across them.	High-throughput flows with several physical disks available to NiFi.

What drives provenance storage growth

Provenance volume is the product of how many events each FlowFile generates and how many FlowFiles move through the flow. Retention and size limits then decide how much of that history stays on disk before the repository prunes it. Change one driver and the others shift, so model them together from measured values.

Event rate through the flow

The FlowFile rate entering the flow sets the base volume for everything else. Measure it from the status history of your source processors over a busy period, not an average day, because peaks set the disk you need. Record bytes and FlowFiles per second separately, since small-record feeds and large-file feeds generate very different disk footprints even at the same count. Ask whether ingest bursts are scheduled, such as month-end batches.

Processors that emit events

Each processor that creates, modifies, routes or sends a FlowFile can record a provenance event, so the number of event-producing steps multiplies the volume. Count the processors on each path a FlowFile takes, not just the ones on the main line. Routing and attribute-update processors are easy to overlook but still add events. Review the flow when a new step is added, because the growth often comes from a later change rather than the original design.

Events per FlowFile hop

Every time a FlowFile crosses a connection and is processed, the repository records the step, so long linear paths with many small transformations produce many events per record. Splitting and merging change the count too, because one parent can produce many child events. Sample a few FlowFiles through the provenance view and count the events each one leaves behind. That count is the multiplier to apply to your event rate.

Retention period

The retention period decides how long events remain queryable before the repository ages them out. Growth under time-based retention is the event rate multiplied by the retention window, so doubling the window roughly doubles steady-state disk use. Confirm with your own requirements how long lineage must be searchable, and whether that need matches the operational need for replay. Do not set retention by habit alone.

Size limit and pruning order

The maximum size caps the repository regardless of retention, and whichever limit is reached first controls pruning. A busy flow can hit the size cap within hours, which shortens the effective retention without any warning in the flow itself. Compare the two limits against your measured daily volume to see which one will trigger. Check the provenance logs after a load test to confirm the limit you expect is the one that is applied.

Measure before modeling

Build the sizing model from numbers taken from your running flow: event rate by peak and average, events per FlowFile for each path, and observed bytes per event on your data. Run the model across the retention and size settings you are considering, and check it against a sustained test window. Ask your vendor or partner to show these measurements in the quote, so the disk plan rests on your own flow.

Choosing a provenance setting by flow profile

Flows that carry regulated or contractual data often need to show where each record came from, what happened to it, and where it went. For these flows, the recommended direction is to keep the persistent provenance repository enabled with a retention window and size limit that your compliance owner has agreed to in writing, so that lineage questions can be answered after the fact. Keep in mind that provenance retention is bounded by disk, so the limits you pick decide how far back you can answer a question.

Persistent repository: Use the default persistent provenance repository rather than volatile or no-op storage, so events survive restarts.
Retention and size: Set the time and size limits from the retention period your records policy requires, then confirm the disk can hold it at peak event rate.
Replay scope: Confirm which processors must support replay and whether content retention is long enough for the replay window you need.
Access to lineage data: Decide who may view provenance, since event attributes can contain sensitive values.

Before choosing, ask the data owner which audit questions must be answered, how far back they reach, and who will sign off on the retention period.

How to configure and verify provenance settings

This sequence takes a provenance repository from a loose idea of retention to a configured, measured and verified setting. Each step produces a record you can check, so the final configuration matches the lineage your flows actually need and the disk you actually have.

Inventory flows and lineage requirements
The flow owner and the compliance or audit lead list every process group that needs lineage, the questions lineage must answer (for example, which system received a record and when), and the period those answers must cover. Flows that only need delivery status can be marked for a lighter setting or no provenance at all. Done means each flow carries a written label: full lineage, short retention, or no provenance, signed off by the owner who will be asked to explain a gap later.
Measure event rate on a representative load
The platform team runs the busiest flow in a non-production cluster with production-like input and counts provenance events per second and per day, using the NiFi status history and the disk growth of the existing provenance directory. Confirm the average and peak periods with the data owners, since month-end batches can dominate sizing. Done means you have a daily event volume and an average event size that you can multiply by the retention period you chose in the first step.
Choose the provenance repository type
The platform team selects the persistent provenance repository for flows that need lineage across restarts, the volatile repository for flows that need short-lived in-memory lineage and accept losing events on restart, or the no-op repository for high-volume flows with no lineage requirement. Record the choice per node configuration and confirm the implementation class name in the current administration guide before editing anything. Done means each cluster has one documented repository type and the team has agreed which flows depend on it.
Set retention, size limit and index shard size
The administrator edits the provenance properties in nifi.properties with the retention time and maximum storage size that match the volume from step two, then sets the index shard size so index files stay at a manageable size. Property keys change between releases, so copy the exact names from the current Administrator's Guide and keep the values in version control. Done means the file validates on a staging node and the chosen retention and size match the sign-off in step one.
Configure storage directories on separate volumes
The administrator defines the provenance storage directories, and where the design calls for it, multiple directories on separate volumes so index and log writes do not compete with the content repository. Confirm that each directory exists, is owned by the NiFi service account, and sits on a disk sized from the calculation in step two. Done means a directory listing on each node shows the expected paths and free space, and no provenance path shares a volume with the content or FlowFile repositories unless that trade-off was accepted in writing.
Restart nodes in a controlled window
The operations team applies the changes one node at a time, using the cluster disconnect and reconnect steps in the administration guide so flows keep running on the other nodes. Watch the bootstrap log and the NiFi application log for provenance initialization messages and confirm each node rejoins the cluster. Done means every node is connected, the log shows the repository started with the new settings, and no node reports a configuration error.
Verify disk use, index behavior and query results
The administrator watches the provenance directories over at least one full retention cycle that the team can observe, checks that growth tracks the calculation and that the size limit enforces deletion of the oldest data, and runs a provenance query in the UI for a known test FlowFile. Confirm with the flow owner that the returned events show the expected lineage. Done means the disk growth curve, the query result and the owner's sign-off are recorded, and any gap between estimate and observation is explained before the setting goes to production.

How provenance settings affect disk and content repositories

Provenance retention and size limits decide how much disk the provenance repository can consume, and that consumption competes with the content repository in ways that are easy to miss. Provenance events describe what happened to each FlowFile and point to the content claims held by the content repository. When no live FlowFile references a content claim, the content repository can reclaim that space, unless content archiving keeps the bytes for replay. A long provenance window combined with archiving therefore keeps content on disk for longer, so the two limits must be set together. Confirm in the NiFi System Administrator's Guide how time-based and size-based limits interact on your version and which events are removed first when a limit is reached.

The risk grows when the provenance, content, FlowFile and status repositories share one volume. A burst of events or a large content backlog can fill that disk, and NiFi then cannot write the records it needs to keep moving data. In practice the flow slows before it stops: processors wait on repository writes, queues grow, and back pressure reaches upstream sources. A full provenance volume is therefore a flow availability risk, not only a gap in lineage reporting. Ask your team to test this on a non-production cluster by filling the provenance directory to a known level while watching queue depth and processor timing, so the behavior is understood before production load arrives. Separate volume planning is covered in the sibling topic on repository disk layout and sizing.

Record the decision in writing before changing any setting. For each flow, note how long lineage must be searchable, how long archived content must remain replayable, and how much free space each volume must keep. Then alert on provenance and content volume usage well before the limits are reached, and treat repeated back pressure as a disk question before treating it as a processor question. The sibling topic on content repository archiving and disk usage explains the archive and back pressure percentages that interact with these limits. If the retention period is not defined, ask your compliance lead to confirm it first, since DeployOpen prepares systems and evidence for assessments but does not set the retention requirement.

Security and operational considerations for provenance data

Provenance events record what happened to each FlowFile, including attributes and content references, so the repository is a data store in its own right. The cards below cover who can search and view it, what it reveals, how long it is kept, and how to watch its disk and index health before retention or sizing decisions are tested in production.

Restrict provenance search and lineage

Anyone who can open the provenance view can search events and trace lineage across a flow, which can expose data passing through processors they would not otherwise see. Map the provenance read permission to a small operator group in your authorization policies and review it as flows change. Before go-live, confirm which roles can view event details and lineage graphs in your NiFi version.

Attributes can carry sensitive values

Provenance events store FlowFile attributes, and flows often place account identifiers, customer names or tokens into attributes during routing. Those values persist for the full retention period even if the content is later deleted. Inventory attributes at each processor that writes them and remove or hash sensitive values before the event is recorded.

Content references outlive content

Provenance events point to content claims, so replay can recover payloads while the content still exists in the content repository. Anyone able to replay an event can re-read data that the flow was meant to discard. Decide whether replay is needed for each flow, and limit it to the operators who handle that data class.

Set retention to match data policy

Retention decides how long provenance records exist, and that period often differs from the retention your data policy sets for the underlying records. Keeping events longer than the data itself is permitted extends exposure. Set the time limit and size limit together so the policy you document is the one the repository enforces, and confirm the behavior when the size limit is reached first.

Monitor provenance disk use

The provenance repository grows with event volume, and a full volume can stop flow processing once back pressure applies to the repositories that share it. Track disk usage for each provenance directory and set alerts well before the configured limits. Verify that the directories sit on volumes sized for the retention you chose.

Check index health over time

Provenance search depends on indexes that are written in shards and merged as events arrive. When indexing falls behind or index files accumulate, searches slow and operators lose the audit trail they rely on. Review search response times and index file counts in your monitoring, and confirm after each upgrade that searches still return expected results.

Questions teams ask before sizing provenance

What does sizing the provenance repository cost in-house compared with a partner?

The main cost is engineer time, and it is paid either way. In-house, your platform or data engineers spend time reading flow exports, measuring event rates, choosing retention and limits, and testing the result on a non-production cluster. With a partner, you pay for that same analysis under a written scope, and your own staff review the output and own the change. Costs rise with the number of flows, the number of distinct event types, and how many environments must match. Recurring cost comes from storage growth and periodic reviews as flows change. Ask any quote to separate the one-time analysis from the ongoing review and to state which environments and flows are covered.

How long does a provenance sizing exercise take?

Duration depends mainly on how many flows are in scope and how easily you can get production-like data. A single flow group with exported flow definitions and recent event rates is a short exercise. Many process groups, several clusters, or flows that touch regulated data take longer because each one needs its own retention decision and a load test. Access also matters: if the team must wait for change approvals or cannot replay representative traffic outside production, the schedule grows. No general duration is given here. Ask for a timeline that lists the flows, the environments, and the test runs it includes.

What must our team provide before sizing starts?

Three inputs make the sizing concrete:

Flow exports from each NiFi environment, so the number of processors that emit provenance events can be counted.
Event rates per flow at peak and average, taken from status history or your monitoring system.
A written retention policy stating how long lineage must be queryable and who needs it.

Also provide the hardware layout, including whether provenance will share a disk with the content or FlowFile repositories, and the compliance owner who approves retention. Missing inputs are the most common reason sizing estimates turn out wrong.

Can a small team size provenance on its own?

A small team can do it if one person understands NiFi configuration and the flows well enough to measure them, and if someone else can test changes before production. The work is mostly measurement and documentation rather than deep specialization, so the risk lies in skipping the load test or underestimating peak rates. Teams that lack a non-production copy of the flows, or that must answer audit questions about lineage, usually benefit from outside review. Decide based on who will own the retention settings after the work ends.

Can we switch to the volatile or no-op provenance repository later?

Yes, the setting can usually be changed later, but the consequences should be understood first. The NiFi Administrator's Guide describes the VolatileProvenanceRepository as keeping provenance in memory, so events are lost on restart. The NoOpProvenanceRepository disables provenance storage to reduce resource use. Once you switch to either, previously recorded lineage is no longer available through the normal provenance search and replay path, and any audit or troubleshooting process that depended on it must change. Confirm the exact configuration procedure and restart behavior for your NiFi version in the administration guide before planning the switch.

How do event rates change the sizing answer?

Event rates set how fast provenance grows, so they drive both the retention period and the size limit. A flow that creates many events per FlowFile fills the repository faster than a flow with few processors, even at the same data volume. Bursts matter as much as averages, because a short peak can push the repository to its limit and cause older events to be removed earlier than the retention policy expects. Measure peaks across normal business cycles and use them, not averages, when checking whether the limits leave room for growth. The sizing should also account for flows that are added later.

What goes wrong when provenance shares a disk with other repositories?

Provenance writes compete with the FlowFile and content repositories for disk throughput and space. When they share a volume, a provenance backlog or a full disk can slow or stop flow processing, which is harder to diagnose than a problem on a dedicated volume. Engineers usually discover this after an outage rather than during design. Sizing should therefore consider volume layout alongside retention and limits. Confirm the recommended directory layout for your NiFi version in the administration guide and test the failure mode before you rely on the design.

How DeployOpen scopes and delivers provenance tuning

DeployOpen starts from the flows that generate provenance, the lineage your teams and auditors actually need, and the disk and retention limits you can commit to. We then set the provenance repository, index and storage directories against those decisions and document the result for your operators.

How we scope

Inventory which flows need lineage, replay and audit trails, and which high-volume flows could run with a volatile or no-op repository.
Record the event rate, retention period and disk budget you need, so the written scope fixes the retention and size limits to test.
Engagements are fixed-scope: deliverables and timelines for provenance sizing are agreed in writing before work starts.

How we build

Configure the persistent provenance repository, index shard size and multiple storage directories in the nifi.properties file for your nodes.
Place provenance directories on volumes separate from the FlowFile, content and status repositories, and confirm the layout on each node.
Load-test representative flow volumes, check disk growth against the retention limits, and adjust settings before the production cutover.

How we hand over

Deliver a settings record covering retention, size limits, shard size, directory paths and the reasoning behind each value.
Document which flows use the volatile or no-op provenance repositories, what lineage they lose, and how that choice was approved.
Walk your operators through monitoring of provenance disk usage, alert thresholds and the steps to change limits later.

How we support

Ongoing support, upgrades and periodic review of provenance growth are available as a retainer after handover.
Re-check sizing when event rates, flow counts or retention requirements change, and record each change in the settings record.
Confirm with your compliance team how long lineage must be kept, since that requirement drives the retention setting more than any default.

Sizing and tuning the NiFi provenance repository