Merging small files in NiFi: MergeRecord and MergeContent setup

Merging small files in NiFi before they reach your storage target

If a flow writes one FlowFile to object storage, a lakehouse table or a warehouse load, it will eventually produce millions of tiny objects, and a merge step placed before the write is the usual fix. In a typical design, a source such as an API, a message queue or a directory of exports delivers many small records or files, and each one becomes its own output. The target then pays for every request, listing and metadata operation, and query engines scan far more files than the data requires. NiFi's MergeRecord and MergeContent processors combine many incoming FlowFiles into fewer, larger ones. MergeRecord reads the records inside each FlowFile with a record reader and writer, so it can merge by record count and write a valid, single output file. MergeContent works on raw bytes and merges by size, count or age without parsing the content, which suits binary or non-record payloads.

The core choice is record-based or byte-based merging. Record-based merging keeps the output schema-aware and lets you control the number of records per file, which matters when a table format or warehouse loader expects consistent file contents. Byte-based merging is simpler and faster when content is opaque, but it can split a logical record across two outputs if you do not design around it. A team needs this pattern when monitoring shows high object counts per prefix, slow listing or loads, or many small writes billed per request. Before you choose, confirm with your storage and table format vendors the file sizes they recommend, and measure the arrival rate of your sources, because that rate determines how long a bin takes to fill.

Why tiny FlowFiles and tiny output files cause problems

Small files cost you in two places: inside NiFi, where every FlowFile occupies queues, repositories and processor time, and downstream, where each object, file or commit adds work for storage, lakehouse tables and warehouses. The cards below cover the costs to look for and how to size them in your own environment.

Queue and repository pressure

Each FlowFile carries attributes and a content claim, and NiFi tracks it in the FlowFile repository and in connection queues. When a source emits millions of tiny records as separate FlowFiles, queue counts and back pressure thresholds are reached long before data volume is high. Check the queue depth and repository disk usage on your nodes during a normal peak, and compare them with the record count per hour arriving from each source.

Per-object overhead in object storage

Object stores such as S3-compatible systems charge requests and track metadata per object, so a directory of thousands of small files costs more to list, copy and replicate than the same bytes in a few large objects. Writes made by PutS3Object or similar processors multiply the request count. Ask your storage team for request and listing limits, and count the objects a single day of flow output produces.

Slow reads and extra commits in lakehouse tables

Table formats such as Iceberg, Delta Lake and Hudi create a commit and snapshot for each write, and query engines must open every data file listed in the metadata. A stream of small appends therefore produces many tiny data files that slow scans and grow metadata. Plan compaction jobs and a target file size with your table platform, and check the file count per partition after a day of ingestion.

Load overhead in warehouses

Cloud data warehouses and bulk loaders process each staged file as a unit, so many small files add per-file setup, parsing and load-status tracking. Loads that wait on a file count or a scheduled batch can fall behind when the input is fragmented. Confirm the recommended file size range with your warehouse vendor, then compare it with the output of MergeRecord before you change any load schedule.

Flow-level delays from waiting bins

Merging only completes a bin when a size, count or age threshold is met, so a low-volume source can hold data for longer than expected. Max bin age exists to release partial bins, but a short value creates small output files again. Review the time each source takes to fill a bin and set the age limit against your freshness target, not only your file size target.

Uncontrolled correlation keys

Merging across the wrong attribute mixes records from different sources, tables or schemas into one output, which breaks downstream parsing and lineage. Correlation attributes must match the grouping your targets expect, such as table name, partition date or source system. Test the chosen attributes with a representative sample and confirm that every merged file contains one schema and one target partition.

MergeRecord and MergeContent settings and when each fits

This table compares the bin settings in MergeRecord and MergeContent and the trade-off each one makes between output file size, latency and memory. Read each row as a dial: raising a threshold usually produces larger outputs and waits longer, while lowering it sends smaller outputs sooner. Confirm exact default values in the processor documentation for the NiFi 2 release you run.

Setting	What it controls	Trade-off	When to choose it
Minimum number of entries	Smallest record or FlowFile count a bin needs before it can merge	Higher values produce fuller outputs but hold data longer and use more heap	Set when downstream tables or files need a minimum row count per write
Maximum number of entries	Upper bound on records or FlowFiles in one bin	Caps bin memory and output size; too low creates many small outputs	Set to protect heap on large inputs and to bound each output file
Minimum group size	Minimum total byte size before a bin is released	Larger minimums yield bigger files at the cost of latency on slow sources	Use when the target prefers files near a size floor, such as object storage objects
Maximum group size	Upper byte limit per merged output	Keeps files within writer or store limits but may split batches more often	Use to match the sizing your warehouse loader or lakehouse table expects
Max bin age	Time after which an incomplete bin is released regardless of thresholds	Short ages cap latency but leave small files; long ages fill bins but delay data	Always set on sparse or bursty sources so trickle data is not held indefinitely
Max bin count	Number of bins open at once for the processor	More bins mean more memory and more partial outputs; fewer bins may delay unrelated groups	Raise when many correlation values arrive interleaved, and watch heap while doing so
Correlation attribute name	Attribute that decides which FlowFiles share a bin, such as a table name or date partition	Good keys keep outputs homogeneous; high-cardinality keys create many sparse bins	Use when outputs must map to one table, partition or tenant
Merge strategy and output format	Bin-Packing or defragment strategy for MergeContent, and the record writer or content format for MergeRecord	Bin-Packing packs by size, defragment reassembles fragments; writer choice affects file size and read speed	Choose by target format, such as columnar output for lakehouse tables or line-delimited output for simple ingestion

Choosing MergeRecord or MergeContent for your data

When your input is structured records that share a schema, MergeRecord is usually the right choice. It reads records through a record reader, combines them through a record writer, and keeps the schema intact across the merged output. That means the writer you choose, such as a Parquet, Avro, JSON or CSV writer, defines the file format your storage or table receives, so the output format is set in one place rather than assembled from byte fragments. Because it understands record boundaries, it can group by correlation attributes and produce one well-formed file per bin.

Schema handling: MergeRecord needs a reader that can resolve the incoming schema. Confirm that every FlowFile in a bin shares a compatible schema, or the bin will fail or produce mixed output.
Output format control: Set the record writer explicitly, including compression and schema embedding options where the writer supports them.
Bin settings: Use minimum and maximum record counts and sizes, and set max bin age so that low-traffic periods still release partial bins.
Downstream loading: Single well-formed files suit table loaders such as PutDatabaseRecord and lakehouse writers that expect consistent schemas.
Ordering: Confirm whether your target needs ordered records, since merging can change the order in which records reach the output.

Before you commit, sample real inputs and check that the reader infers or accepts every schema variant in production traffic.

Sizing bins and max bin age for object storage, lakehouse tables and warehouses

Derive the bin size from the destination, not from a number you like. Start by asking what each target expects: object storage consumers that list and read many small objects, lakehouse tables that accumulate data files per partition and later need compaction, and warehouse bulk loaders that prefer a modest number of reasonably sized load files. Each destination has its own guidance on file size and file count, and that guidance, not a generic figure, should set Minimum Group Size and Maximum Group Size on MergeRecord or MergeContent. Confirm the recommended file size with your table format, object store and warehouse documentation, and test the chosen size against your real record volumes before you fix it in the flow. Remember that Maximum Number of Records and Maximum Group Size are both limits, so a bin closes on whichever is reached first, and the output size you see in production depends on how fast data arrives.

Correlation attributes decide what is allowed to share a bin. Set Correlation Attribute Name on MergeContent, or make sure MergeRecord sees attributes that identify the source, the target partition or the destination table, so that records bound for different tables, tenants or retention classes never land in the same output. Without this, a merged file can mix partitions, which forces the destination to rewrite or split it later. Keep the attribute set small and stable. Attributes with high cardinality, such as a per-request identifier, produce one bin per value and recreate the tiny-file problem in a different form.

Max bin age sets the latency ceiling for sparse sources. A bin that never reaches its minimum size would otherwise wait indefinitely, so Max Bin Age forces it out after a set period, even if it is small. Choose this value from the freshness your consumers actually need, and accept that a quiet source will produce small files whenever the age limit fires first. Confirm the behavior of each processor version in your release, since bin-age handling is one of the settings to verify during testing. Where a destination tolerates some small files, a longer age limit reduces file count at the cost of delay. Where it does not, add a periodic compaction job downstream and document who owns it.

Setting up a merge flow in NiFi step by step

This sequence turns a stream of tiny FlowFiles into batched, correctly sized output files, with a check at each stage so you know the merge behaves as designed before it reaches production targets.

Confirm the target file size guidance
Before touching any processor, ask the team that owns the destination (object storage bucket, lakehouse table or warehouse loader) for its recommended file size range and its limits on file count per load. Write those numbers down with the source of each, since the merge thresholds depend on them. Your own team usually owns this decision, and a partner can help you document it. This step is done when you have a written target size range and a maximum acceptable file count per batch, each attributed to the destination's documentation or owner.
Set the correlation attribute
Choose the FlowAttribute that decides which FlowFiles belong together, such as a table name, source system, date partition or tenant ID, and make sure every upstream processor sets it. Records with different values must never share a bin, or the output mixes data that should be loaded separately. In MergeRecord and MergeContent, set the correlation attribute name and confirm it is populated on every incoming FlowFile. Check this by running a small sample through the upstream processors and inspecting the attributes in the queue. The step is complete when every sampled FlowFile carries a non-empty value for the chosen attribute.
Configure bin thresholds and max bin age
Set the minimum and maximum number of entries and the minimum and maximum size for each bin, using the target range from step one as the guide. Set Max Bin Age so that quiet correlation groups still close and flow out, because a bin that never fills would otherwise hold its data indefinitely. Keep the maximum size below your destination's ceiling so that a bin cannot overshoot it. Verify the settings by feeding a known number of FlowFiles and watching the Merge processor's queue and bin behavior. The step is done when a test run produces bins that close either on size, on count or on age, as you intended.
Choose the record reader and writer
For MergeRecord, select a Record Reader that matches the incoming format and a Record Writer that produces the format the destination expects, such as Parquet, Avro or JSON. Confirm that the writer's schema and the reader's schema agree, and decide whether the output schema is inherited from the input or fixed in a schema registry. For MergeContent, pick the merge strategy that suits raw bytes, such as Defragment or Bin-Packing, and confirm the header, footer and demarcator settings. Check the result by merging a sample and opening one output file with the tool your destination team uses to read it. The step is done when the output file reads cleanly and its schema matches what the destination expects.
Route merged and original relationships
Connect the merged relationship to the next processor, and route the original relationship to a separate path that is either archived, dropped after a successful merge or kept for replay, according to your retention rules. Connect the failure relationship to an error path so that malformed input does not block the bin. Confirm that the original FlowFiles are not left stranded in the queue after a merge completes. Check this by counting FlowFiles on each relationship during a test run: the number entering the merge should equal the sum of the originals routed and the merged output, plus any failures you expected. The step is done when those counts reconcile.
Test with a replayed sample
Capture a representative sample of real input in a provenance or staging location, then replay it through the flow in a non-production environment. Count FlowFiles in and out, record the number and size of the output files, and compare them with the target range from step one. Inspect a sample of output files for correct record counts, schema and correlation values, and check that no bin has been left open after the age limit. Repeat the replay after each change to thresholds or the writer. The step is complete when output file sizes fall within the target range for the bulk of the sample and the in and out counts reconcile, and the results are written down for the change record.

Security and operational considerations for merge flows

Merge processors hold data in memory and on disk while bins fill, so the risks sit in what they retain, who can reach the destination, and how the cluster behaves when a node stops. The cards below cover those risks in the order a team usually meets them during design and rollout.

Sensitive data in merged files and provenance

A merged file contains every record that landed in its bin, and provenance events record attributes and content references for each FlowFile that entered the merge. If a bin mixes tenants, regions or data classes, one output file can expose data that should have been separated upstream. Check which attributes are kept in provenance, whether content claims are retained after merge, and who can view provenance events in the UI. Confirm the correlation attribute does not carry personal identifiers that should not be stored in event history.

Destination credentials through parameters

Merge flows usually end in a PutS3Object, PutDatabaseRecord or similar processor that needs credentials. Store those values in parameter contexts and mark them as sensitive so they are not shown in the canvas or exported in plain text. Limit which process groups can read each parameter context, and separate the credentials used for test and production targets. Ask who can change a parameter context, because a changed bucket name or endpoint redirects every future merged file without any change to the flow itself.

Back pressure and repository sizing

Bins hold FlowFiles in the content and FlowFile repositories until a merge fires, so a slow or stopped destination can fill disks while merges wait. Size the content repository for the expected bin volume plus the backlog you must absorb during a destination outage. Set back pressure thresholds on connections feeding the merge so upstream processors slow down before disks fill. Review repository and JVM settings against the administration guide before raising bin limits, because larger bins keep more data in flight.

Behavior on node restart and clustering

Bins are in-memory state, so a node restart or crash can leave partly filled bins that must be rebuilt from queued FlowFiles, and the bin age clock may reset. Test a restart in a non-production cluster while bins are populated and count the outputs produced afterward. In a cluster, decide how primary node-only processors and load-balanced connections route related FlowFiles to the same node so a correlation group is not split. The clustering options in the administration guide, including ZooKeeper or Kubernetes state, determine how quickly coordination recovers.

Monitoring bin contents and output counts

A merge that quietly holds data or emits too many small files only shows up in queue depths and destination listings. Track the number of FlowFiles queued in front of each merge processor, the age of the oldest waiting FlowFile, and the count and average size of objects written to the destination. Alert when the output file count rises sharply relative to input volume, which usually signals bin limits or max bin age settings that no longer match the traffic. Record these figures in a runbook so an operator can tell a slow source from a stuck merge.

Questions teams ask before merging small files in NiFi

What drives the cost of building a NiFi merge flow?

The main driver is how many distinct sources, formats and destinations the merge must serve, because each combination needs its own correlation logic, bin settings and output checks. Cost also rises with the volume of tiny files you already have, the number of target tables or prefixes that need separate sizing, and how much testing is needed against realistic arrival patterns. Ask any provider to price these items separately in the written scope: flow design, testing with representative data, deployment to each environment, and handover. Recurring costs depend on how often the flow changes and who operates it after go-live.

How long does a typical small-file merge implementation take?

No duration for this work is supplied with the research for this page, so treat any figure you see as something to verify. The schedule is shaped by the number of flows to merge, whether the sources already have stable correlation attributes, the quality of sample data available for tuning, and whether the target platform needs its own load testing. Teams that have to design correlation keys from scratch usually spend more time than teams that already tag files by source and partition. Ask for a timeline tied to a written scope with named milestones, not a single number.

What does the work need from our team?

You need someone who knows the source systems and can explain how files arrive, including batch windows, file naming, typical sizes and any late or out-of-order delivery. You also need a owner for the target platform, such as the object store, lakehouse table or warehouse, who can confirm acceptable file sizes and any compaction jobs already in use. Your team should provide representative sample data, access to a non-production NiFi environment, and a decision-maker who signs off the bin thresholds and the maximum latency the business accepts. Without these inputs, tuning becomes guesswork.

Can our in-house team build this without a partner?

Yes, a capable NiFi team can build a merge flow in-house, provided it has time to learn the MergeRecord and MergeContent settings and to test them under real load. The skills that matter most are understanding how bins fill, how max bin age forces output when traffic is quiet, and how correlation attributes group related FlowFiles. Teams without recent NiFi experience should budget for that learning curve and for a staging environment that mirrors production volumes. A partner makes sense when the scope is large, the deadline is fixed, or internal staff cannot take time away from current operations.

How do we validate that output file sizes are right?

Start by agreeing on a target size range with the team that owns the destination, because object stores, lakehouse tables and warehouses each have different performance characteristics. During testing, record the size, record count and age of every output file, then compare the distribution with that target over several representative days. Check that the flow still produces files when input drops, which confirms that max bin age and minimum entry settings behave as intended. Keep these measurements as acceptance evidence and repeat them after every flow change or volume increase.

What goes wrong when bin settings are set too loosely?

Loose settings let bins wait too long, which delays data and can hold large numbers of FlowFiles in the repository until a bin closes. Very large bins can also produce outputs that exceed what a downstream loader or query engine handles well. Teams should watch queue depth, bin age and provenance for stalled groups, and confirm that the repositories have enough disk for the held data. Tune one setting at a time and document why each value was chosen.

Does the merge flow need to be versioned and reviewed?

Yes. NiFi 2 includes Flow Registry Clients for GitHub and GitLab that store flow versions in a Git repository, which lets you review changes to merge settings before they reach production. Agree on who approves changes to bin thresholds, since a small edit can change output file sizes across many tables. Keep the Git history and test results together so that the reasoning behind each setting is traceable.

How DeployOpen scopes and delivers merge flows

DeployOpen starts with the destination, not the processor. Before any bin setting is chosen, we agree with your data platform owners what a healthy output file looks like for each target: the object storage prefix layout, the table format and its file-size guidance in a lakehouse, or the load batch size a warehouse loader handles well. We also measure the real input profile, including arrival rate per source, file counts per hour, record sizes and the longest acceptable delay before data must be visible. Those numbers drive the minimum entries, maximum entries, maximum bin age and size thresholds in MergeRecord or MergeContent, so the flow is sized from requirements rather than from defaults copied from another project. Confirm early which of these limits your target platform itself enforces.

The work is delivered as a fixed-scope engagement with written deliverables. Typically these include the merge flow process group as a versioned flow in your own repository, parameterized correlation attributes and thresholds, a documented schema and record reader and writer choice, provenance and monitoring checks for bin counts and oldest bin age, and a runbook for changing thresholds safely. Testing uses representative data samples supplied by your team, including bursty arrivals, a quiet period that exercises max bin age, and malformed records routed away from the bins. Acceptance is measured against the output file sizes and latency targets agreed in the scope. Ask any partner to show these artifacts in the quote, and to state what is excluded, such as changes to the source systems that create the small files.

After go-live, merge settings usually need adjustment as volumes change, so DeployOpen can support the flow through an optional retainer. Under a retainer we review output file size distributions and bin behavior on a schedule you choose, retune thresholds when sources change, and update the flow when the target platform or NiFi release changes. Ask a provider how retainer hours are counted, how urgent changes are handled, and whether the retainer covers the NiFi version you run. The costs that recur are usually the review cadence and the change work, while the initial build is one-time, so compare the two in the quote rather than only the build price.

Merging small files and batching records in NiFi