Why record-oriented processing is the default for NiFi data flows
Picture a team whose NiFi flow receives a daily CSV export of several hundred thousand rows, then uses a SplitText processor to break the file into one FlowFile per line, routes each line through attribute extraction and a RouteOnAttribute step, and finally merges the results back into a file. Every row becomes its own unit of work, with its own FlowFile, provenance events and queue handling. Record-oriented processing changes that model. The whole file stays one FlowFile, and the flow reads it as a set of typed records through a Record Reader, transforms those records in place, and writes them back out through a Record Writer. The reader and writer controller services hold the format knowledge, so the processors themselves only manipulate fields. The NiFi 2.12.0 components index lists the readers the platform ships, including CSVReader, JsonTreeReader, AvroReader, XMLReader, ExcelReader, GrokReader, CEFReader, SyslogReader, YamlTreeReader and ProtobufReader, and the matching writers, including JsonRecordSetWriter, CSVRecordSetWriter, AvroRecordSetWriter and XMLRecordSetWriter.
Teams usually need this feature when they convert formats, enforce a schema, filter or aggregate rows, or feed a lakehouse or database where the record shape matters. Processors such as ConvertRecord, UpdateRecord and QueryRecord run on these controller services, and QueryRecord lets a team express filtering, renaming and aggregation as SQL against the FlowFile contents. The decision to make is how the schema reaches the reader. Schema inference is quick to start with but depends on the sample data, so a changed column type can surprise the flow later. An explicit schema, whether in the controller service or from a schema registry, makes failures visible at the reader and is the safer default for production. Confirm in your own test environment which readers and writers your chosen NiFi release provides, and check the behavior of each reader on malformed input, since error routing differs by format.
Record processing does not remove the need for splitting. A flow that must route individual messages by content, call an external API per event, or apply a processor that only works on a single FlowFile still needs to break records apart, and the cost of many small FlowFiles is real. The practical rule is to keep the file intact through conversion, validation and enrichment, and split only at the point where a downstream system needs one record per message. Measure throughput and queue depth on a representative dataset before settling the design, because the gain from record processing depends on your data volumes, record size and cluster sizing. Ask whether your team has the schema ownership to maintain explicit schemas over time, since that ownership decides whether the record-based design stays stable as sources change.
Record readers, writers and processors and when each fits
This table compares the record readers and writers that NiFi 2.12.0 ships, then the three record processors that use them. Read each row for the typical use and the trade-off, and confirm component availability in your build before designing around a format.
| Component | Typical use | Trade-off | Confirm before choosing |
|---|---|---|---|
| CSVReader and CSVRecordSetWriter | Flat tabular exports from ERP, banking and reporting systems | Types are weak without an explicit schema, so inferred numbers and dates can drift between files | Header handling, delimiter and quote settings for each source feed |
| JsonTreeReader and JsonRecordSetWriter | API payloads and event messages with nested objects and arrays | Nested paths need careful schema design, and large arrays can be heavy to hold in memory | Whether records are one object per FlowFile or a top-level array |
| AvroReader and AvroRecordSetWriter | Compact typed data moving between Kafka, NiFi and data lake tables | Requires a schema for every record, so schema versioning must be managed with the producers | Where schemas come from (embedded, schema text or an external registry) |
| XMLReader and XMLRecordSetWriter | Legacy and government feeds, SOAP-era messages and industry XML standards | Repeated elements, attributes and mixed content map awkwardly to flat records | Which element is the record root and how attributes should be named |
| ExcelReader | Spreadsheets supplied by business teams or partners as recurring files | Cell formatting and merged cells vary by author, so layout changes break mappings | Sheet selection, header row position and whether formulas are stored as values |
| ProtobufReader | Protobuf-encoded messages from services and telemetry sources | Needs the compiled message definition or a schema source, which adds a dependency to version | Schema delivery method and who owns changes to the .proto definitions |
| ConvertRecord | Changing format, for example CSV to Avro or JSON to Parquet, without changing field content | Only as good as the reader and writer schemas, and type coercion errors surface at runtime | Parquet writer availability in your build and the target table's type rules |
| UpdateRecord | Setting, renaming or deriving field values, masking or normalizing data per record | Expressions run for every record, so complex logic adds CPU cost on large batches | Which fields are sensitive and need masking before they leave the trust boundary |
| QueryRecord | Filtering rows, selecting columns and aggregating with SQL over FlowFile content | SQL is convenient for shaping data, but joins across FlowFiles are not possible and complex queries are harder to debug | Query design against a sample of real records, plus the expected output schema |
Schema inference or an explicit schema: how to choose
When a producer owns a documented contract and changes it through a release process, an explicit schema is usually the simplest choice. The reader is configured with a fixed Avro, JSON or CSV definition, so every FlowFile is parsed against the same structure and the flow fails predictably when a field is missing or retyped. Inference adds little here because the structure is already known, and it can still misread a sparse column as a different type than the contract intends.
- Fixed field types: Declaring types stops a numeric-looking identifier from being read as an integer and losing leading zeros.
- Predictable failures: A contract violation routes to failure or parse-error relationships instead of silently producing different output.
- Reviewable change: The schema lives in version control with the flow, so a change to the contract shows up in a diff.
- Cheaper runs: Readers do not need to sample each batch to decide what the fields are.
- Shared definition: Downstream writers and queries use the same schema text, which keeps renames and removals visible.
Before you commit, confirm with the producer how they version the contract and whether they announce breaking changes before release.
When you receive data from partners, legacy exports or new internal systems whose structure changes without notice, inference lets the flow keep moving while you learn what the data looks like. The reader samples incoming records and proposes field names and types, which suits discovery work and exploratory pipelines. The cost is that the schema is derived from whatever the sample contains, so the flow's behavior depends on the data it happened to see.
- Sample-dependent types: A field that is empty or all digits in the sample may be typed as a string or integer, and later records fail or lose precision.
- Missing optional fields: Fields that appear only in some records may be dropped or added inconsistently across batches.
- Silent drift: A new column can change downstream output without any flow change, so monitor the writer output and not only the job status.
- Discovery value: Inferred output gives a starting schema that you can review, correct and then pin as an explicit schema.
- Unknown ordering: Field order may change between files, which matters for fixed-width or positional consumers.
Test inference against several representative files, including the rare edge cases, and confirm which inference settings your NiFi version exposes for the reader you use.
When records carry personal, financial or health data, the governance question is not only whether a record parses but whether it matches a defined and approved structure. An explicit schema gives reviewers a single artifact to approve, lets you show which fields exist and how they are typed, and makes rejected records traceable to a specific rule. Inference is hard to defend in this setting because the approved structure would be whatever the sample happened to produce, and that sample may not be the one a reviewer examined.
- Approved artifact: A versioned schema gives change management and auditors one document to review for each data flow.
- Field-level control: Explicit types make it possible to decide which fields may enter the flow, and which must be dropped before writing.
- Traceable rejection: Records that fail validation can be routed to a quarantine path with provenance that points to the rule.
- Separation of duties: Security or data governance can approve the schema while platform engineers operate the flow.
- Evidence for assessors: The schema, its version history and failure routing can be shown as design evidence, although the assessment itself is run by the auditor.
Ask your compliance and data owners to sign off the schema before production, and confirm how your organization requires schema changes to be approved.
How to build a record flow step by step
The sequence below takes a record flow from an unknown input file to a tested pipeline that reads, converts, filters and writes data with a known schema at every stage. Each step ends with a check so that problems surface before the flow reaches production.
Confirm the input format and sample files
Collect representative files from the real source, including the awkward cases: missing fields, extra columns, empty values, unusual encodings and malformed rows. Confirm with the data owner which format the source produces (CSV, JSON, Avro, Parquet, XML or Excel) and whether the format version or delimiter can change. You know this step is done when the team holds a sample set that covers both normal and bad records and the data owner has signed off on what a valid record looks like.
Define the schemas for each stage
Write down the schema the flow expects on input and the schema it must produce on output, with field names, types, nullability and date formats. Decide whether the reader infers the schema from the data or uses an explicit schema, and record that decision, because inference is convenient in development but can change silently when a source adds a column. You know this step is done when each field in the output has a named type and an owner, and the schema is stored in version control next to the flow definition.
Configure the reader and writer controller service pair
Create a Record Reader that matches the input format, such as CSVReader or JsonTreeReader, and a Record Writer that matches the output format, such as JsonRecordSetWriter or AvroRecordSetWriter. Set the schema access strategy explicitly, point the reader at the schema definition, and check that the header, delimiter, timestamp format and null handling match the sample files. You know this step is done when the controller services are enabled and a test FlowFile read through the reader produces the expected record count with no schema errors.
Add the record reader and parse the input
Place the processor that consumes the source, then the ConvertRecord processor (or UpdateRecord, when only specific fields change), and wire the reader and writer into it. Route the failure relationship to a dedicated error path that keeps the original FlowFile, so malformed input can be inspected rather than dropped. You know this step is done when a sample file passes through with a converted output, and a deliberately malformed sample lands in the failure path with its content intact.
Add QueryRecord for filtering or aggregation
Where the flow needs to filter rows, select or rename fields, or aggregate values, add QueryRecord and write the SQL against the field names defined in the schema. Use the same reader and writer pair or deliberately choose a different writer for each relationship, and add a relationship for each output the flow needs. You know this step is done when a query returns the expected rows for a known sample, and a query that matches no rows produces an output that downstream processors handle correctly.
Test with representative files and inspect the output
Run the full flow against the representative sample set, including the bad records, and compare the output with expected results field by field. Check record counts at each relationship, confirm that types and date formats survived the conversion, and use data provenance to trace a few records through every processor. You know this step is done when the counts reconcile across all relationships, the failure path holds only the records you expected to reject, and the team can explain each output row.
Promote the flow with its schemas and tests
Move the tested flow into a versioned environment, keeping the schemas, reader and writer settings and sample test files together so later changes can be compared against the same inputs. Confirm that controller services resolve their schema references in the target environment and that the failure path has an owner who receives alerts. You know this step is done when a repeat run in the target environment reproduces the test results without manual edits.
When splitting is still the right design in NiFi
Splitting stays the right design when a single record must leave the flow on its own path, when each record needs its own provenance trail, or when the input format has no reader. Record-oriented processing keeps a whole FlowFile together, so it cannot send one row to a different destination on its own. In those cases, split first, then process each part with records: use a RouteOnAttribute or RouteRecord step only where the routing decision truly requires per-record handling, and keep the rest of the transformation in a record reader and writer. Confirm which record-based processors in your NiFi version expose the routing behavior you need before you commit the design.
The cost of splitting grows with volume. Every split creates a new FlowFile with its own content claim, attributes and provenance events, so a file with a million lines can generate a million FlowFiles that each pass through every downstream processor. The queues, the content repository and the provenance repository all absorb that load. Ask your team to measure FlowFile counts and queue depth on a representative file before choosing a per-line design, and size the repositories against that measurement rather than against average file sizes.
A practical pattern is to split by a coarse boundary, such as one batch, one source partition or one hour of data, and then apply ConvertRecord, UpdateRecord or QueryRecord to each chunk. Where a format has no reader, such as a proprietary binary layout, use a custom or scripted step to convert it into a supported format early, then return to record processing. Document the split boundary and the reasoning in the flow's comments so that reviewers can test whether the split is still needed after schemas or source formats change. Confirm the chunk size and provenance requirements with your compliance owner before the design is frozen.
Security and data handling considerations for record flows
Record-based flows move whole datasets through a few components, so one misconfigured reader, writer or SQL statement can affect every record at once. The cards below cover where sensitive data appears in record flows, how schemas and failure routing should be controlled, and which NiFi authorization and provenance settings make the flow auditable.
Sensitive fields in writer output
A record reader keeps every field it parses, so a writer will emit personal or account data unless the flow removes it first. Check each ConvertRecord and UpdateRecord step for fields that should be dropped, masked or hashed before the RecordSetWriter runs. Confirm that downstream destinations accept the full field set, and review sample output from each writer type (CSV, JSON, Avro, Parquet) because each one serializes the same record differently.
Schema files as controlled artifacts
An explicit Avro or JSON schema defines which fields exist, their types and whether they are nullable. When a schema file changes, a flow can silently drop columns or fail on type mismatches. Store schema files in the same Git repository as the flow definitions, require review for changes, and tie each schema version to the flow version that uses it so a rollback restores both together.
Malformed records routed to failure
A record reader that hits a malformed record can send the entire FlowFile to the failure relationship, which may hold valid records alongside bad ones. Decide whether the flow should reject the whole batch or split out bad records for review. Route failure output to a controlled queue with a retention limit, and make sure the failure path does not write partial data to a production target.
Provenance of transformed content
Record transformations change content, so the original and transformed FlowFiles can look alike in a downstream system. NiFi provenance records each transformation event, which helps show what was changed and when. Confirm that the provenance repository retains enough history for your audit window, and that access to provenance data is limited to roles allowed to see the underlying record content.
Controller service access in authorization
Record readers and writers are controller services, and a user who can change a controller service can change how every processor using it parses or writes data. Review the policies for controller services in the NiFi authorization model, separate the roles that build flows from those that enable and configure shared services, and check which identities can view or change the schemas and writer settings.
Sensitive values in QueryRecord SQL
QueryRecord statements are stored in the processor configuration and appear in the flow definition, so a literal account number or token written into a SQL filter is visible to anyone who can read the flow. Use parameter contexts for sensitive values, keep SQL free of credentials and personal identifiers, and check that the flow definition committed to version control does not expose them.
Questions teams ask before building record flows
Can our team build record-oriented flows in NiFi without outside help?
Yes, many teams can, provided they already know NiFi basics such as processors, relationships, controller services and FlowFile attributes. Record flows add a layer of concepts: a Record Reader parses incoming content into records using a schema, and a Record Writer serializes them back out. The learning curve is mostly in schema handling and in testing transformations against real sample data. Teams without that experience usually benefit from a short design review before the first production flow, so that schema choices and error routing are settled early. Ask whether your engineers have built at least one flow with ConvertRecord and a JSON or CSV reader, and whether anyone owns the schema definitions after the build is finished.
What makes a record flow take more effort to build?
Four factors drive most of the effort. Format variety matters most: each additional format (CSV with irregular headers, nested JSON, Avro, Parquet, XML, Excel) adds reader configuration and test cases. Schema quality comes next, since inferred schemas on messy data need correction before they are trusted. Volume affects how you tune batching and whether you need a split or merge step at all. Finally, existing split-based flows add migration work, because each one must be compared with its record-based replacement before cutover. Effort rises fastest when many sources feed one shared schema. Ask any provider to break its estimate down by format, source and destination.
What should a partner scope include for record flow work?
A written scope should name each source and destination, the formats involved, and the schema approach for each: inferred, explicit, or resolved from an external registry. It should list the flows to be built, the QueryRecord or UpdateRecord logic they need, and how errors are routed to failure relationships with a clear retry or quarantine path. Testing should be defined, including sample files, expected output comparisons and volume tests. Deliverables should cover exported flow definitions, parameter usage, documentation of schemas and a handover session for your operators. The scope should also state what is excluded, such as changes to upstream producers. DeployOpen works from written scopes agreed before work starts.
How long does a record flow project usually take?
Duration depends mainly on how many flows are in scope and how deep the testing goes. A small set of flows with one or two formats and a clear schema is a different project from a platform-wide migration of many split-per-line flows with several sources. Testing depth changes the timeline more than most teams expect, because schema drift, malformed records and volume behavior only appear under realistic load. No general duration applies, so ask for a timeline tied to a flow inventory and a testing plan. A provider should be able to explain which assumptions would move the schedule.
Does Apache NiFi have a license fee for record processing?
No. Apache NiFi is an Apache Software Foundation project released under the Apache License 2.0, and the project itself offers no paid edition. Record readers, writers and processors such as QueryRecord are part of the same open-source distribution. Costs you will see are the infrastructure to run NiFi, staff time to build and operate flows, and any services you buy for planning, implementation or support. Check the NiFi download page for current release information before planning upgrades. Confirm with your legal team how the license applies to any custom processors you package and distribute internally.
How do we migrate split-per-line flows to record flows safely?
Migrate in parallel rather than switching a flow in one step. Start by inventorying every split-per-line flow, its input format, its output consumers and its error handling. Build the record-based replacement and feed both versions the same sample data. Compare outputs field by field, including null handling, date formats and ordering. Then route a small share of live traffic to the new flow or run both in shadow mode, and keep the original flow available for rollback until consumers confirm the output. Use provenance to trace records through both paths during the comparison period. Document each schema change so downstream teams can react before cutover.
When do we still need to split files in NiFi?
Splitting remains useful when a destination requires one message per record, when an upstream system delivers files that cannot be parsed as records, or when a step can only act on individual FlowFiles, such as calling an external API per item. Splitting also helps when you need to isolate a single bad record for error handling. The engineering question is whether the split is a deliberate boundary or a habit left over from line-based processing. Keep the split at the edge where it is needed, and keep the rest of the flow record-based. Review each split during design so that it does not multiply FlowFile counts without a reason.
How DeployOpen scopes record-oriented NiFi work
Record-oriented NiFi work starts with a written scope agreed before any flow is changed. DeployOpen fixes the scope, deliverables and timeline in writing, and for this topic the scope names the source and destination systems, the file formats in play (CSV, JSON, Avro, Parquet, XML or Excel), the record readers and writers each flow will use, and the outputs that must be produced. Most of the cost and schedule risk sits in this document. If a flow needs a schema registry lookup, a Parquet writer or a SQL transformation through QueryRecord, the scope should say so, because each one adds design and test effort that a split-per-line rewrite does not.
Before building, DeployOpen inventories the existing flows and collects representative sample data. The inventory records every processor that splits, parses or converts content, the schemas currently assumed downstream, and where header rows, nulls, date formats and encodings vary between producers. Sample files should include the awkward cases: malformed rows, schema drift, empty files and the largest batch the source produces. Implementation then runs inside your own accounts, clusters or data centers, with schemas for each reader and writer documented and flow standards applied to the new process groups. DeployOpen tests the flows against those representative files and prepares the test evidence and schema documentation for your team to keep.
DeployOpen prepares this documentation and test evidence for your compliance and audit teams, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies perform those assessments. Ongoing schema changes, NiFi version upgrades and new producers are normally handled through a retainer for support, rather than a new project each time. Ask any provider, DeployOpen included, to state in the quote which reader and writer schemas are included, who owns the sample data and test files after handover, and how schema changes from upstream teams will be reviewed.
How DeployOpen works on this
DeployOpen plans and builds NiFi record processing inside your own infrastructure, starting from a written scope that names the record formats, schemas and destinations involved. The work covers flow design, testing against representative files, documentation and handover to your team, with ongoing support available as a retainer.
How we scope
- Inventory the source formats, such as CSV, JSON, Avro, Parquet, XML and Excel, and the destinations each flow feeds.
- Decide with you where explicit schemas are required and where schema inference is acceptable for a given feed.
- Record in the written scope which flows use record readers and writers, which still need splitting, and what is out of scope.
How we build
- Design ConvertRecord, UpdateRecord and QueryRecord flows with controller services configured for each reader and writer.
- Test each flow against representative sample files, including malformed rows, missing fields and unexpected types.
- Keep flow definitions in version control and parameterize schemas and paths so environments can be promoted cleanly.
How we hand over
- Document each schema, reader and writer setting, and the reasoning behind every split or merge decision.
- Walk your operators through error relationships, record failure routing and how to diagnose a bad record in production.
- Deliver the flows, test inputs and runbook as one package that your engineers can run and change without us.
How we support
- Offer ongoing support, schema change handling and NiFi upgrade work as a retainer that follows the agreed scope.
- Review record flows again when source formats change, when schemas evolve or when a NiFi 2.x release changes components.
- Confirm with your team which support hours, response paths and change process apply before work starts.
