How Apache NiFi data validation keeps bad records out of targets
Inbound files and messages from partners and internal systems break targets when a required field is missing, a number arrives as text, or a date follows a format nobody documented. Validating data inside Apache NiFi, before it reaches a database, warehouse or API, lets you stop those records at the edge of the flow instead of discovering them in a failed load or a silently corrupted table. Teams need this pattern when they accept data they do not control, when several sources feed one target, or when a downstream load is expensive to roll back. The processor you choose depends on the content: ValidateJson checks JSON text against a schema, ValidateCsv checks delimited content against a column definition, and ValidateRecord checks structured records read through a record reader, which covers most other formats.
ValidateCsv validates FlowFile content or an attribute value against a user-specified CSV schema, so it suits flat files where column count, order and value types matter. Before you build it, decide where the schema comes from, whether a maintained definition owned by the data provider or one derived from a reviewed sample, and who approves changes to it. Confirm in the processor documentation which validation behaviors and relationships your NiFi version exposes, and test them with real rejected samples from the partner rather than assumed ones. For record-based content, the same questions apply to the schema registry or inline schema that ValidateRecord uses, and to whether type coercion is strict or lenient. Lenient handling can pass a value such as "12.50" into a decimal column, which is convenient until a downstream report disagrees with the source.
Choosing ValidateRecord, ValidateJson or ValidateCsv
Use ValidateRecord when the incoming FlowFile is structured data that NiFi can read through a record reader, such as CSV, JSON, Avro or other formats for which a reader controller service exists. The processor pairs a reader with a schema and checks each record against it, so one flow can validate the same logical data regardless of the wire format the sender uses. The schema can come from the reader (for example, a schema text property, a schema registry or schema inferred from the content), and the choice determines who owns the contract.
- Reader decides the parse: Records that the reader cannot parse fail before schema checks run, so reader settings such as header handling and delimiter must match the sender.
- Schema source: Confirm whether the schema is explicit, registry-backed or inferred, and whether inference is acceptable for a production contract.
- Strict versus lenient types: Confirm in the component documentation for your version how the processor treats values that can be coerced, such as numeric strings, and whether nulls are accepted for required fields.
- Routing: Valid and invalid outputs are separate relationships, so invalid records can be sent to a reject path with their original content intact.
- Record-level reporting: Check what error detail is written to attributes or content so rejects can be explained to the sender.
Before relying on any of this, confirm the exact property names and relationships in the NiFi component documentation for your version.
Use ValidateJson when the FlowFile content is a JSON document and the goal is to confirm that it is well-formed and conforms to a schema you supply, without converting the content into records. This suits single-document payloads, webhook bodies and message formats that do not map cleanly to a tabular record model. Because the processor works on the document itself, it is a good first gate for messages that arrive from external systems before any record-oriented processing begins.
- Well-formedness first: Malformed JSON should fail at the gate with a clear reject path rather than surfacing later as a parsing error in a downstream processor.
- Schema supply: Confirm how the schema is provided, whether as inline text, a file path or another supported source, and who maintains it.
- Version control of schemas: Store the schema under change control alongside the flow, so a sender change is reviewed before the flow is updated.
- Scope limits: Nested structures are checked as a whole document, so very large or deeply nested payloads may need splitting before validation.
- Routing and rejects: Keep invalid documents on a separate relationship with enough context for the sender to fix the payload.
Confirm the exact property names and any schema dialect support in the NiFi component documentation for your version.
Use ValidateCsv when you need to check CSV data against an explicit column schema, either for the whole FlowFile content or for a single attribute value. According to the NiFi ValidateCsv documentation, the processor validates FlowFile content or an attribute value against a user-specified CSV schema. This is useful for fixed-layout exports, for fields carried in attributes that were extracted from a file name or header, and for flows where the schema is a documented contract with the sender rather than something inferred from sample data.
- Explicit schema: Define column order, types and constraints in the processor configuration, and review them with the data owner before go-live.
- Content or attribute target: Choose the target deliberately, since attribute validation suits single values while content validation suits full files.
- Header and delimiter alignment: Confirm how the processor treats headers, quoting and delimiters so valid files are not rejected for formatting reasons.
- Failure handling: Route failures to a reject path with the original FlowFile preserved for investigation.
- Pairing with readers: When the same CSV is also read as records later in the flow, check that the reader schema and the ValidateCsv schema agree.
Confirm the exact property names and supported constraint syntax in the ValidateCsv documentation for your NiFi version.
How schema sources and type strictness change what passes
Choose the schema source before you choose any threshold, because the source decides what counts as valid. ValidateRecord and ValidateCsv can take a schema defined directly in the processor, a schema looked up from a schema registry or named reference through a Record Reader, or a schema inferred from the incoming data. An inferred schema is convenient for exploration, but it describes what the file happens to contain, not what the partner agreed to send. If a column is missing or a type changes, inference adapts to the new data and the check passes it. For partner data, use an explicit schema that is version-controlled alongside the flow, so a change on the partner side produces a route to failure instead of a silently different table. ValidateCsv works the same way: it checks content or an attribute against a user-specified CSV schema, so the schema text itself becomes a reviewable contract.
Strict and lenient handling answer different questions. A strict check rejects a record when a value cannot match the declared type exactly, for example text in a numeric field, a date in an unexpected pattern, or a value missing from a required field. Lenient handling may coerce values that a reader can interpret, such as turning numeric text into a number or accepting an extra column, depending on the reader and processor settings. Coercion keeps data flowing but can hide upstream defects, such as a locale-specific decimal separator that quietly changes amounts. Confirm the exact strictness and coercion options in the NiFi component documentation for your NiFi version, and do not assume a default. Record the chosen behavior in the flow change record so reviewers know which mode each route uses.
Test both modes on a representative sample set before production. Build a sample that includes known-good files, boundary values, nulls, wrong types, extra and missing columns, and the malformed rows your partner has actually produced. Run the same sample through the strict and lenient configurations and compare the success and failure relationships, the attributes written to each FlowFile, and the values in the output records. Where coercion changes a value, the test should assert the new value, not only that the record passed. Ask the partner for a sample of real rejects so the test reflects their error patterns. Repeat the test whenever the schema, reader or NiFi version changes, because the behavior of validation settings can differ between releases.
Validation options and when each fits
This table compares the main validation choices for NiFi flows: where the schema comes from, how strictly types are checked, where invalid data goes, and whether the check applies to content or attributes. Read each row against the data you receive, then confirm property names and allowed values in your NiFi 2.x release before you build.
| Option | Fits when | Trade-off | Confirm in your release |
|---|---|---|---|
| Schema from a reader service | Records share one format that a record reader can parse, such as JSON or CSV with a stable header | One schema change in the source can route many records to failure, so the schema needs an owner | Reader and schema access strategy settings for ValidateRecord |
| Schema from a schema text property | The contract is short, fixed and kept in the flow itself, for example for a small CSV export | Changes need a flow edit and redeploy, and the schema can drift from the real source | Schema text property and its accepted formats |
| Strict type checking | Downstream loads into typed columns or APIs that reject a wrong type | Records with harmless variations such as numeric strings can be rejected in large numbers | Strict type checking option and its default |
| Lenient type checking | Sources are known to vary in formatting and the target can coerce values | Bad values can reach the target if coercion is not checked later, so add a downstream test | Coercion behavior and how it treats empty or null values |
| Route invalid records to a separate relationship | You need to keep good records moving while rejects are reviewed or returned | Splitting records adds routing logic and a second path to monitor and test | Valid and invalid relationship names and whether a whole FlowFile fails or only some records |
| Fail the whole FlowFile | A partial load would break a batch, such as a file that must be all or nothing | A single bad row blocks the file, so senders must fix and resend the complete batch | Behavior on a single invalid record in the processor documentation |
| Content validation (ValidateCsv, ValidateJson, ValidateRecord) | The payload itself must match a structure before any parsing downstream | Full content is read for each check, which costs more on large files | Which processor reads the content and its size limits for your files |
| Attribute validation | A value in a header or attribute, such as a file name or source ID, controls routing or acceptance | Attributes do not prove the content is correct, so pair this with a content check | Attribute expression language support and how the value is evaluated |
Setting up validation in a NiFi 2 flow
This sequence turns an agreed data contract into a working validation path in NiFi 2, where each record is either accepted for the target or routed to a reject path that the sender can see. Each step ends with a check so the team knows the flow behaves as designed before it touches production data.
Collect sample good and bad inputs
Gather a representative set of files from the sender, including records you already know are wrong: missing required fields, wrong types, extra columns, bad delimiters and out-of-range values. Your team and the sender's data owner label each sample as valid or invalid and record the reason. The step is done when the labeled set covers every rule you expect to enforce and you can point to at least one known bad row per rule.
Agree the schema with the sender
Write the expected schema as a contract the sender signs off on: field names, order, data types, nullability, date and number formats, and allowed values. Decide which mismatches reject a whole file and which reject only the affected record, and agree who fixes rejects. The step is complete when the sender confirms the schema in writing and the agreed version is stored with the flow, so a later change can be traced to a decision.
Configure the record reader and validator
Set up a record reader that matches the incoming format, then configure the validator: ValidateRecord for record-oriented data, ValidateJson for JSON documents, or ValidateCsv for delimited files against a CSV schema. Point the schema property at a controlled source, choose strict or lenient type handling to match the contract, and set the invalid handling so that failures do not stop the whole batch unless the contract requires it. The check is a test run of the sample set through the processor, with the first few FlowFiles inspected to confirm that the reader parsed them as expected.
Route valid and invalid relationships
Connect the valid relationship to the target path and the invalid relationship to a separate reject path, and leave no relationship unconnected. Add attributes that record the source, the rule that failed and the schema version, so downstream steps can act on them. The check is a run of the labeled sample set where the count on the valid connection matches the accepted rows and the count on the invalid connection matches the known bad rows.
Build the reject file and notification path
Send rejected FlowFiles to a location the sender's team can read, such as an object store prefix or a secured folder, and write a short error report that lists row numbers and reasons. Add a notification step for the owning team, and set an alert if the reject rate passes a threshold you define during contract agreement. The check is that a deliberately bad file produces a reject file, a readable report and a notification, all of which the sender's team can open without access to NiFi.
Test the flow with known bad rows
Run the full labeled sample set through the flow end to end, including the bad rows from the first step, and compare the output against the expected result for every record. Include edge cases such as empty files, a header-only file and a file with an unexpected extra column. The step is complete when every known bad row lands on the reject path, every valid row reaches the target, and no record is lost between the two.
Confirm provenance for rejected FlowFiles
Open the provenance events for a sample of rejected FlowFiles and confirm that each one shows the validator as the routing point, the failure relationship taken and the attributes that explain the reason. Check that the content claim of a rejected file can be viewed and that the record is retained for the period your own policy requires. The check is a short written trace for one reject per rule, which the team keeps with the schema contract as evidence that the validation behaves as agreed.
Reporting rejects back to senders without losing data
Once ValidateRecord, ValidateJson or ValidateCsv has split bad input from good, the reject path decides whether senders can fix their data and whether auditors can trace what happened. The cards below are decisions to settle with the sender and the compliance owner before the flow goes live.
Choose the reject file format
Decide whether rejects return in the sender's original format, in a normalized format, or as a separate report with one row per failed record. Original-format rejects are easiest for senders to correct and resubmit, but they may not carry a reason field. A structured report such as JSON or CSV keeps the reason beside the record ID. Check that the sender's tooling can parse the format you choose, and test it with a sample file containing each failure type, such as a missing mandatory field and a wrong type.
Decide when to send a summary
Settle whether the sender receives a notice for every file, only for files over a rejection threshold, or once per batch on a schedule. A per-file notice gives fast feedback but can flood a sender who submits many small files. A threshold notice avoids noise but can hide a steady trickle of errors. Agree the threshold and the delivery channel in writing, and name who receives the notice when the sender's contact is unavailable.
Include the failure reason
Each rejected record should carry the validation message that explains why it failed, such as the field name, the expected type and the schema version used. Keep messages specific enough for a sender to fix the source system, but avoid echoing internal schema locations or stack traces that expose platform details. Ask the sender which fields they can act on, and confirm that the messages are understandable to a non-NiFi audience.
Retain rejected content for investigation
Rejected content often contains personal or payment data, so retention is a compliance decision, not only an operations one. Agree how long rejects are kept, where they are stored, who can read them and how they are deleted at the end of that period. Confirm the retention period with the compliance owner and your records policy, since this document does not set one. Restrict access in the storage layer as well as in the NiFi flow.
Stop rejects looping back in
A rejected file that is resent unchanged or re-ingested by the same flow can create endless retries and duplicate alerts. Route rejects to a separate destination rather than back to the input queue, and tag each file with an identifier that marks it as a resubmission. Only accept corrected files through a defined channel, and check that the validation step does not re-process files already marked as rejected.
Questions teams ask before validating data in NiFi
What drives the cost of validating data in NiFi flows?
The main drivers are the number of sources, how complex each schema is, how rejects are handled and how much testing the flows need. Each additional sender usually adds schema work, sample data analysis and a route for rejects. Nested or variable records, multiple CSV dialects and schemas that change often all add effort. Costs are one-time for the initial build and recur when schemas change or new senders are onboarded. Ask any partner for a quote that itemizes these drivers against your written scope, and ask who is responsible for schema changes after go-live.
How long does a NiFi data validation project take?
Duration depends on the scope agreed in writing, not on a fixed template. The scope fixes the number of sources and flows, the schema formats, the reject handling and the test coverage. A project with a few well-documented senders will run far shorter than one covering many formats and unclear ownership. We do not publish timelines; a fixed-scope proposal sets the deliverables and the schedule before work starts. Ask for milestones that tie to schema sign-off, test results and handover.
What does our team need to provide before validation work starts?
Your team needs to provide representative sample data for each source, including good and bad records, and named schema owners who can confirm field types, required fields and allowed values. You also need contacts at each sending system who can receive reject reports and fix upstream issues. Access to a non-production NiFi environment and a decision on how rejects will be reported back to senders are needed early. Missing samples are the most common cause of rework, so ask for this list in writing as part of the proposal.
Can an internal team build NiFi validation flows without outside help?
Yes, a team with NiFi experience can build these flows internally, especially for a small number of stable sources. The processors involved, ValidateRecord, ValidateJson and ValidateCsv, are documented in the NiFi components reference, and the work is mostly schema definition, routing and testing. The harder parts are usually strict versus lenient type decisions, reject reporting to external senders and keeping tests current as schemas change. An outside partner can add value when deadlines are tight or when the team lacks NiFi operations skills. Decide this by comparing skills and capacity against the source count.
Which validation processor should we use for CSV files?
Use ValidateCsv when the input is CSV and you can describe its columns and rules in a schema. ValidateCsv checks FlowFile content or an attribute value against a user-specified CSV schema, so it suits fixed-layout exports. Record-based inputs such as JSON or other structured formats are better handled with ValidateRecord, and JSON-specific checks with ValidateJson. Confirm the exact property names and behavior against the processor documentation for your NiFi version before you rely on them.
How do we confirm validation rules are safe for sensitive data?
Treat validation as a data handling step, not only a format check. Limit who can view reject files that contain personal or financial values, and set provenance and queue access in line with your access model. Validation errors can echo field contents into logs or alerts, so review what each processor writes and where. Confirm with your security team how rejects are retained and deleted. Any statement about compliance outcomes must come from your independent auditor, since DeployOpen prepares controls and evidence but does not issue audit reports.
What should we ask a partner to include in a validation proposal?
Ask for a written scope listing each source, its format, the schema owner and the reject path. The proposal should name the NiFi version and deployment target, the test approach with sample datasets, and the handover materials such as flow documentation and runbooks. It should also state what is one-time and what recurs, including support for schema changes after go-live. Request a quote that separates build, testing and ongoing support so you can compare options with your internal estimate.
How DeployOpen scopes and delivers NiFi validation
DeployOpen plans, builds, documents and supports validation stages in Apache NiFi flows inside infrastructure you control. Work starts from a written scope that names your sources, schemas, reject handling and targets, so validation rules can be tested before they guard production data.
How we scope
- Inventory each source feed, its format and the schema owner who approves changes
- Agree in writing which checks use ValidateRecord, ValidateJson or ValidateCsv and where each sits
- Define strict and lenient type rules per field, with the reject outcome for each failure
- Confirm the reject report format, recipients and retention before any build starts
How we build
- Wire schema sources such as record readers and controller services into versioned flow groups
- Route invalid records to a quarantine path with error attributes that identify the failed rule
- Test valid, malformed and boundary samples from your own feeds against each validation stage
- Keep flows in a Git-based registry client so validation changes go through review
How we hand over
- Deliver flow documentation covering schemas, rule intent, routing paths and reject fields
- Provide test datasets and the procedure used to confirm each rule behaves as written
- Walk your operators through reject triage, schema change requests and reprocessing steps
- Record configuration, parameters and sensitive value handling in the handover pack
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover
- Retainers can cover schema updates from source owners and re-testing after NiFi upgrades
- Ask in your quote which environments, reject reports and alert channels are included
- Fixed-scope engagements keep deliverables and timelines agreed in writing before work starts
