Which NiFi processor should reshape your JSON?
JSON from APIs and event sources rarely arrives in the shape your target expects. Field names differ, nested objects need flattening, and some records must be dropped before they reach a database, search index or SIEM. The Apache NiFi project, released under the Apache License 2.0, provides five processors for this work: JoltTransformRecord, JoltTransformJSON, JSLTTransformJSON, UpdateRecord and QueryRecord. Each one fits a different kind of change, so the first decision is the shape of the change, not the processor brand. Use Jolt when the job is a structural remap of a known document, JSLT when you need expressions, conditionals or functions over the JSON, and the record processors when the data is already handled as records with a schema and you are changing values, selecting rows or filtering.
Jolt and JSLT are both JSON-to-JSON transforms, and the difference is mostly in how the logic reads. Jolt specifications are declarative spec maps that move, rename and default fields, which suits a mapping a reviewer can check line by line. JSLT expressions are shorter for conditional logic and computed fields, but a long expression can be harder to review than a spec. The *JoltTransformRecord* and *UpdateRecord* processors work on records through a configured reader and writer, so they avoid parsing each FlowFile as a standalone document. Before you commit, confirm with your team which format your reviewers can maintain, then run one representative sample from each candidate through the processor in a test flow and compare the output by hand. Measure throughput in your own environment, since NiFi performance depends on the flow, hardware and record size.
How each processor fits a JSON transform
This table compares JoltTransformJSON, JoltTransformRecord, JSLTTransformJSON, UpdateRecord and QueryRecord on how they read input, which transform language they use, and where each fits. Read each row as a question to confirm against the processor documentation for your NiFi version, since reader and property details can differ. The main takeaway is that FlowFile-level processors suit whole-document reshaping, while record processors suit tabular or repeated structures.
| Processor | Input handling | Transform language and typical fit | Testing approach |
|---|---|---|---|
| JoltTransformJSON | Operates on the whole FlowFile content as JSON, with no record reader | Jolt specification; structural restructuring such as moving, nesting and renaming keys | Run the spec against sample JSON files outside NiFi, then confirm output in a test flow |
| JoltTransformRecord | Uses a record reader and writer, so it handles many records per FlowFile | Jolt specification applied per record; suited to restructuring repeated records | Test the spec on sample records, and confirm reader and writer schema handling in the processor docs |
| JSLTTransformJSON | Operates on FlowFile content as JSON, with no record reader | JSLT expressions; JSON-path style edits, field selection and computed values | Keep expressions in version control and test with representative documents before deployment |
| UpdateRecord | Uses a record reader and writer and applies changes to records in place | NiFi Record Path expressions; targeted field updates and value changes | Verify Record Path results on sample records and confirm null and type behavior in the docs |
| QueryRecord | Uses a record reader and writer and exposes one relationship per query | SQL over record sets; filtering, aggregation, renaming and routing by condition | Test queries against sample batches and confirm supported SQL features and limits for large inputs |
How Jolt and JSLT differ in practice
Jolt is specification-driven: you describe the output shape as a JSON document whose keys mark what to keep, move, default or remove. The shift operation copies and renames fields, default fills in values that are missing, and remove drops fields you do not want to pass on. This suits flat reshaping and simple renames well, because each rule maps one input path to one output path. The trade-off appears with deep nesting. A specification that reaches several levels down, with wildcards and references back to matched keys, becomes hard to read in review, and a small mistake in one key can silently move data to the wrong place. Before you commit to Jolt for a complex mapping, have someone who did not write it trace three or four sample documents through the specification by hand.
JSLT takes the opposite approach. You write a template-like expression that builds the output object, using constructs that look much like the result you want, with conditionals, loops and functions where needed. When the target document is itself nested, or when several output fields are computed from the same inputs, JSLT is usually easier to read because the output structure is visible in the expression. It is also easier to explain to a colleague who knows the target schema but not the transformation tool. The cost is that expressions grow in logic rather than in structure, so a long template with many conditionals can become a small program that needs its own tests. Confirm the JSLT version and syntax documentation that your NiFi 2 release bundles before you rely on a specific function.
The record-based processors, JoltTransformRecord, UpdateRecord and QueryRecord, work on records rather than raw JSON text, so they depend on the configured Record Reader and Record Writer controller services. The reader decides how the incoming bytes become fields and which types they carry, and the writer decides how the result is serialized. This changes schema handling in two ways. First, a schema defined or inferred in the reader determines whether a field is missing, null or wrongly typed before the transformation runs. Second, a transformation that changes field names or types must agree with the writer's schema, or the output will fail or lose columns. Test each record processor with a representative sample that includes missing fields, nulls and unexpected types, and check the output after the writer, not only the transformed fields in the tree.
Which transform fits which change?
Most JSON reshaping work falls into a few recurring changes: renaming fields, flattening arrays, filtering, aggregating and coercing types. Each change points to a different processor, and the right choice depends on whether the input is one document or many records and on how well your team reads Jolt or JSLT specifications. Check the case below before you pick a processor for the flow.
Field renames and restructuring
Moving or renaming keys in a single JSON document is the classic Jolt case. JoltTransformJSON fits when each FlowFile holds one document and the change is a shift or default spec. A Jolt spec is declarative but dense, so the team needs at least one person who can read it. Test the spec against a representative sample file, including documents with missing keys, before it reaches production.
Flattening nested arrays
Turning an array of line items into one flat record per item usually calls for splitting, not a single reshape. JoltTransformRecord can restructure records, but a deep flatten across several levels is easier to read in JSLT, where a query can project the nested values directly. Check how the output handles empty arrays and nulls, since those edge cases decide whether downstream tables stay consistent.
Conditional filtering
Dropping or routing records by value is a row-level decision. QueryRecord handles it with SQL and gives each condition its own relationship, which makes routing rules easy to review. Use UpdateRecord only for changing values in place. Filtering in a Jolt spec is possible but awkward, so ask whether the rule will be maintained by people who do not write Jolt.
Aggregation across records
Totals, counts and group-by results need records to be read together, which a per-document transform cannot do. QueryRecord runs aggregate SQL over a FlowFile's records, but its limits on large inputs matter, so test with realistic batch sizes. If many FlowFiles must be combined, the merge processors are the better tool and should be designed in a separate step.
Type coercion on records
Converting strings to numbers, dates or booleans is a schema question as much as a transform question. UpdateRecord with a record path and a configured record reader and writer keeps types explicit, and it works on multi-record input. Confirm the schema source and how invalid values are handled, because a silent cast can hide bad data that should be rejected.
Single document versus multi-record input
The input shape often decides the processor more than the change does. Single documents with a fixed structure suit JoltTransformJSON or JSLTTransformJSON. Batches of records with a schema suit the record processors, which avoid parsing each document separately and use the configured reader and writer. Confirm the input shape with the source owner before choosing, because a mixed feed may need both paths.
How to build and test a JSON transform in NiFi
This sequence takes a JSON reshaping requirement from sample payloads to a tested flow that routes bad input to a failure path. Each step closes with a check you can run before moving on, so the transform is proven on real data rather than assumed to work.
Capture representative sample payloads
Collect real payloads from the source system, including optional fields, nulls, arrays with zero, one and many items, and a few malformed messages. Your integration owner or the source team supplies them, and your own team stores them as test fixtures with the source version noted. Load them with GetFile or a test ListenHTTP route, then confirm each sample parses with a JSON reader and that the set covers every field the target needs. The step is done when the fixture set includes both well-formed and rejected cases.
Choose the processor for the shape change
Match the transform to the job. JoltTransformJSON works on single JSON documents and suits flat or nested restructuring with a declarative spec. JoltTransformRecord applies the same spec to each record in a larger batch. JSLTTransformJSON uses a JSON query and transform language that suits expressions, conditionals and reshaping with less spec nesting. UpdateRecord changes individual field values with record paths, and QueryRecord filters rows or renames fields with SQL. Write down the choice and the reason, and note whether the team that will maintain the flow can read the chosen language. The decision is recorded when the team can explain why the other options were rejected.
Write the spec or expression against the fixtures
Write the Jolt specification or JSLT expression outside the flow first, using a JSON editor or a unit test harness, and run it against each fixture. For Jolt, define shift, default and remove operations so unmapped fields are dropped on purpose. For JSLT, guard optional fields explicitly. Compare each output with the expected document, keeping the expected files under version control next to the spec. The step is complete when every fixture produces its expected output and the outputs match the target contract field for field.
Configure readers, writers and the schema
For record processors, set a JsonTreeReader and a JsonRecordSetWriter, or the writer type your target expects. Decide whether the schema is inferred or supplied as an Avro schema through an AvroSchemaRegistry or a schema text property. Use an explicit schema for production flows so that a source change surfaces as an error rather than a silent change in output. Check the output by running a test FlowFile through the processor and inspecting the content, confirming the field names, types and nesting match the target contract. Confirm this with the target owner, since schema naming decisions are theirs to approve.
Wire success, failure and parse paths
Connect the success relationship to the next step and route failure to a dedicated path. Record-based processors send unparseable input to failure, and a Jolt or JSLT error on one document does not stop the rest of the batch when you split the input first. Send failures to a connection that is monitored and that holds FlowFiles with their original content and error attributes. Keep the failure path separate from the retry logic, which is covered in the error handling pages. The step is done when every relationship is connected or deliberately auto-terminated with a reason noted on the canvas.
Run malformed input through the failure path
Feed the malformed fixtures and a payload with an unexpected type, such as a string where an array is expected, into the flow. Confirm that each one lands on the failure connection, that the original content is preserved, and that the error message is present in the FlowFile attributes or the processor bulletin. Confirm also that valid records in the same batch still reach success. The check is complete when no malformed fixture reaches the target and no valid fixture is lost.
Verify output with provenance and test FlowFiles
Run a set of valid fixtures through the full flow and open provenance for a sample FlowFile. Compare the content at the transform's output event with the expected document, and confirm the transform event shows the processor you chose. Check that the attributes carried through the flow match what the next processor expects. Keep the fixture run as a repeatable test so that a spec change, a NiFi upgrade or a source change can be checked again. The step is done when the output matches the expected files and the provenance trail confirms each record's path.
Readability, performance and testing trade-offs
Readability decides who can review, change and debug a transformation after the original author has moved on. Jolt specifications are JSON documents built from operation blocks such as shift, default and remove, which makes them compact but unfamiliar to developers who have not used Jolt before. JSLT expressions read more like a small functional query language, which many developers find easier to follow for conditional logic. UpdateRecord and QueryRecord are often the easiest to review for analysts and data engineers because they use record paths or SQL that resemble familiar syntax. Plan for the mix of skills on your team rather than assuming every maintainer knows the same language.
- Specification size: Long Jolt specs with nested shift blocks become hard to scan. Split them by output section and document each block in a comment-bearing README kept with the flow.
- Language familiarity: Confirm which team members already know Jolt or JSLT before standardizing on one of them.
- Logic placement: Keep conditional business rules in one place. Mixing rules across Jolt, UpdateRecord and a downstream script makes debugging slow.
- Naming: Use field names in the output that match the target schema so reviewers can compare input and output without decoding the transform.
- Handover material: Record which processor owns each field so the next maintainer knows where to edit.
Check that at least two people on your team can explain each transformation in a review before it goes to production.
Throughput and memory behavior depend on how each processor reads the payload. Processors that parse the whole document into memory handle deeply nested or very large JSON with more memory pressure than those that stream records, so the size and depth of your documents matter as much as the processor choice. Record-based processors such as JoltTransformRecord, UpdateRecord and QueryRecord operate on batches of records through a record reader and writer, which suits many small records but can be costly when one document is very large. Performance depends on payload size, record shape and cluster sizing, so the numbers you see in a test with sample files may not hold in production.
- Whole-document parsing: JoltTransformJSON and JSLTTransformJSON work on one FlowFile at a time, so very large single documents need enough heap and a sensible content size per FlowFile.
- Record-level processing: JoltTransformRecord and UpdateRecord apply the transform per record, which helps when one FlowFile holds many records.
- Nesting depth: Deep recursive transforms cost more CPU than flat ones. Measure with your deepest real documents, not the average.
- Chained processors: Each extra transform step adds reads and writes of content, so combining steps in one specification can reduce work.
- Cluster effects: Load balancing and node count change the effective throughput, so retest after any cluster change.
Benchmark each candidate with representative payload sizes and nesting depths from your own sources, and record heap use and processing time per FlowFile.
Testing turns a transformation into something you can change safely. The practical approach is to store input samples, expected outputs and the transformation definition together in version control, then run the flow or the transform logic against those fixtures before each change. Keep fixtures small enough to review, but include the edge cases that matter: missing fields, null values, arrays with zero or many items, unexpected types and deeply nested objects. Compare outputs with a structural check rather than a byte comparison when key order or whitespace may vary.
- Fixture set: Keep one input file and one expected output file per case, named by scenario, such as
missing-address.json. - Edge cases: Cover empty arrays, absent optional fields and type mismatches, since these are where transforms usually fail silently.
- Specification files: Store Jolt specs and JSLT expressions as separate files under version control so changes appear in diffs.
- Automated comparison: Use a test harness that parses both outputs and compares structure, so harmless formatting changes do not fail the build.
- Flow-level checks: Verify routing on failure and on malformed input, not only on successful transforms.
Before promoting a change, confirm that the fixtures were reviewed by someone other than the author and that the expected outputs match the target schema.
Security and operational risks in JSON transforms
A JSON transform is code that touches live data, so its risks sit in three places: the payloads it handles, the people who can change it, and the records that show what it did. The cards below cover where sensitive content can leak, how to control edits to transform specs, and how to keep definitions under review like any other production change.
Sensitive fields in FlowFile content
A Jolt or JSLT spec runs against the full FlowFile content, so every field it reads or writes is exposed to whoever can see that data in a queue. Identify which fields are personal, financial or credential data before you pick a transform, and confirm whether those fields should be removed or masked early in the flow. Check that queued FlowFiles holding those fields are covered by your access and retention rules, since a transform that copies a field into a new attribute can widen its exposure.
Provenance holds transformed content
NiFi provenance records events for each FlowFile, and content claims can remain available for replay until they age out of the content repository. Anything you transform is therefore visible in provenance for as long as its content is retained, even if the downstream system never stores it. Ask who can view provenance and content, and set retention for both to match the sensitivity of the source data rather than the convenience of debugging.
Restricting who can edit specs
A transform spec changes what data leaves the flow, so editing it is a privileged action. Limit edit rights on processors that hold Jolt or JSLT specs to a small group, and separate that group from people who only start, stop or view flows. Confirm which NiFi authorization policies apply to the processor, the process group and the parameter contexts that feed the spec, because access to any of them can change the output.
Sensitive values in spec parameters
Specs that embed API keys, tokens or masking secrets should not hold those values as plain text in the processor configuration. Use sensitive parameters or a supported secrets source, and confirm how the value appears in exported flow definitions and in the UI for users with read access. A spec that reveals a masking key or salt can make the masking worthless, so review each spec for values that must stay out of version control.
Logging without leaking payloads
Bulletins, log lines and the debug output you add during testing can carry record values into places with wider access than the data itself. Keep log messages to FlowFile identifiers, counts and error types, and avoid logging the transformed body or the original field values. Before a flow goes live, check the log level, bulletin display and any custom logging you added, and confirm the destination log store meets the same access rules as the source system.
Change control for transform definitions
A spec that changes field mapping or drops a field can break downstream consumers without any error in NiFi. Keep each Jolt or JSLT definition, and the flow version that contains it, in a reviewed change process with a named approver and a recorded reason. Use Git-based flow versioning where your environment supports it, and test each spec change against representative sample payloads before promotion, so the reviewer sees exactly what the output will look like.
Questions teams ask before building JSON transforms
Can our team build JSON transforms in NiFi without outside help?
Yes, if your team already runs NiFi and has someone comfortable with Jolt specifications, JSLT expressions and NiFi record readers and writers. Most in-house gaps are not in the syntax but in testing discipline: saving real payloads, comparing outputs and reviewing changes before they reach production. Outside help tends to pay off when the transform set is large, the source payloads are poorly documented, or the team has no one to own the flow after handover. Confirm your team's skills against the specific processors you plan to use before deciding.
How long does a JSON transform build usually take?
The duration depends on drivers, not a fixed figure. The main ones are the number of distinct payload shapes, how many output schemas must be produced, how much logic sits in the transform itself (renames, conditionals, array handling, lookups elsewhere in the flow), how consistent the source documentation is, and how many systems must accept the output for testing. Transforms that map one stable source to one target move quickly. Transforms with undocumented variants, nested arrays and several downstream consumers take longer, mainly because discovery and test cases grow. Ask for a timeline in the written scope, tied to these drivers.
What cost components apply in a fixed-scope JSON transform engagement?
A fixed-scope engagement usually has a one-time cost for discovery, transform design, flow build, test automation and handover, and an optional recurring cost for support, retainers or changes after go-live. The one-time part is driven by the number of payload variants and output targets. Recurring work depends on how often source schemas change and how much your team wants retained support. Ask the quote to list each deliverable, the environments it covers, what is excluded (such as changes to the source system), and which items are fixed versus billed by change request. DeployOpen does not publish prices; a quote follows from the written scope.
What does DeployOpen need from our team before a transform build starts?
The build needs representative input and access. Typically that means:
- Sample payloads from each source, including edge cases such as missing fields, nulls, arrays with one item and unexpected types
- Any existing schemas, API documentation or contract tests for the source and target
- A written description of the expected output for each consumer
- Test access to a non-production NiFi instance and to the source and target systems or stubs
- A named owner who can answer field-level questions during discovery
Samples from production should be anonymized or masked where personal data is involved. Confirm data handling terms in the scope before sharing any production records.
How are NiFi JSON transforms tested before handover?
Testing starts with fixed input and expected output pairs for each transform, stored alongside the flow definition so they can be rerun after any change. For Jolt and JSLT, the transform specification can be exercised against those pairs outside the full flow, and record processors can be tested inside a test process group with sample FlowFiles. Good test sets include the edge-case payloads from discovery and a few deliberately malformed inputs to confirm that failure relationships route data as designed. Handover normally includes the test suite, a runbook for changing transform specifications, and version control of flows. Ask how the flows will be versioned, since NiFi 2 supports Git-based Flow Registry Clients for GitHub and GitLab.
Which NiFi version should a new JSON transform project target?
Target NiFi 2. The NiFi downloads page states that NiFi 1.28 is the last minor release of the version 1 series, and that 1.28.1 reached end of support on December 8, 2024. The project also explains that 1.x cannot be maintained further because it depends on Jetty 9.4, Spring Framework 5.3 and AngularJS 1.8. Building new transforms on 1.x creates a future migration. Confirm the current release and the Java 21 runtime requirement in the NiFi administration guide before planning the environment, and check the processor names in your version because some record processors have changed between releases.
Who owns the transform after the engagement ends?
Ownership should be written into the scope before work begins. In a typical handover, your team owns the flow definitions, transform specifications, test data and the runbook, and can run the tests and change the specifications without the partner. Ask which artifacts will be delivered, in which repository, under which license, and whether the test suite runs in your CI environment. Ongoing support, upgrades and changes after handover are available as separate retainers or change requests, so confirm which of these are in scope and which are not.
How DeployOpen scopes JSON transformation work
DeployOpen scopes JSON transformation work as a fixed-scope engagement agreed in writing before any flow is built. The written scope lists which transforms are in play (for example, a Jolt specification in JoltTransformRecord, a JSLT expression in JSLTTransformJSON, or UpdateRecord and QueryRecord steps), which source and target schemas they connect, how many distinct payload shapes must be handled, and which NiFi environments will carry the work. DeployOpen plans, implements, tests and documents these transforms inside your own infrastructure, whether that is your cloud accounts, data centers or an air-gapped network, so the flow definitions, test fixtures and sample payloads stay under your control.
Testing and documentation are part of the fixed scope, not an afterthought. Each transform is verified against representative input records, including malformed and edge-case payloads, and the expected output is recorded alongside the flow so that a later change can be checked against it. The deliverables typically include the flow versions, transform specifications, test records and an operations note explaining how to update them. Ask any partner to name these deliverables and the acceptance criteria for each, and to state what happens when your source payloads change after sign-off. Ongoing support, upgrades and continuous testing of the transforms are available as optional retainers, which cover changes after the initial build rather than the build itself.
DeployOpen does not publish a price list. A quote follows from the written scope, so the inputs that move the number are the same ones you should specify: the count and complexity of transforms, the number of source and target systems, the volume and variety of payload shapes, the testing depth you require, and whether you want a retainer after go-live. When you request a quote, ask the provider to separate one-time build effort from any recurring support, to state which items are excluded, and to confirm which NiFi version the work targets, since that affects the transform components available to you.
