Apache NiFi and Apache Iceberg integration

Apache NiFi and Apache Iceberg integration with PutIcebergRecord

Your operational sources already produce records that NiFi collects, and the lakehouse team needs those records in Iceberg tables sitting on object storage. The supported path in NiFi is the PutIcebergRecord processor, which writes records through a REST catalog controller service and a file IO provider matched to where your data lives: S3, GCS or ADLS. Before you design the flow, confirm the exact property names, the catalog options and the Iceberg version supported by the NiFi release you plan to run, because these details are set in the component documentation for that release and can change between versions. The Apache NiFi download page is the starting point for matching your NiFi version to its current component set.

The decisions that shape this integration are made before any processor is placed on the canvas. Choose the catalog that owns table metadata and confirm that NiFi's service account can create, commit and read tables in the target namespace. Pick the file IO provider that matches your storage account, and scope its credentials to the bucket or container prefixes the flow writes to. Decide the record reader that parses incoming FlowFiles and whether its schema matches the Iceberg table schema, since a mismatch is one of the first failures teams see. Also decide how partition values are derived from record fields, because partitioning is fixed once a table is created and is costly to change later.

Plan for failure as part of the design. Commit conflicts, catalog outages and schema drift each leave data in a different state, so test each path against a non-production table with the same catalog and storage configuration as production. Verify that retried FlowFiles do not create duplicate rows, and confirm how failed records are routed for review. Ask your implementation partner to document the tested Iceberg and catalog versions, the credential model and the expected behavior for each failure path, so your lakehouse team can operate the flow without relying on the original builders.

Prerequisites for Apache NiFi and Apache Iceberg integration

Most delays in an Iceberg write path come from items nobody confirmed before the build started. The cards below cover the runtime, catalog, storage, credentials, network and test data that must be in place, each with a named owner and a way to check it. Confirm them in your own environment before scheduling the build.

NiFi 2.x release and Java 21

Confirm which NiFi 2.x release you will run and that every node has Java 21 installed, the minimum runtime listed in the NiFi administration guide. Your platform team owns the hosts, and the check is java -version on each node plus the version shown in the NiFi UI. Avoid 1.x for new work, since the project has stated that 1.x cannot be maintained further.

Reachable REST catalog endpoint

Identify the Iceberg REST catalog that NiFi will call and confirm its base URL, supported authentication and whether the endpoint is reachable from the NiFi network. Your data platform team owns the catalog. Ask them to confirm the namespace where the test table will live and who can create tables in it, because the write path needs both catalog access and table permissions.

Bucket or container access

Agree on the storage bucket, container or data lake path for the Iceberg data files, and whether it is on S3, GCS or ADLS. Your cloud team owns the storage account and the identity that reads and writes it. Verify the grant with a manual write and read from a node that runs NiFi, and record the path that the catalog uses for table metadata.

Named credentials and their owner

List every credential the flow needs, including the catalog token, the storage identity and any service account, and name one owner for each. Store them in a secrets manager or NiFi sensitive parameters, not in flow definitions committed to Git. Confirm rotation dates with the owner so a flow does not stop when a key expires.

Network paths from every node

Confirm that each NiFi cluster node, not only the primary node, can reach the catalog endpoint and the storage endpoint over the ports your environment requires. Your network team owns firewall rules, proxies and DNS. Test from each host with the same tools the flow uses, because a path that works from one node can fail on another and surface only under load.

Test table with a known schema

Create a test Iceberg table in a non-production namespace with a schema that matches a sample record from your operational source. Record the field names, types and nullability in writing before the build starts. Include at least one timestamp and one nested field if your source has them. This table lets the team confirm the record reader and schema mapping before real data arrives.

Choosing the file IO provider for S3, GCS or ADLS

Amazon S3 is often the first storage target for a NiFi-to-Iceberg pipeline, and the file IO choice mostly depends on how NiFi authenticates and which network path the cluster uses. Confirm the provider's exact name, the credential options it accepts and its endpoint settings in the NiFi component documentation for your NiFi 2 release before you fix the design, because option names and defaults can change between releases.

Credentials: Confirm which methods the provider accepts, such as static access keys, a named profile, or an instance or pod role. Prefer a method that lets you rotate access without editing flow definitions, and keep long-lived keys out of exported flows.
Network path: Decide whether NiFi reaches S3 through the public endpoint, a gateway endpoint in the VPC or an interface endpoint. Verify that every cluster node, not only the primary node, resolves and reaches the endpoint.
Bucket and prefix layout: The bucket and key prefix become part of the table location recorded in the catalog. Plan separate prefixes per lakehouse zone before the first write, because metadata points at those paths and moving them later is disruptive.
Multi-cloud reads: If some tables also live in GCS or ADLS, each provider needs its own credential and endpoint configuration on every node that writes or reads those paths.

Before any production table is created, check the credential method, endpoint policy and bucket policy together on a test cluster.

Configuring the Iceberg integration in NiFi, step by step

This sequence takes a NiFi cluster from no Iceberg configuration to a tested flow that writes sample records into an Iceberg table, with a check at each stage so that problems surface before production data is involved.

Create the record reader and writer
Choose the record format your source produces, such as JSON or Avro, and configure a record reader for it in the NiFi controller services. Configure a matching record writer for the Iceberg side, and confirm that field names and types in the reader's schema access strategy match the columns you intend to write. Verification: run a small sample FlowFile through a test ConvertRecord or a simple UpdateRecord step and check that the output records show the expected field names, types and nulls before you move on.
Configure the REST catalog controller service
Add the REST catalog controller service that PutIcebergRecord uses to find and commit tables. Set the catalog URI, warehouse location and any authentication properties exactly as your catalog team has defined them, and keep credentials in sensitive properties or parameter contexts rather than in plain flow text. Verification: enable the service and confirm it reports a valid state, then list the namespaces it can see so you know the catalog is reachable from every node that will run the flow.
Configure the file IO provider and its credentials
Add the file IO provider that matches where your table data lives, whether S3, GCS or ADLS, and supply credentials from the storage team using the identity model they have approved for this bucket or container. Confirm the storage path that the catalog reports for your namespace is one the provider can read and write. Verification: write a small test object to the table location through the same provider and read it back from a node other than the one that wrote it, which proves the credentials and network paths work cluster-wide.
Create or confirm the target table
Check whether the Iceberg table already exists in the catalog with the schema and partition spec your downstream consumers expect. If it does not, create it through the catalog's own tooling or an agreed engine such as Spark or Trino, and record the schema and partition fields in your change record. Do not let the flow create tables implicitly unless the data owner has approved it. Verification: describe the table through the catalog and compare its column names, types and partition fields against the record schema from the first step, noting any mismatch before writing anything.
Add PutIcebergRecord to a test flow
On a separate test process group, add PutIcebergRecord and point it at the record reader, the REST catalog service, the file IO provider, the namespace and the table name. Feed it from a GenerateFlowFile or a stored sample file rather than the production source, and route its failure and retry relationships to a visible sink such as a log or a queue you can inspect. Verification: start the processor, confirm the configuration validates without errors, and check the bulletin board for warnings about schema or permissions before sending data.
Run a small sample and check the snapshot
Send a small, known set of records through the test flow, for example a file with a count and a few edge-case values such as nulls and timestamps. After the run, check that a new snapshot was committed to the table and that the snapshot's added record count matches the number you sent. Verification: query the table from a separate engine, such as Spark or Trino, and compare the row count, the sample values and the column types with your source file. Any difference means you should stop and fix the mapping before scaling up.
Record the verified configuration
Export the working process group to your flow registry or change system, store controller service properties and credentials references in your secrets tooling, and write down the table, namespace, catalog endpoint and the checks you ran. Keep the test sample and its expected results so the same checks can be rerun after upgrades or changes. Verification: confirm that a clean deployment from the stored version reproduces the same snapshot and query results, which shows the configuration is reproducible and not dependent on manual edits.

Mapping source fields to Iceberg table columns

This table compares the mapping decisions that NiFi flows make before PutIcebergRecord writes a record into an Iceberg table. Read each row as a question to settle with your data owners before the first load, then confirm the reader and writer settings for each type in the component documentation.

Source element	What to map	Decision or risk	What to confirm
Timestamps and time zones	Epoch values, ISO strings or zoned timestamps mapped to Iceberg timestamp or timestamptz columns	Mixing zone-aware and zone-naive values silently shifts event times; pick one convention per table	Which timestamp formats the record reader parses and how the writer serializes them
Decimals	Source numeric types mapped to a fixed precision and scale	Precision loss or rejected values when the source scale exceeds the target column	How the reader represents decimals and whether the writer enforces precision and scale
Nested records	Structs, arrays and maps from JSON or Avro sources mapped to Iceberg nested types	Field names and nesting must match exactly; flattening early changes the table contract	Whether the record path and writer handle nested and repeated fields for your Iceberg version
Nulls and nullability	Source optional fields mapped to required or optional Iceberg columns	A null in a required column fails the write or lands in an error path, so define defaults upstream	How null values from the reader are written and which errors route to failure relationships
Identifier fields	Business keys used to deduplicate or update rows across retries	Iceberg writes do not deduplicate by themselves; the key design determines whether replays create duplicates	Whether the processor supports the update or merge behavior you need, or whether deduplication belongs in the flow
Partition fields	Source columns or derived values (for example, a date from an event time) used as partition keys	Partition choice is hard to change later and drives file layout and query cost	Partition transform support in your Iceberg catalog and in the writer configuration
Schema changes	New source fields mapped to added Iceberg columns	Added columns need a deliberate schema update; unexpected fields can break or be dropped	Whether schema evolution is handled by the writer or must be applied through the catalog first
Column names and case	Source field names mapped to Iceberg column names	Case or character differences create duplicate-looking columns in downstream queries	How the record reader and writer match names and whether the catalog is case sensitive

How NiFi records become Iceberg snapshots

Each PutIcebergRecord write turns a batch of FlowFiles into data files and one table commit. The processor reads each FlowFile's content with the configured record reader, writes the records as data files through the file IO provider you selected (S3, GCS or ADLS), and then commits those files to the Iceberg table through the REST catalog service. Each successful commit produces a new snapshot. Iceberg readers see whole snapshots, so a half-written batch is not visible until the commit succeeds. The practical consequence is that every FlowFile or group of FlowFiles processed in a run adds files and a snapshot, so the flow's scheduling and batching decide how often the table changes. Confirm in the processor documentation for your NiFi 2 release how batching and failure handling behave, and test a run with deliberately failed records before go-live.

Batch size and FlowFile grouping are the main levers on file count. If the flow writes many small FlowFiles, each run can create many small data files, and the table accumulates snapshots and metadata faster than queries can benefit from them. Small files slow scans, increase catalog and object storage requests, and make later compaction more expensive. Upstream, use MergeRecord or a similar merge step to group records by a sensible size or age threshold before the write, and tune the run schedule so that commits happen at a rate the table can absorb. Ask your team to measure data file count and average file size per partition after a representative week of load, and plan a regular maintenance job for snapshot expiry and file compaction, since those jobs are not performed by the flow itself.

Schema drift shows up as record-level write failures or silent changes, so ownership must be decided before go-live. When a source adds, renames or retypes a field, the record reader infers a schema that may not match the Iceberg table, and the write either fails or lands in an unexpected form. Decide who may change the table schema, whether the flow may add columns automatically, and who approves a type change before any new source is connected. A common pattern is to let the data platform team own table DDL through a reviewed change process, while the integration team owns the mapping in the flow and must raise a change request when the source changes. Record that agreement, along with how failed records are routed to a dead-letter path, in your change management documentation.

Securing the catalog, storage credentials and table access

A NiFi flow that writes to Iceberg touches three trust boundaries: the flow itself, the catalog that owns table metadata, and the object storage that holds data files. Each needs its own credential, its own permissions and its own audit trail, so that a compromise or mistake in one does not silently grant access to the others.

Keep storage credentials in controller services

Define the catalog and file IO credentials once in controller services rather than in processor properties, so they are reviewed in one place. Mark secret fields as sensitive so NiFi encrypts them in the flow configuration and masks them in the UI. Check that no FlowFile attribute or parameter context carries a raw key, and that the sensitive properties key is managed and backed up outside the flow repository.

Scope storage roles to table locations

Grant the NiFi storage identity write access only to the warehouse prefixes of the tables it feeds, not to the whole bucket or container. Ask your platform team to confirm whether the file IO provider you choose (S3, GCS or ADLS) supports the narrowest policy you need, including the paths used for metadata files and deletes. Test the policy with a dry run against a non-production table before the flow goes live.

Encrypt catalog and storage connections

Use TLS for every connection from NiFi to the REST catalog and to the storage endpoint, and verify certificates against your internal CA rather than disabling validation to get a flow running. Confirm the trust store used by the NiFi JVM covers both endpoints, since a certificate trusted by the browser is not necessarily trusted by the processor. Record the cipher and protocol settings your security team approved.

Separate write and read roles

Keep the identity that NiFi uses to commit snapshots distinct from the identities that analysts and query engines use to read tables. Downstream engines should hold read-only grants on the catalog and storage paths, so a misconfigured query cannot alter table metadata. Review which engines can call catalog operations that change table state, not only those that read data files.

Control who can change the flow

Choose an authentication provider for the NiFi UI, such as LDAP, OpenID Connect or SAML 2.0, and avoid relying on single-user access for shared production flows. Use NiFi policies to limit who can view or modify the processor that writes to Iceberg and who can read its sensitive properties. Confirm that the Kerberos provider is not part of your plan, since the administration guide lists it as deprecated.

Audit catalog and flow changes

Collect provenance from the write flow and the catalog's own logs so that every snapshot commit can be traced to a flow version, a NiFi user and a time. Version the flow in a Git-based registry so changes to the Iceberg writer have a reviewed history. Ask your catalog vendor which operations are logged natively and what retention applies, since that determines what an auditor can reconstruct.

Testing commits, duplicates and schema changes before production

Before a PutIcebergRecord flow writes to a production table, the team should run a written test plan that proves four things: row counts match a known input, a failed write leaves no partial data visible, retries and node restarts do not create duplicates, and a schema change does not break the queries analysts run. Each test needs a named owner and a recorded result, and the data platform owner and the analytics lead should sign off before the flow is promoted. Run the plan against a non-production catalog that uses the same REST catalog service, storage provider and FlowFile sizes as production, because a test on a small local warehouse will not show object storage behavior or commit timing.

Start with a replay. Keep a fixed input set, such as a labeled extract from the source system, and replay it through the flow. Compare the row count in the target table with the expected count from the source, then compare a few key aggregates and sample keys. Next, force a failure during the write, for example by revoking write permission on the storage bucket for one run, and confirm through the query engine that no rows from that run are visible. Then repeat a run after a simulated retry and after restarting a cluster node while a write is in progress. The duplicate check should use the same business key the team will rely on downstream. Record every test result in the change ticket, including the NiFi version, the Iceberg library versions and the processor configuration used, so that a later failure can be traced to a specific build. Confirm the exact Iceberg and processor settings against the NiFi component documentation for your release.

Test schema changes in a non-production table first. Add a column through the flow's schema handling, then confirm that old FlowFiles without the column still write, that new records populate it, and that the reader and writer schemas in NiFi match the table definition. Finally, query the table from the engine analysts use, such as Trino, Spark SQL or another engine your team has approved, and check that existing dashboards and saved queries return the same results as before the change. Ask the analytics lead to approve the query results, and ask the security owner to confirm that access grants on the new column follow your data classification policy. Promotion to production should not happen until these checks pass in the test environment.

Common failures in NiFi to Iceberg pipelines and how to fix them

This table lists the failures teams most often hit when PutIcebergRecord writes from NiFi into a REST-catalog lakehouse, with the likely cause, the fix and the diagnostic step to run first. Work down the rows in order of where the flow fails, starting with catalog and storage access before inspecting data content. The main takeaway is that most failures are configuration or identity problems that show up in NiFi bulletins and the catalog's own logs, so the first check is usually cheap.

Failure	Likely cause and fix	First diagnostic step
Catalog authentication errors	Expired or wrong token, client credentials or OAuth settings in the REST catalog service. Correct the credential source and retest the controller service.	Test the REST catalog controller service in the NiFi UI, then check the catalog server's authentication log for the rejected request.
Storage permission denials	The NiFi runtime identity lacks read, write or delete rights on the table location in S3, GCS or ADLS, or the file IO provider uses the wrong identity. Grant the scoped role on the table prefix.	Compare the identity in the file IO provider with the table location and run a single write test from the same host outside the flow.
Endpoint unreachable from some cluster nodes	DNS, proxy, firewall or egress rules differ between nodes, so only some nodes reach the catalog or storage endpoint. Align network policy and proxy settings across all nodes.	Run a connectivity check to the catalog and storage endpoints from each node, and compare results per node in the cluster view.
Schema mismatch between records and the table	Field names, types or nullability in the record reader do not match the Iceberg schema, or the table evolved without the flow changing. Align the reader schema or evolve the table deliberately.	Compare the record reader schema with the table schema in the catalog and inspect the failed FlowFile's content and attributes.
Timestamp or time zone shifts	Source timestamps are parsed with the JVM default time zone or with a different precision than the Iceberg timestamp type, so values move by hours or lose precision. Set an explicit format and zone in the reader.	Write one known timestamp and read it back with a query engine, then compare with the source value and the NiFi host time zone.
Small files that slow queries	Frequent small commits from short run schedules or low batch sizes create many small data files. Increase batch size, lengthen the run schedule and schedule compaction on the table.	Count data files and average file size per partition in the table metadata before changing the flow.
Duplicate rows after retries	A failed or retried FlowFile is written again after the commit succeeded, or a retry follows a partial success. Make writes idempotent with a stable key and a deduplication step before the commit.	Query the table for repeated business keys and match them to the FlowFile UUIDs and provenance events for the same time window.
Credential expiry	Short-lived storage or catalog credentials expire mid-run or after a restart, and writes fail intermittently. Use a credential provider that refreshes, or rotate on a planned schedule.	Check the expiry time of the credential in use and the timestamps of the first failure, then review the provider's refresh behavior.

How much effort does an Apache NiFi and Apache Iceberg integration take?

What drives the cost of writing NiFi data into Iceberg tables?

The cost follows how many moving parts the flow has, not the processor count alone. The main drivers are catalog and storage access (whether you use a REST catalog service with S3, GCS or ADLS file IO providers, and how many accounts and buckets are involved), the number of source systems feeding each table, schema complexity (nested records, evolving fields and type mapping), deduplication rules for replayed or late-arriving records, and the testing scope agreed before build. Each source adds its own extraction, validation and error handling, so a flow feeding three systems is more than three times the design work of one. Ask any partner to break the quote into these components so you can see which ones you can reduce.

Which parts of an Iceberg integration are one-time and which recur?

The build is one-time work: flow design, catalog and storage configuration, schema mapping, deduplication logic, testing and handover. Recurring work follows the system after go-live. Typical recurring items are flow maintenance when sources change, schema changes when table owners add or rename columns, credential rotation for the catalog and object storage accounts, monitoring of failed writes, and NiFi upgrades. NiFi 2 is the current line, and upgrade work should be planned against the NiFi migration guidance for your version. Ask whether a retainer covers these tasks and what response expectations apply.

How long does an Iceberg integration project usually take?

Duration depends on the agreed scope, so no general timeline applies. A single source written to one table with an existing catalog and storage account is a much smaller project than several sources, multiple table owners and new cloud accounts. The timing is set in the written, fixed-scope agreement, which lists deliverables, test coverage and handover dates before work starts. Ask for milestones tied to the flows you need in production, and confirm which decisions, such as catalog choice or table layout, must be closed before build begins, since open decisions are the most common cause of schedule slip.

What does the client team need to provide before work starts?

The client team provides the access and decisions that the build depends on. Expect to supply:

The catalog endpoint and the authentication method the REST catalog requires
Storage access for the S3, GCS or ADLS location, with a service identity scoped to the table paths
Named table owners who approve schemas and accept write behavior
Sample data that reflects real records, including edge cases such as nulls, late events and duplicates
A test environment and a target NiFi version, with a decision on whether NiFi runs in your cloud account or data center

Missing sample data is the most frequent cause of rework, because schema and deduplication rules cannot be tested without it.

Can an in-house team build an Iceberg integration without a partner?

Yes, if your team already runs NiFi and has engineers who understand Iceberg table semantics, catalog configuration and cloud storage permissions. The skills that are often thin in-house are record schema design, exactly-once or deduplicated write patterns, and testing against late or replayed data. A partner is most useful when the team lacks one of these skills, when the timeline is fixed, or when independent testing and documentation are needed for an audit. Whichever path you take, check that the team can operate the result: someone must own the flow, the credentials and the NiFi upgrade plan after handover.

What causes an Iceberg integration to overrun its effort estimate?

Most overruns trace to decisions made late or assumptions that were never tested. Common causes include table schemas that change after the flow is built, unclear deduplication keys across sources, storage or catalog permissions requested one at a time, and sample data that does not match production volume or shape. Test scope is another driver: a flow that handles only the happy path is cheaper to deliver but costly to run. Agreeing on test cases, data samples and the list of open decisions before signing reduces the risk of change requests after work starts.

Does the cost of the integration change if we switch catalog or cloud storage later?

Yes, a change in catalog or storage usually reopens part of the design. Moving from one file IO provider to another changes credentials, endpoints and permissions, and a different catalog can change how table metadata is read and committed. The processor configuration may be reusable, but the access model, tests and runbooks must be revisited. For that reason, ask the partner to document which parts are provider-specific, so a later move is scoped as a defined change rather than a rebuild. Confirm the exact catalog and storage options supported by your NiFi version in the component documentation before you commit.

What should a quote for this work include?

A useful quote lists the scope in writing: the source systems, target tables, catalog and storage configuration, deduplication and schema rules, test cases, environments, documentation and handover. It should separate one-time build effort from any recurring support or retainer, state what the client team provides, and name the deliverables that close the engagement. DeployOpen does not publish a price list, so the quote follows from that written scope. Ask for each assumption to be stated so you can compare proposals from your own team and partners on the same basis.

How DeployOpen scopes and delivers the Iceberg integration

DeployOpen plans, builds, documents and supports self-hosted open-source platforms inside your own infrastructure, including the NiFi flows that feed Iceberg tables. The work is agreed as a fixed scope in writing before it starts, so catalog, storage and flow decisions are settled before any data moves.

How we scope

Inventory source systems, record formats, write volumes and the target Iceberg tables and namespaces for the lakehouse.
Confirm the REST catalog product, the storage provider (S3, GCS or ADLS) and the credential model NiFi will use.
Agree in writing which flows, controller services, cluster settings and test cases are in scope and what is excluded.

How we build

Design PutIcebergRecord flows with record readers, schema handling and error routing for malformed or rejected records.
Configure the REST catalog service and file IO provider for your chosen storage, with credentials held in NiFi sensitive values.
Test commits, retries and duplicate handling against non-production tables before any production source is connected.

How we hand over

Deliver version-controlled flow definitions, parameter contexts and a runbook covering catalog and storage configuration.
Document verification steps for row counts, snapshot commits and table schema changes against your Iceberg catalog.
Walk your platform and data teams through monitoring, provenance review and the steps for adding a new source table.

How we support

Ongoing support, NiFi upgrades and continuous testing are available as retainers once the initial handover is complete.
Retainers can cover catalog or storage configuration changes, new source onboarding and review of failed commits.
Confirm response expectations, service levels and upgrade windows in the written agreement before work begins.

Apache NiFi and Apache Iceberg integration