Why NiFi flows that wrote to Hadoop need re-pointing before the move to NiFi 2
Flows that write to HDFS, Hive or HBase must be re-pointed before or during the move to NiFi 2, because those Hadoop components are no longer part of NiFi 2. The replacement is a set of storage and table targets: object storage such as S3, Azure Data Lake Storage or Google Cloud Storage for files, and Apache Iceberg tables for data that analysts and query engines read. NiFi 2.12.0 provides PutIcebergRecord with a REST Iceberg catalog service and S3, GCS and ADLS Iceberg file IO providers, so the sink change is a design decision you can make in advance. How much work that takes depends on the number of flows, the number of sinks each flow writes to, and any custom processors or scripts that call Hadoop client libraries directly.
Before any sink is changed, the 1.x to 2.x prerequisites apply to the whole installation, not only to the storage paths. The NiFi migration guidance requires upgrading to NiFi 1.27.0 before moving to 2.0.0, and changing the flow configuration file from flow.xml.gz to flow.json.gz. Several 2.x changes affect flows directly, including the replacement of the variable registry scope with an ENVIRONMENT scope and the renaming of distributed map cache components. Confirm the Java 21 runtime and your Python version for any Python-based processors against the NiFi administration guide, and inventory every flow that uses a removed component before you schedule cutover.
Start with an inventory that lists each flow, its Hadoop sink, the file format and partitioning it uses, and who reads the output. Decide early whether a target table becomes an Iceberg table in a catalog your query engines already use, since the chosen catalog determines the connection settings each flow needs. Confirm with your platform team which catalog and object storage your engines support, and test one flow end to end before changing the rest. Any delivery guarantee, duplicate handling or schema change that the Hadoop sink relied on must be restated for the new target.
Which Hadoop-era flow patterns map cleanly to object storage and Iceberg
This table compares common Hadoop-era NiFi flow patterns with their closest equivalent in NiFi 2.12.0 writing to Iceberg tables on object storage. Read each row as a design question: the mapped rows can usually be re-pointed with configuration, while the flagged rows need a table or access pattern decision before any flow is rebuilt.
| Source pattern | Target in NiFi 2.12.0 and Iceberg | Mapping status |
|---|---|---|
| Writing files to HDFS with PutHDFS-style flows | PutIcebergRecord writing data files through S3, GCS or ADLS file IO providers | Maps with change of destination and credentials |
| Loading Hive tables from record streams | PutIcebergRecord against an Iceberg catalog, such as the REST catalog service | Maps, but Hive DDL must be redesigned as Iceberg table definitions |
| Hive-partitioned directory layouts (key=value folders) | Iceberg hidden partitioning defined on the table, not in folder paths | Needs a partition design decision |
| ORC or Avro output files | ParquetIcebergWriter produces Parquet data files | Format change, confirm downstream readers support Parquet |
| Schema handling on load | Iceberg schema evolution without rewriting tables, confirmed against the record schema in the flow | Maps, validate column type changes before cutover |
| HBase row-key writes and point lookups | No direct equivalent; Iceberg tables are scanned by partition and file statistics | Does not map, redesign the access pattern first |
| Hive SerDe behavior and Hive-specific DDL | Not carried over; Iceberg defines its own table metadata and types | Does not map, review each table definition |
| Append-only loads into existing tables | Iceberg commits create snapshots, so duplicate and retry behavior must be designed | Needs a table design decision |
Choosing the storage and catalog target for each flow
NiFi 2.12.0 provides Iceberg file IO providers for S3, GCS and ADLS, so the storage provider is the first decision for a flow that wrote to HDFS. The provider choice is driven by where your data already lives, which cloud account owns the buckets or containers, and which region and network path your sources can reach. Storage location does not by itself decide which engines can query the data, so treat it as the physical layer and choose the catalog separately.
- Cloud provider alignment: Keep compute, storage and catalog in the same provider and region where possible, so flows do not pull large volumes across accounts or networks.
- Credential model: Confirm how NiFi will authenticate to each store (instance roles, workload identity, access keys or managed identities) and who rotates those credentials.
- Self-hosted object storage: If the target is an on-premises or private S3-compatible service, confirm its S3 API compatibility for the operations Iceberg writes need, and check the Iceberg file IO path against your vendor's documentation.
- Prefix and bucket layout: Plan table locations and ownership per domain before writing, since moving table locations later means rewriting metadata.
- Retention and lifecycle rules: Set object lifecycle policies with Iceberg snapshot expiry in mind, so cleanup does not remove files that time travel still references.
Before committing, confirm with your cloud or storage vendor how each file IO provider authenticates in your environment and what your network policy allows.
Trino's Iceberg connector selects its catalog through iceberg.catalog.type and supports Hive metastore, AWS Glue, JDBC, REST, Nessie and Snowflake catalogs. If your team already runs Trino, the catalog type you pick for NiFi writes must match a catalog that Trino can open, or the tables will be invisible to it. The connector reads Parquet by default, with ORC and Avro also supported, which matters when comparing against the Parquet output of the NiFi ParquetIcebergWriter.
- Hive metastore: Often the closest fit for teams migrating from Hadoop, since it keeps the metastore model familiar, but confirm it is still operated and patched.
- AWS Glue: A managed option for AWS-hosted tables, where the access policy for NiFi and Trino must be set on the same Glue resources.
- REST catalog: Use when a catalog service exposes the Iceberg REST specification, and confirm that both NiFi's REST catalog service and Trino's REST configuration point to the same endpoint and authentication scheme.
- JDBC and Nessie: Confirm the database backing a JDBC catalog and the version compatibility of a Nessie server before relying on them for shared writes.
- File format: Match the file format on the NiFi side with what Trino reads, and test a sample table before migrating volumes.
For each catalog type, confirm in the Trino Iceberg connector documentation and with your platform team that the catalog version and the NiFi catalog service are supported together.
Some downstream teams query through managed platforms rather than their own engines. Snowflake can read Iceberg data through its Horizon Iceberg REST Catalog API for Snowflake-managed Iceberg tables, and OneLake exposes Delta tables to external Iceberg readers through metadata virtualization. These paths change who owns the table definition and which system is the source of truth, so decide that before a flow writes to a location those platforms will read.
- Snowflake-managed tables: Snowflake stores the data on its managed storage or an external volume, so confirm whether a NiFi-written table belongs in Snowflake's catalog at all or should be an externally managed table.
- Horizon REST access: Confirm which credentials, roles and endpoints external engines must use to read Snowflake-managed Iceberg tables.
- OneLake representation: Microsoft documents that OneLake stores tables as Delta Parquet or Iceberg, so confirm which format and virtualization setting applies to your Fabric workspace.
- Ownership and writes: Decide which system writes each table, because concurrent writers from NiFi and a managed platform need coordinated commit paths.
- Cost visibility: Ask each platform vendor how reads from external engines are metered, since platform fees may apply to access patterns the source flows did not have.
Before finalizing a design, verify with the Snowflake and Microsoft documentation and with the platform owners which system holds the authoritative table definition.
Amazon Athena engine version 3 queries Iceberg tables with time travel through FOR TIMESTAMP AS OF and FOR VERSION AS OF clauses. Teams moving from Hadoop often depend on point-in-time reads for audits, reprocessing and debugging, so whether the catalog and engine version support those reads is a functional requirement, not a detail. Iceberg itself provides snapshot history and rollback, but the reader must confirm that each engine in the chain honors those snapshots.
- Engine version: Confirm that the Athena workgroups in use run engine version 3 before relying on time travel syntax.
- Catalog behavior: Test time travel against the same catalog NiFi writes to, since a read path that uses a different catalog may not see the same snapshots.
- Snapshot retention: Set snapshot expiry to match how far back auditors or reprocessing jobs need to read, and document the setting.
- Rollback procedures: Agree who may roll a table back and how that action is recorded in your change log.
- Writer and reader alignment: Verify that NiFi commits produce snapshots that the reader engine can resolve during a test read.
Before you commit to a design, run a time travel query from each consumer engine against a test table written by NiFi, and confirm the results with the Athena and Iceberg documentation for your versions.
Documented paths for re-pointing NiFi writes and reading existing data
NiFi 2 does not carry the Hadoop-era processors forward, so the replacement work starts with the documented components that write to object storage and Iceberg tables. The cards below cover each component, the configuration file change that comes with the move, and the checks to run against the components index before any flow is rebuilt.
PutIcebergRecord processor
PutIcebergRecord is the NiFi 2 processor that writes record data into Iceberg tables and takes the place of flows that previously wrote to Hive or HDFS directories. Map each existing target table to an Iceberg table identifier, then confirm in the components index that the processor's properties match the record schema your sources produce. Test a single table end to end before changing the remaining flows.
REST Iceberg catalog service
The REST Iceberg catalog service is the controller service that tells the writer which catalog holds each table, replacing the Hive metastore references that Hadoop flows often hard-coded. Decide early which catalog your organization will run, because catalog choice determines credentials, namespaces and how other engines find the same tables. Confirm the catalog's supported endpoints and authentication options with its vendor before wiring the service.
Object storage file IO providers
NiFi 2 ships separate Iceberg file IO providers for S3, GCS and ADLS, so the storage target you pick sets which provider and credentials the flow needs. Plan one provider per cloud or bucket boundary and keep credentials in a secrets store rather than in flow parameters. Check each provider's configuration keys in the components index, since names and required fields can differ between providers.
ParquetIcebergWriter
The ParquetIcebergWriter controls how records are serialized into Parquet data files before they are committed to an Iceberg table. File sizing, compression and schema handling all affect how many small files the table accumulates and how fast downstream engines can read it. Agree these settings with the teams that query the tables, and benchmark them on a representative sample of your real volumes.
Flow configuration file migration
The NiFi migration guidance requires moving from the flow.xml.gz configuration file to flow.json.gz as part of the 2.x upgrade, and it requires upgrading to NiFi 1.27.0 before moving to 2.0.0. Flows that still reference Hadoop processors will not load in NiFi 2, so each one must be replaced or removed during that step. Back up the existing configuration and run the conversion in a non-production instance first.
Confirm every replacement in the index
Hadoop-era processors such as the HDFS, Hive and HBase components are not available in NiFi 2, so there is no automatic one-to-one mapping for every flow. Treat the components index as the source of truth for each replacement, and record for each flow which processor, controller service and property set was chosen. Ask the vendor or project directly about any component whose behavior you cannot confirm from its documentation.
Sequence for moving flows with a parallel run and cutover
This sequence moves each NiFi flow from Hadoop sinks to object storage and Iceberg tables in controlled stages, so every cutover is backed by comparison evidence and a clear decommissioning point. It keeps the Hadoop writers running until consumers have confirmed the new tables.
Inventory flows, sinks and consumers
Your team lists every NiFi flow that writes to HDFS, Hive or HBase, including processor type, target path or table, file format, write frequency and any controller service that holds Hadoop configuration. Alongside each flow, record which downstream jobs, dashboards and users read the output, because those consumers decide when a flow can move. The step is done when every flow has one owner, one sink and a named list of readers, and the list is reviewed by the data platform team. Ask DeployOpen to draft this inventory if you want a second pair of eyes on hidden dependencies.
Upgrade the NiFi estate to 1.27.0 first
Before any flow changes, the NiFi 1.x deployment is upgraded to 1.27.0, which the NiFi migration guidance requires before a move to 2.0.0. Your team backs up the flow definitions and repositories, runs the upgrade on a non-production copy, and confirms that every flow starts and processes sample data. The step closes when the 1.27.0 instance runs your flows without new errors in the bulletin board. Only then is the instance moved toward 2.x, including the change from flow.xml.gz to flow.json.gz and the 2.0 renames in the variable registry and distributed map cache components.
Build a pilot flow against a test bucket and catalog
Pick one low-risk flow and rebuild its sink with PutIcebergRecord, a REST Iceberg catalog service and the file IO provider for your object store, using a test bucket and a test catalog namespace. Confirm the writer produces Parquet files through ParquetIcebergWriter and that the table schema matches what consumers expect. The pilot is complete when a full run lands data in the test table, the schema has been reviewed by the consumers, and a repeat run does not create duplicates. Keep credentials as parameters or sensitive values from the start.
Run Hadoop and Iceberg outputs in parallel
Both paths run on the same input for an agreed observation window, with the original Hadoop writer still feeding production consumers. Provenance data from each path is compared so you can trace record counts, failures and retries per run, and a reconciliation query checks row counts, key uniqueness and aggregates between the HDFS or Hive output and the Iceberg table. Differences are logged with a root cause before they are accepted or fixed. The step is done when reconciliation passes for the full window, and the owners of each consumer sign off on the results.
Cut over one flow at a time
Each flow is switched to Iceberg in a planned change, with consumers repointed to the new table, a rollback plan kept ready, and the Hadoop writer disabled rather than deleted. Your team changes one flow per window so that any issue can be traced to a single cause. Time travel and rollback on Iceberg tables help here, because a bad load can be reverted to an earlier snapshot. The cutover closes when the flow has run through at least one full cycle on Iceberg, consumers confirm reads, and the change record is stored in your change management system.
Decommission HDFS, Hive and HBase writers
Once every consumer of a flow has moved, its Hadoop processors and controller services are removed from the NiFi canvas and the associated parameters, credentials and Hadoop configuration files are deleted from the NiFi nodes. Your team archives the last Hadoop output and the flow version in Git through your Flow Registry Client before deleting anything, so the change can be audited later. The step is done when no flow references HDFS, Hive or HBase, and a search of the flow definitions confirms it. Retiring the Hadoop storage itself is a separate decision owned by your data platform team.
Deciding between managed cloud storage and self-hosted S3-compatible storage
Managed S3, ADLS and GCS are the documented targets for the NiFi 2.12.0 Iceberg path, since the PutIcebergRecord processor ships with S3, GCS and ADLS Iceberg file IO providers. Choosing a managed service keeps the storage layer under your cloud provider's operations and support, and it removes the question of how the object store itself is licensed and patched. Your decision then centers on bucket or container layout, identity for the NiFi nodes, encryption keys and the region that sits closest to your Trino, Athena or Spark readers. Confirm the exact file IO provider configuration for your chosen cloud against the NiFi components index before you write the flow.
Self-hosted S3-compatible storage such as MinIO needs a separate license and support decision before any flow is built. The MinIO community server is released under the GNU AGPL v3.0, and its README says commercial or proprietary use must be validated against AGPLv3 obligations. The minio/minio, minio/operator and minio/mc repositories are archived, the community edition is distributed as source code only, and legacy binaries will not receive updates. The README points to AIStor for commercial licensing and SLA-backed support, and says community support is best effort. Ask the vendor to confirm the current status of each option in writing, because repository and release status can change.
Before choosing MinIO, check whether your organization already runs object storage, for example Ceph is not a source for this topic, so confirm its S3 gateway, its support contract and its Iceberg-compatible access path with your storage team. Reusing an existing platform with a support agreement usually carries less risk than adopting an archived community project. If you still pick a self-hosted option, record who owns patching, who can answer a license question and what your exit path is to a managed service.
What drives the cost and duration of a Hadoop exit
This table lists the factors that move the effort of re-pointing NiFi flows from HDFS, Hive and HBase to object storage and Iceberg, and marks which costs are paid once and which recur. Read each row as a question to answer during scoping, because the answers decide whether a quote is driven by a small rewrite or by a full data migration.
| Driver | Why it changes cost or effort | How to keep it down |
|---|---|---|
| Number of flows and sinks | Each HDFS, Hive or HBase writer needs a new target, a new schema mapping and its own tests, so effort grows with the count of write paths. | Inventory every PutHDFS, PutHiveQL and HBase processor, then group flows that share a sink and schema. |
| Custom processors and scripts | Scripted or custom Java processors that call Hadoop client libraries must be rewritten or replaced before they run on NiFi 2. | Confirm early which custom bundles reference Hadoop APIs and retire any that no longer earn their place. |
| Hive DDL and table definitions | Hive partitions, bucketing and storage formats map to Iceberg partition specs and file formats only with careful review, and some layouts need redesign. | Export the DDL for every table and classify each one as move as-is, redesign or retire before estimating. |
| HBase schema and access patterns | Row-key designs built for HBase lookups rarely translate directly to analytical tables, so the target model is a design task, not a copy. | Document which applications read HBase by key and agree the replacement access path with those teams. |
| Data volume and history to backfill | Historical backfill is a separate load with its own throughput, validation and time-window planning, independent of the flow rewrite. | Decide the backfill window with the data owners and separate it from the ongoing flow cutover. |
| Downstream consumers and engines | Each engine that reads the tables (for example Trino, Spark, Athena or Snowflake) needs catalog access, permissions and compatibility checks. | List every consumer and its catalog type before choosing the table format and catalog. |
| Catalog choice | Iceberg catalogs such as REST, Hive metastore or AWS Glue differ in operations, access control and which engines they support. | Choose the catalog against the engines in use, and confirm supported catalog types with each engine's documentation. |
| Object storage platform decision | Choosing S3, ADLS or GCS, or a self-hosted S3-compatible store, changes identity integration, network paths and the file IO provider settings. | Settle the storage platform before design starts. Self-hosted options need a current license and support review, since the MinIO community edition is no longer maintained. |
| Compliance evidence needs | Audit scope may require retained provenance, access logs, change records and tested rollback for each migrated flow, adding documentation and test effort. | Agree the evidence list at scoping and build it into the test plan rather than adding it at the end. |
| Parallel-run length | Running old Hadoop flows and new Iceberg flows side by side doubles operating cost and needs reconciliation checks until each consumer is switched over. | Set reconciliation criteria per table and a fixed cutover date for each consumer group. |
| One-time work | Redesign of flows and tables, testing, backfill, parallel run and cutover are paid once per migration wave and are the bulk of the initial quote. | Ask for fixed-scope deliverables and acceptance criteria for each wave in the written statement of work. |
| Recurring work | Catalog operations, storage capacity and access management, NiFi and Iceberg upgrades, and monitoring continue after cutover and need an owner. | Ask who runs these after handover and whether the quote includes a support retainer or internal ownership. |
What usually goes wrong when NiFi writes move to Iceberg
Moving NiFi writes from Hive, HDFS and HBase to Iceberg tables on object storage fails in predictable places: the schema, the partition layout, the cutover period and the consumers downstream. The cards below name each risk and the check that prevents it, so the migration can be tested before any production table changes.
Silent schema drift from Hive
Hive tables often carry column types, nullability and column order that were never enforced, and a PutIcebergRecord flow can write records that differ from them without an error. Compare the Hive DDL, the Avro or Parquet schema the flow emits and the Iceberg table schema field by field, including decimal precision, timestamp zones and nested records. Use Iceberg schema evolution deliberately, and record each change in version control so that a rename or type widening is an approved decision, not a side effect.
Partition layout mistakes
Partition specs chosen to mirror Hive directory layouts often produce many small files or skewed partitions in Iceberg. Iceberg's hidden partitioning lets you define partition transforms on columns without exposing them as path segments, so the layout should be designed from query patterns instead of copied. Test the write path with production-like volumes, then inspect file counts and sizes per partition before you expand the flow to all sources.
Duplicate writes during parallel run
Running the HDFS or Hive flow and the new Iceberg flow side by side means the same source events can be written twice, once to each target. Decide which target is authoritative for each dataset, add a stable record key, and confirm that retries and back pressure in NiFi do not re-emit records that already committed. Reconcile row counts and key sets per partition at the end of each parallel-run window.
Wrong assumptions about HBase row access
HBase offers row-key lookups, cell versions and column-family access patterns that do not carry over to Iceberg tables, which are scanned by query engines rather than read by key. Flows that used HBase as a lookup store for enrichment need a different design, such as a cache, a relational table or a periodically refreshed snapshot. Identify every HBase read and write in the flow, and document the access pattern it needs before choosing the replacement.
Archived MinIO tooling
If the target object store is MinIO, note that its community repositories are archived and read-only, and the last tagged community release is dated October 2025. Community builds are distributed as source, and support through GitHub and Slack is best effort. Confirm the license obligations of AGPLv3 with counsel, decide whether to use an AIStor edition or another S3-compatible store, and plan for patching the server and the mc client yourself.
flow.json migration errors
NiFi 2 expects the flow configuration in flow.json.gz rather than flow.xml.gz, and the upgrade path requires moving to NiFi 1.27.0 first. Processors, controller services and parameter contexts that were renamed or removed, such as the distributed map cache components and the variable registry scope, can stop the flow from loading. Upgrade in a staging copy, load the converted flow, and compare the processor and controller service inventory against the original before any production start.
Consumers reading stale or mixed tables
Analysts and jobs in Trino, Athena, Snowflake or Spark may query a table while a NiFi commit is in progress, or read a snapshot that mixes old and new batches. Iceberg snapshots and time travel help you isolate a known-good state, but only if each consumer pins or checks snapshots it depends on. Publish a freshness expectation for each table, agree a rollback procedure using snapshot IDs, and tell consumers which catalog they should query during and after cutover.
Questions teams ask before moving NiFi off Hadoop
What does moving NiFi flows off Hadoop cost and what drives it?
The cost depends mainly on how many flows write to HDFS, Hive or HBase, how many distinct targets they feed, and how much custom code sits inside them. Each flow needs its write path redesigned, its schema mapped to Iceberg tables, its credentials and storage access rebuilt for the target cloud, and its output checked against the existing data. The main cost components are engineering time for flow rework and testing, the object storage and catalog services you choose to run, and the platform team time to operate them. Some costs are one-time, such as the migration itself and parallel running, while storage, catalog hosting and support recur. Ask any partner for a quote built from a written inventory of flows, sinks and consumers, with the testing and cutover work listed separately. Also confirm which catalog type you will use, since that choice changes the components you must run.
How long does a move off Hadoop take?
Duration is set by the number of flows, the number and type of downstream consumers, and how many of those consumers must be moved before the Hadoop path can be retired. Teams with a small set of flows writing to a single table family usually finish faster than teams with many flows feeding Hive jobs, HBase lookups and reporting tools that all depend on exact table layouts. Timing also depends on your change approval process, on how quickly data owners can confirm reconciliation results, and on whether the cloud accounts, network paths and identity setup are ready before flow work begins. No duration should be assumed until the inventory is complete. Ask for a plan with phases for inventory, build, parallel run and cutover, and ask for each phase to have exit criteria that your data owners sign off on.
What does the internal team need to provide?
The internal team needs to supply the knowledge the work cannot be done without. That includes the owners of each flow and table, the business rules behind existing transformations, the consumers that read Hive and HBase data, and the approvers for schema and retention changes. Platform engineers need to provide the cloud accounts, network access, identity roles, key management and storage buckets, and someone must own the catalog and its backups after handover. Security and compliance staff should review where data lands and who can read it before cutover. If these owners are not named early, the build will stall on questions that only they can answer, so agree on them before the inventory starts.
Can our own team move NiFi flows off Hadoop without outside help?
A team that already runs NiFi 2, understands Iceberg table design and has access to the target cloud can do the work in house. The skills that are hard to build quickly are Iceberg table layout and maintenance, reconciliation design across old and new storage, and planning a cutover that does not break existing consumers. Teams without these skills often spend their effort learning the stack during the migration rather than delivering it. Many organizations therefore split the work, keeping flow ownership in house while using an implementation partner for the inventory, the design of the Iceberg tables and the parallel-run checks. DeployOpen implements, integrates, hardens and documents self-hosted platforms in your own infrastructure under a written, fixed scope. Compare the internal capacity you have against the inventory before deciding.
Does NiFi 2.12.0 write Iceberg tables to S3, GCS and ADLS?
Yes, NiFi 2.12.0 provides the PutIcebergRecord processor, a REST Iceberg catalog service, Iceberg file IO providers for S3, GCS and ADLS, and a ParquetIcebergWriter. This means the writing side of the move is covered for all three clouds within the component set. Confirm the exact provider configuration for your cloud and catalog in the component documentation, and test the write path with your own schemas, partitioning and credentials before you rely on it in production. Coverage of the component does not replace checks on how your catalog handles commits, concurrent writers and permissions. Start with a single flow per cloud as a proof of the write path, and record what you verified.
How do we keep Hive consumers working during the parallel run?
Keep the existing Hadoop path running and write the same records to the Iceberg tables in parallel, so that consumers can be switched one at a time. Iceberg tables can be read by several engines, including Trino, Spark, Flink, Presto, Hive and Impala, so a Hive consumer may be able to read through a shared catalog after its query is checked. Trino's Iceberg connector reads Parquet by default and can use Hive metastore, AWS Glue, JDBC, REST, Nessie or Snowflake catalogs, selected with iceberg.catalog.type. Reconcile row counts, key distributions and aggregates between the two paths for each table, and keep the Hadoop output until every consumer has been verified. Confirm that each consumer's engine version and access path supports the catalog you choose.
Which object storage should we choose for the move?
The choice depends on the cloud you already run, the skills and contracts you hold, and the sovereignty and licensing rules that apply to your data. S3, Azure Data Lake Storage and Google Cloud Storage are all supported by the NiFi 2.12.0 Iceberg file IO providers. For self-hosted object storage, check the license and support position before you commit. The MinIO community server is released under the GNU AGPL v3.0, its repositories are archived, and its README says commercial or proprietary use must be validated against AGPLv3 obligations, with supported editions sold separately. Confirm this with the vendor and your legal team, and do not assume a community project will receive support or security updates.
What the platform team and data consumers must change
The platform team and the flow owners carry different parts of this move, and the split matters more than the tooling. Platform engineers own the catalog, the credentials and the storage policy: which Iceberg catalog the PutIcebergRecord processor writes through, which file IO provider (S3, GCS or ADLS) holds the data, and which service identity NiFi uses to reach the bucket or container. Those choices should be made once, documented and reviewed, not left to each flow author. Before any flow is rebuilt, the platform team should confirm the catalog type the downstream engines will use, because the NiFi 2.12.0 REST Iceberg catalog service has to agree with what Trino, Athena, Snowflake or OneLake expect to read. Ask the vendor or catalog provider to confirm that interoperability in your own environment rather than assuming it.
Flow owners carry the write path itself. A flow that once called PutHDFS, PutHiveQL or an HBase processor now needs a record schema, a partitioning decision and a rule for what happens when a write fails partway through. Iceberg commits are table-level operations, so the error handling that worked for appending files to a directory has to be redesigned around failed and retried commits, duplicate records and schema changes. Flow owners should write down how each flow proves a batch landed, and should test a forced failure in a non-production table before cutover. Consumers, meanwhile, need to move their queries. Iceberg supports schema evolution without rewriting tables, and time travel to exact historical snapshots and rollback to an earlier state are core features. Those are useful only if analysts and jobs are told to query snapshots deliberately and if someone owns the decision to roll back a bad load.
How DeployOpen scopes and delivers a NiFi Hadoop exit
A Hadoop exit touches flows, storage, catalogs and downstream readers, so the work is scoped in writing before any flow is touched. The cards below follow the path from agreed scope through inventory, pilot, parallel run and cutover to documentation and optional ongoing support.
Written scope before work starts
Most engagements are fixed-scope, so the flows in scope, the target storage and Iceberg catalog, the deliverables and the timeline are agreed in writing before work begins. Ask for the scope to name every flow that writes to HDFS, Hive or HBase, every downstream reader, and the acceptance tests each flow must pass. A quote follows from that written scope, so an unclear scope usually means an unclear quote.
Inventory and target design
The first phase maps every processor that touches Hadoop, including PutHDFS, Hive and HBase processors, controller services, credentials and scheduling. The output is a target design that picks the object storage provider, the Iceberg catalog type and the table layout for each dataset. Confirm early which engines will read these tables, since that choice drives the catalog decision.
Pilot on one flow
A pilot re-points one representative flow to object storage and an Iceberg table inside your own cloud account or data center. It tests write behavior, schema handling, file sizing and credentials on real data volumes. Agree the pilot exit criteria in the scope, such as row counts, schema checks and read access from each downstream engine.
Parallel run and cutover
During the parallel run, the legacy HDFS path and the new Iceberg path both receive data so outputs can be compared before any consumer moves. Cutover then happens flow by flow, with a rollback path kept until the comparison is accepted. Plan the cutover window with the teams that own downstream reads, because the storage change affects them as well as NiFi.
Documentation and handover
Deliverables usually include the flow definitions under version control, parameter and credential inventories, runbooks for the object storage and catalog, and a record of the validation results. Ask for the handover to name who on your side will own each artifact after the engagement ends. DeployOpen prepares evidence for compliance assessments, but it does not issue audit reports, attestations or certifications, so independent auditors handle that step.
Optional retainer after cutover
Ongoing support, upgrades and continuous testing can run under a retainer after cutover. This matters because the Iceberg catalog, the storage platform and NiFi itself keep changing, and the NiFi 1.x line is out of support. Ask what the retainer covers, how changes are requested, and how upgrades are tested before they reach production flows.
