Moving an Apache NiFi cluster to a new data center or cloud

Moving a running Apache NiFi cluster without losing data in flight

Moving a running Apache NiFi cluster to a new data center or cloud is manageable when queued data, repositories, certificates and hostnames are handled on purpose rather than left to the cutover night. The effort depends mainly on three things: how much data is sitting in queues and repositories at the moment of the move, how many flows and process groups the cluster runs, and how many external systems push data into it or pull data from it. Each of those changes which strategy is safe, so the first task is an inventory of all three before anyone chooses a method.

The main decision is whether to drain queues and let the old nodes empty out, or to carry repositories across and reattach them. Draining suits flows where the upstream sources can be paused or buffered, and it leaves a clean node to decommission through the cluster's offload and disconnect steps. Moving repositories suits large backlogs that cannot be drained in a reasonable window, but it requires the new nodes to match the flow and repository layout exactly. For a staged cutover, Site-to-Site lets the new cluster receive data from the old one while both run, and it also needs its own certificate and hostname plan. Confirm the exact node offload and decommission procedure, repository handling and Site-to-Site settings for your NiFi version against the NiFi System Administrator's Guide before you commit to a sequence.

Apache NiFi is licensed under Apache-2.0 and has no paid edition from the project, so the cost of the move is the engineering, testing and infrastructure work, not a license change. DeployOpen plans and executes the move inside your own cloud accounts, data centers or restricted networks, with the cutover sequence, rollback criteria and provenance-based validation agreed in writing before work starts. Ask any partner, including us, to show how they will count queued events before and after the move, which external endpoints will be re-pointed, and how certificates issued for the old hostnames will be replaced.

What carries over to the new site and what must be rebuilt

This table compares each part of a running NiFi cluster with what happens to it at the new site. Read the first column as the item, the second as what carries over, and the third as what must be rebuilt or re-verified. Confirm the source NiFi version first, because the flow configuration format and the upgrade path depend on it.

Component	Carries over	Must be rebuilt or re-verified
Flow definition file	The flow configuration file moves with the node when the version stays the same, provided the source and target run the same NiFi release.	On NiFi 2.x the file is flow.json.gz, while 1.x uses flow.xml.gz. Confirm the source version before choosing a path, because migration guidance requires moving through 1.27.0 before 2.0.0.
FlowFile and content repositories	Queued FlowFiles and their content can be moved on disk if nodes are stopped cleanly and repository paths are kept consistent.	Queues often drain better than they move. Decide per queue whether to drain at the source or relocate data, and verify counts and checksums after the move.
Provenance repository	Existing provenance events can be copied with the repository if retention is needed for audit at the new site.	Provenance is node-local and can be large. Confirm retention needs with your compliance team and validate lineage at the target with test FlowFiles.
Cluster state and coordination	Flow and cluster membership are rebuilt rather than copied wholesale; the coordination service (embedded or external ZooKeeper, or Kubernetes) must be set up for the new site.	Node identities, ZooKeeper or Lease configuration and any external coordination endpoints need reconfiguration and a controlled rejoin.
Certificates and keystores	Existing keystore and truststore material can be reused where the certificate authority and subject remain valid.	Certificate subject names, SANs and trust chains usually change with hostnames. Issue or reissue certificates for the new names and update the truststore on every node.
Hostnames and network addresses	Flow logic that does not embed hosts continues to function unchanged.	Node addresses, load balancer endpoints, DNS entries and Site-to-Site peer URLs must be updated. Check every remote process group and reporting task that points to a host.
Parameter contexts	Parameter definitions and non-sensitive values carry over with the flow definition.	Sensitive parameter values are not readable from the file and must be re-entered or sourced from a secrets provider at the target. Confirm the sensitive key before cutover.
Controller services	Service definitions and their configuration properties move with the flow.	Services that reference hosts, keystores, database endpoints or cloud storage buckets must be repointed and tested, and any that were disabled at source should stay disabled until verified.
External endpoints and credentials	Logic that calls the same external systems keeps the same design.	Firewall rules, allow lists, API tokens and database users tied to the old source addresses need approval and reissue at the target.

Choosing between queue draining, repository moves and Site-to-Site

When queues are small and can be emptied within a planned window, draining is usually the simplest path. You stop new data at the sources, let processors work the existing FlowFiles down to empty connections, then offload and decommission the old nodes. Nothing has to be copied between storage systems, so the repositories on the new hosts start clean and the operator has fewer moving parts to verify. The trade-off is time: draining depends on downstream targets accepting data, and a slow or unavailable destination can hold the window open.

Drain window: Confirm that every connection feeding the outage can reach empty within the change window, with the queue depths measured during a normal business day.
Source stop method: Confirm how each source (listening ports, polling processors, message consumers) will be paused so no new data is accepted during the drain.
Downstream availability: Confirm that targets such as databases and APIs will accept the drained load, since a stalled sink turns a drain into a backlog.
Provenance continuity: Confirm that your provenance retention covers the cutover period, so the team can trace records that finished on the old cluster.
Offload behavior: Confirm how the node offload handles any FlowFiles that remain queued and whether the decommission is blocked until the repository is clear.

Before committing to draining, ask the target environment owner to confirm the queue measurements and the downstream acceptance windows in writing.

Documented tools and interfaces for the move

A NiFi cluster move relies on a small set of documented mechanisms: cluster controls for leaving nodes, Site-to-Site for moving data between instances, versioned flow definitions, provenance for proof, and the flow configuration file. Each one answers a different question during the cutover, so the plan should name which tool answers which question before work starts.

Node offload and disconnect

Offloading moves queued FlowFiles off a node so it can leave the cluster, and disconnecting removes it from cluster membership. Both are available through the cluster UI and the REST API, which makes them scriptable for repeated node-by-node replacement. Before starting, confirm that every node being removed has finished offloading, and that the remaining nodes have enough disk and heap to absorb the redistributed data. Record the time each step completes so the cutover log is auditable.

Site-to-Site transfers

Site-to-Site is NiFi's protocol for sending FlowFiles between NiFi instances, and it is the usual mechanism for a staged cutover where the old and new clusters run side by side. Sources on the old cluster push to remote process group input ports on the new one, so each source flow can be switched over separately. Check that the TLS identities and authorizations for the remote ports are in place on both sides before any production data crosses, and that ports are opened only between the two specific networks.

Versioned flow definitions

A flow registry stores versioned copies of process groups so the same flow can be imported into the new cluster without manual rebuilding. NiFi 2 provides Flow Registry Clients for GitHub and GitLab, which keep flow versions in a Git repository. NiFi Registry is deprecated and planned for removal in NiFi 3.0, so new work on a move should favor Git-based clients. Confirm that parameter contexts and sensitive values are handled separately, because they do not travel inside the versioned flow.

Provenance querying

Provenance records each FlowFile's path through the flow, including receipt, routing, transfer and delivery events. During validation, the team queries provenance on both the old and new clusters to confirm that every source object reached its destination exactly once and that no events are missing. Provenance retention is finite, so decide how long the old cluster keeps its records before decommissioning, and export evidence for the cutover file before it ages out.

Flow configuration file

The flow configuration file holds the canvas definition, components and connections for each node. NiFi 2 uses flow.json.gz in place of flow.xml.gz, so a move that also changes versions must convert the file before the new node starts. Keep a checksummed copy from the old cluster, and confirm that node identities, cluster settings and property references are updated for the new hostnames and certificates. Treat direct edits as a last resort and test them on a non-production copy first.

How the cluster move runs in phases with a parallel run

The sequence keeps the source cluster carrying production data until the target has proven it runs the same flows, so each cutover step can be reversed. It ends with a validated target and a decommissioned source. The phases below are the order we would expect a move to follow, and the change windows should be confirmed against your own environment.

Inventory flows, queues and dependencies
Someone on the platform team, with flow owners, lists every process group, controller service, parameter context, sensitive value, certificate, hostname and external endpoint the cluster depends on. The inventory also records queue depths, repository sizes and which flows are primary-node-only. This step can run in parallel with target infrastructure provisioning. It is done when each flow has an owner, a criticality rating and a named external dependency list that the business has signed off.
Build the target cluster on the same version
Provision the target nodes in the new data center or cloud with the same NiFi version as the source and a Java 21 runtime, as the NiFi administration guide lists Java 21 as the minimum. Apply the same security configuration, coordination setup and repository layout, and confirm the node-to-node and cluster ports are reachable. This step can run in parallel with the inventory and needs no change window because the source is untouched. It is done when every target node starts, joins the cluster and passes a health check with an empty flow.
Pilot with non-critical flows
Deploy a small set of low-impact flows to the target, using versioned flows from your registry, and feed them with copied or test data. The pilot checks parameter contexts, controller services, certificate trust and DNS resolution against the new hostnames. Confirm with each flow owner that outputs match the source. This step needs no change window if the pilot uses separate endpoints. It is done when the pilot flows run through a full processing cycle with results the owners have accepted in writing.
Run both clusters in parallel
Connect the two clusters with Site-to-Site or dual-feed production sources so both process the same data while the target is validated. Monitor back pressure, error rates and downstream receipt on both sides, and keep the source as the system of record. Confirm that duplicate handling in each target system is understood before data flows to both. This step needs a planned window to change source routing, then runs for as long as validation requires. It is done when the target output matches the source across agreed checks for the whole parallel period.
Cut over node by node
Switch producers and consumers to the target, then drain and offload source nodes one at a time using the NiFi offload procedure so queued FlowFiles move to the remaining nodes. Remove each source node from the cluster only after its offload is confirmed. This step needs a change window with rollback steps written in advance, and the source must keep enough nodes to hold the load. It is done when all source nodes report offloaded and disconnected and all external traffic arrives at the target.
Validate with provenance
Use provenance on the target to trace sample FlowFiles from ingestion to delivery, and compare record counts, timestamps and content checksums with the source records for the parallel period. Check that every error path and retry was exercised and that certificates and hostnames appear in the expected events. Record the results for change and audit records. This step runs after cutover and needs no change window. It is done when the reconciliation report has no unexplained gaps and the flow owners sign it off.
Decommission the source cluster
After the validation period closes, stop the source nodes, retain their repositories and configuration under your retention policy, and remove DNS records, certificates and network rules that no longer serve traffic. Confirm that no scheduled job, partner integration or monitoring check still points at the source. This step needs a change window and a documented retention decision made by the data owners. It is done when the source is powered down, its retained artifacts are catalogued and the change record is closed.

What drives the cost and duration of a NiFi relocation

This table lists the factors that change the effort and cost of moving a running NiFi cluster, and whether each one is paid once or recurs. Use it to see which items your own environment makes large, and to know what to put in a written scope before asking for a quote.

Driver	Why it changes cost or effort	One-time or recurring	What to ask for in a quote
Queued data and repository size	Content, FlowFile and provenance repositories set how much data must be drained or copied, and how long cutover waits.	One-time	Repository sizes per node, queue depths by process group, and the drain or copy approach assumed.
Process groups and external endpoints	Each process group, controller service and remote endpoint must be inventoried, re-pointed and retested.	One-time	Inventory of process groups, controller services, remote process groups and external systems, with the owner of each.
Certificates and hostnames	Node certificates, truststores, keystores and DNS names change with the new site, and every client trust path must be updated.	One-time, with renewal recurring	Certificate authority plan, renewal process, and list of clients that pin or trust specific names.
Network path between sites	Cluster protocol ports, Site-to-Site and endpoint connectivity, firewall rules and routing decide whether both sites can run in parallel.	One-time, with monitoring recurring	Port matrix, firewall change requests, bandwidth and latency assumptions, and the parallel-run network design.
Version and configuration upgrades	Moving across a major version, such as from NiFi 1.x to 2.x, adds upgrade work on top of the move, including the flow file format change.	One-time	Current NiFi version, target version, and whether the upgrade is planned before, during or after the move.
Number of approving teams	Network, security, application, data owner and change boards each add review cycles and sign-off steps to the timeline.	One-time	Named approvers, their review steps, and the change windows they permit.
Validation depth	Provenance comparisons, record counts, checksum checks and load tests set how much evidence the cutover produces.	One-time, with ongoing checks recurring	Validation criteria per flow, the evidence delivered, and whether provenance retention covers the cutover window.

Questions teams ask before moving a NiFi cluster

What drives the cost of moving a NiFi cluster?

The cost depends mostly on how many nodes, repositories and flows are involved, how much data is queued at cutover, and how many external systems point at the cluster. The work splits into planning, building the target environment, staging and testing, the cutover itself, and post-move validation. Each part is paid for either by your team's time or by a partner's fixed-scope engagement, and the one-time migration effort is separate from any recurring operating cost on the new platform. Ask any quote to itemize these phases, list the systems that must be re-pointed, and state which validation evidence is included.

How long does it take to move a NiFi cluster?

Duration is driven by the number of flows, the volume of data in queues and repositories, the number of upstream and downstream connections that must change, and the approval steps your change process requires. Teams that run a staged cutover with Site-to-Site spend most of their time on parallel running and validation rather than the switch itself. Because the right answer depends on your environment, ask for a plan with phases and checkpoints written from your inventory, not a generic duration. Confirm the target platform's own onboarding steps with the vendor before committing a window.

What must our team provide before a NiFi cluster move starts?

You will need to provide an accurate inventory of flows, processors, controller services, parameter contexts and external endpoints, along with current node specifications and repository sizes. Access to the source and target networks, cloud or data center accounts, DNS control, and a certificate authority or certificate process is also required. Named owners for each upstream and downstream system, and a change window agreed with those owners, reduce delays. Confirm which flow definitions are already versioned, since an exported flow in version control makes rebuilding the target much simpler.

Can our own team move a NiFi cluster without a partner?

Yes, many teams perform the move internally, especially when they already run NiFi operations and have a tested runbook. The work is manageable when the cluster is small, the flows are well documented and the team has experience with TLS, cluster coordination and repository management. Consider a partner when the cluster is large, the cutover window is tight, regulated data is involved, or the team has not moved a production cluster before. Whichever path you choose, the decision should rest on a written scope that names who performs each step and who signs off.

How do we avoid losing data during a NiFi cluster move?

Data loss is avoided by deciding, for each queue and source, whether data is drained before cutover, moved with repositories, or replayed from the source system. Stop sources that cannot be replayed, let queues drain, and verify with provenance that every expected event reached its destination before decommissioning nodes. Use Site-to-Site for a staged cutover so that both sites can run during validation. Keep the old cluster intact until your reconciliation checks pass, and document the rollback path with explicit criteria for using it.

What changes for certificates and hostnames when a NiFi cluster moves?

Every node's TLS certificate must match the hostname or address that clients, peers and the load balancer use in the new location, so new certificates or subject alternative names are usually required. Update the node addresses in the cluster configuration, the Site-to-Site peer URLs and any client trust stores that pin the old names. Plan for overlap, where old and new certificates are trusted briefly, so that rolling changes do not break connections. Review the NiFi administration guide for the exact security properties your version uses, and confirm the certificate process with your security team before the first node is rebuilt.

Does moving NiFi to a new platform require upgrading the version?

Not necessarily, but the answer depends on the version you run. NiFi 1.x cannot be maintained further because its dependencies cannot be upgraded in that line, so a move is a good point to plan a path to NiFi 2. The migration guidance requires upgrading to NiFi 1.27.0 before moving to 2.0.0 and changing the flow file from flow.xml.gz to flow.json.gz. Mixing a platform move and a major version upgrade increases risk, so decide the sequence explicitly and confirm it against the current migration guidance.

What usually goes wrong during a NiFi cluster move

Most NiFi cluster moves fail in the same few places: data that never leaves the old nodes, copies of state taken while the cluster is still writing, and endpoints that were never updated. Each risk below has a prevention step that you can check before the cutover window opens.

Draining queues that never empty

Connections can keep receiving data from upstream sources, so a queue that looks drained may refill before a node is removed. Check which sources still feed the flow, pause or redirect them, and confirm that every connection on the outgoing node reaches zero before you offload. Watch for queues held by processors that wait on an external system, such as a stalled API, because those will not empty on their own and need a decision on retry, replay or discard.

Repository copies taken from a running node

Copying the flowfile, content or provenance repositories while NiFi is writing produces files that are internally inconsistent and can fail to load on the new host. Stop the node cleanly, or use the offload and rejoin path, before any copy is made. Record the checksums of what was copied and confirm the new node starts with a clean view of the cluster before it receives load.

Certificate chains that do not match new hostnames

Node certificates issued for old hostnames fail TLS checks as soon as nodes or the cluster endpoint move, and the cluster may split while it appears healthy from a single browser session. Issue new certificates with subject alternative names that cover each new node name and the load balancer name, and confirm the full chain is trusted by every node and by the truststore used for Site-to-Site. Verify the trust store contents before the first node joins.

Flows that assume local paths or one node

Processors that read from a local directory, write to a path on one host, or run on the primary node only will behave differently once the cluster moves. Review every processor that uses file system paths, mount points or host-specific credentials, and move those values into parameters or parameter contexts that differ per environment. Confirm primary-node-only processors start on the new primary and that load-balanced connections distribute work across the new node set.

External systems still using the old address

Upstream producers, downstream consumers, DNS records, firewall rules and monitoring targets often keep the old NiFi address long after the cluster has moved. Build an inventory of every system that sends to or reads from NiFi, including Site-to-Site peers and scheduled jobs. Plan a dual-running window with a DNS change or load balancer switch, and keep the old endpoint answering until the inventory shows no remaining traffic.

Validation that checks counts only

Matching record counts between the old and new sites can hide lost lineage, duplicated events or records routed to the wrong relationship. Compare provenance for a sample of flowfiles across the cutover, including the parent and child events that show where data was received, transformed and sent. Agree on which provenance events must appear at each site, and keep the old cluster's provenance available for the audit period your organization requires.

What the move asks of your team

Moving a running NiFi cluster asks your team for decisions and access before any node is touched. Someone must own the cutover plan, approve the change windows, and decide whether the new site is a rebuilt cluster that receives flows, or a relocated cluster that keeps its existing repositories. Those choices change what the work involves, so settle them in writing first. You will also need named owners for DNS, the load balancer or ingress in front of the UI and Site-to-Site listeners, the certificate authority, and the identity provider (LDAP, OpenID Connect, SAML 2.0 or X.509 client certificates, depending on how your clusters authenticate today).

On the infrastructure side, your team must provide accounts or hosts in the target environment, routed network paths between old and new nodes during the overlap period, and access to the existing ZooKeeper ensemble or Kubernetes namespace that coordinates the cluster. Confirm which downstream systems will be reached from the new site and whether firewall rules, allow lists and service accounts on those systems can be changed inside your change process. Ask for a change calendar that covers freeze periods for the data sources feeding NiFi, because queue drain and cutover timing depend on them.

DeployOpen can plan the move, prepare the target environment, run the node offload and decommission steps, adjust flow configuration, and perform validation with provenance, all inside infrastructure you control, including your cloud accounts, data centers or restricted networks. The work runs under the access you grant and the change procedures you already use. Your team keeps approval over production changes and over any step that touches keys, certificates or data-holding systems. Confirm these responsibilities, access levels and approval points in the written scope before work starts, since they determine how much of the cutover your staff must staff and watch.

How DeployOpen scopes, delivers and supports a NiFi cluster move

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own infrastructure, including their cloud accounts, data centers and air-gapped networks. For a NiFi cluster move, that means the work starts with a written scope covering node sequence, repository handling, certificates and provenance checks before any node is touched.

How we scope

Inventory the running flows, controller services, parameter contexts and external endpoints that the cluster depends on today.
Agree in writing which nodes move, which repositories are carried over and which queues are drained before cutover.
Confirm the target platform, network paths, hostnames and certificate authority, and record the NiFi version on both sides.

How we build

Build the target cluster in parallel, with TLS, authentication and cluster coordination configured before any flow is deployed.
Stage data movement with Site-to-Site or queue draining, and rehearse offloading and decommissioning on a non-production copy.
Test the cutover with provenance queries that confirm every sampled record was received, processed and delivered at the target.

How we hand over

Deliver a runbook listing each cutover step, its owner, its rollback point and the evidence it produces.
Document the final topology, certificate inventory, DNS and load balancer changes, and the retained provenance records.
Walk your operations team through monitoring, alerts and the decommissioning checklist for the old site.

How we support

Fixed-scope engagements end with the agreed deliverables, and any remaining work is quoted separately from the written scope.
Ongoing support, upgrades and continuous testing are available as a retainer after the move is complete.
Confirm with DeployOpen what post-cutover monitoring, patching and NiFi version upgrades would be included before signing.