Migrate Apache NiFi from virtual machines to Kubernetes

Moving a VM-based NiFi cluster to Kubernetes is mostly a storage, coordination and cutover problem. This page explains what maps cleanly, what needs redesign and how to plan the parallel run.

On this page

What moving a VM-based NiFi cluster to Kubernetes involves

Moving a VM-based NiFi cluster to Kubernetes usually keeps the flows themselves largely intact. The work sits elsewhere: giving each node persistent volumes for its repositories, replacing ZooKeeper with Kubernetes-native coordination, placing session affinity at the ingress so that client sessions reach the node that holds them, moving certificates and passwords into Secrets, and draining or transferring queued data without losing it. A team that plans around those five areas will find the flow review is the smaller part of the project.

The effort depends on three things you should measure before scoping. The first is cluster size and how many repositories, content and provenance volumes each node holds, since those decide the storage design. The second is flow complexity, especially primary-node-only processors, load-balanced connections and any flow that assumed one node. The third is the current state of your ZooKeeper ensemble and certificate setup. A cluster with a clean ZooKeeper deployment and certificates issued by a known authority is a much smaller job than one with hand-edited properties files and expired or self-signed certificates. The figures and settings described on this page come only from the NiFi and Kubernetes documentation, so confirm sizing, storage classes and ingress behavior against your own environment and the vendor documentation for your chosen distribution.

Treat the NiFi 1.x to 2.x question as a separate decision. A move to Kubernetes is often the point where teams also plan an upgrade, but the two changes have different risks. NiFi 1.28 is the last minor release of the 1.x series, and the project states that 1.x cannot be maintained further because of its Jetty, Spring and AngularJS dependencies. Moving to 2.x requires going through NiFi 1.27.0 first and changing the flow configuration file format, so combining both changes in one cutover makes it hard to tell which change caused a fault. Decide early whether the platform move happens on the current version, with the upgrade scheduled as a later phase, or whether both happen together, and record that choice before any cluster is built.

Which VM-based NiFi capabilities map to Kubernetes and which need redesign

This table pairs each capability of a VM-based NiFi cluster with its Kubernetes counterpart and marks where the move needs redesign rather than a direct translation. Read the middle column as the current state and the right column as the target, then check the last row in each group for work that a lift-and-shift will not cover. The main takeaway is that node-local state, host-bound identity and host scripts need deliberate replacements before cutover.

CapabilityOn virtual machinesOn Kubernetes
Repositories and stateFlowfile, content and provenance repositories on local disks of each hostOne persistent volume claim per node for each repository, with storage class and reclaim policy chosen up front
Cluster coordinationZooKeeper ensemble for coordination and leader electionKubernetes Leases or ConfigMaps for state and leader election, with a service account that holds the required RBAC permissions
Node identity and hostnamesStable hostnames and host certificates tied to each machineStable pod names from a StatefulSet, with certificates issued for the headless service names that nodes advertise
Process managementsystemd unit files that start, stop and restart NiFiStatefulSet ordering and rolling updates, with readiness and liveness probes that reflect cluster membership
Client access and session affinityLoad balancer or reverse proxy with sticky sessions configured by handIngress with session affinity configured on the service, verified against the UI and API paths used by operators
Certificates and credentialsKeystore and truststore files on disk, managed by handKubernetes Secrets mounted into pods, with encryption at rest and least-privilege RBAC configured on the cluster
Network controlsHost firewall rules between nodes and to upstream systemsNetwork policies and service exposure, which must be confirmed against the cluster network plugin in use
Backups and maintenanceCron jobs and shell scripts that copy repositories and configurationScheduled jobs and volume snapshots, which need validation because snapshot consistency for a running cluster must be confirmed
Node-local paths in flow.json.gzAbsolute paths to host directories, scripts and certificates referenced inside flows and controller servicesPaths must be replaced with mounted volume paths or parameters, and every reference reviewed before the first pod starts

Documented paths for moving flows and NiFi configuration

NiFi documents several ways to carry flows and configuration forward, but none of them converts a VM installation into a Kubernetes deployment in one step. The cards below cover the paths the project describes, the configuration format change that applies to any move, and the parts you will copy by procedure. Treat each path as one input to the cutover plan, and confirm the steps against the version you run in each environment.

Flow configuration file and version steps

The NiFi migration guidance describes moving the flow configuration from flow.xml.gz to flow.json.gz, and requires upgrading to NiFi 1.27.0 before moving to 2.0.0. On a VM, the flow file sits in the flow configuration path of each node, so copying it onto a new pod volume is only valid after the version path is complete. Check that every node carries an identical flow before you copy it, and keep the original file untouched as a rollback point.

Format change applies to every move

The move from flow.xml.gz to flow.json.gz is required for any flow move in the 2.x line, not only for this one. A flow that reaches a Kubernetes volume still has to be in the 2.x format before NiFi starts from it. Plan the conversion as a separate, reviewed step with its own test run, and verify that processors, controller services and parameter contexts load without errors before the pods receive traffic.

Versioned flows through Git registry clients

NiFi 2 includes Flow Registry Clients for GitHub and GitLab that store versioned flows in a Git repository. If your VM flows are already under version control, the cleanest route is to deploy from that repository into the new cluster and then compare the running flow with the committed version. Confirm which registry client, repository and branch your team uses, and whether sensitive parameter values are excluded from commits.

NiFi CLI for flow deployment

The NiFi toolkit CLI can import, start and inspect versioned flows against a running cluster, which makes it useful for repeating the same deployment across environments. Use it to rebuild the process group structure after the cluster is up, rather than copying internal state by hand. Confirm the CLI version matches the NiFi release you run, and script the steps so the cutover can be rehearsed in a staging cluster first.

Kubernetes tooling for manifests

Kubernetes manifests for StatefulSets, Services, ConfigMaps, Secrets and Leases are applied with kubectl, typically from a Git-managed set of files. Keep the NiFi properties, certificate references and repository volume claims in the same reviewed change set, so the cluster state can be reproduced. Confirm which Kubernetes release your cluster runs and check the manifest API versions against it before applying anything.

No automated VM importer

No automated VM-to-Kubernetes importer is documented by the NiFi project. Content repositories, provenance repositories, flow file repositories and state must be copied or drained by procedure, and each copy needs a documented checksum and rollback point. Ask any partner or tool vendor to show exactly which files move, which are regenerated on first start, and which must be empty before the cutover begins.

How the migration runs from inventory to decommission

The sequence moves a running VM-based NiFi cluster onto Kubernetes in gated phases, so each stage produces evidence that the next one can rely on. Every phase closes against exit criteria agreed with the client before work starts.

  1. Inventory flows, repositories and cluster state

    Map every flow, process group, parameter context, controller service and custom NAR on the current cluster, along with the size and location of the flowfile, content, provenance and database repositories on each node. Record the certificates and hostnames in use, the ZooKeeper ensemble and znodes that hold cluster state, and the queue depth on each connection. The output is a written inventory that the client signs off, including a list of components that will not run unchanged on the target. Confirm with the client which NiFi version is in production, because the version path must be settled in this step.

  2. Settle the NiFi version path before cutover

    Confirm the running NiFi version and decide the upgrade route before any cluster is built. The NiFi migration guidance requires upgrading to NiFi 1.27.0 before moving to 2.0.0, and changing the flow configuration file from flow.xml.gz to flow.json.gz. NiFi 1.x can no longer be maintained, so a 1.x cluster should not be the long-term target. Agree the target version, the test order of upgrades and the rollback point with the client. The exit criterion is a written upgrade plan, reviewed against the flows and custom code captured in the inventory.

  3. Build a pilot cluster in its own namespace

    Stand up a pilot NiFi cluster on the chosen Kubernetes distribution in a dedicated namespace, with a service account scoped to the Lease and ConfigMap permissions NiFi needs for coordination. Store keystore, truststore and sensitive parameter values as Kubernetes Secrets, and plan encryption at rest and least-privilege RBAC for them before anything is loaded. Persistent volumes are bound to each repository per node. Import a representative subset of flows and confirm that the nodes elect a primary, join the cluster and pass health checks. The exit criterion is a pilot that starts, clusters and processes test data without manual repair.

  4. Run in parallel with source-side routing

    Route a defined subset of live data to the Kubernetes cluster while the VM cluster continues to serve the rest. Use Site-to-Site or source-side routing so the same input is either duplicated or split by an agreed rule, and confirm that downstream targets accept writes from both clusters without conflict. Compare outputs record by record using provenance and target-side counts over a period the client agrees. Ingress session affinity is tested here, since NiFi UI and cluster traffic depend on requests reaching the node that holds their state. The exit criterion is matching output counts and no unexplained provenance gaps.

  5. Confirm the ZooKeeper to Lease state move

    Verify that cluster coordination on Kubernetes uses Leases and ConfigMaps rather than the ZooKeeper ensemble, and that the state migration from ZooKeeper has been performed and checked on the pilot. Confirm the leader election settings, the lease prefix if several NiFi clusters share a namespace, and that primary-node-only processors start on exactly one node after a forced primary failover. The exit criterion is a documented failover test that the client has witnessed, with the ZooKeeper ensemble still available as a rollback path until cutover completes.

  6. Drain queues or transfer repositories at cutover

    At the agreed cutover window, stop new intake to the VM cluster and choose the transfer method. Draining lets existing queues empty through the flows, which is slower to verify but leaves the least state to move. Transferring repositories copies flowfile, content and provenance data into the persistent volumes, which keeps queued work intact but needs integrity checks on every node. Either way, reconcile connection counts before and after, and sign off the cutover only when the queue depth reaches the value agreed in the inventory and the Kubernetes cluster accepts the full intake.

  7. Decommission the virtual machines

    Keep the VM cluster in a read-only or stopped state for the rollback period agreed with the client, then remove nodes from the cluster, revoke the old certificates and hostnames, and retire the ZooKeeper ensemble if no other system depends on it. Archive the configuration, repository snapshots and provenance exports required for the client's records. Update monitoring, backup jobs and runbooks to point to the Kubernetes cluster. The exit criterion is a signed decommission record listing every retired component and where its evidence is stored.

Cutover options for queued data: draining versus moving repositories

Draining means you stop new data from entering the VM cluster, let every queue empty through the existing flows, and only then start the Kubernetes cluster as the sole owner of the sources. Nothing is copied between clusters, so the new cluster starts with empty repositories and the cutover is a sequence of controlled stops. The cost is time: the window ends when the slowest queue drains, and sources that cannot be paused, such as external senders pushing to a listening port, need a separate holding point.

  • Downtime: Ingestion stops on the VM cluster before the switch, so the outage is the drain period plus the time to start and verify the Kubernetes flows.
  • Duplicate risk: Low, because each FlowFile is processed by exactly one cluster, but producers that retry during the pause can create duplicates at the destination.
  • Provenance evidence: Provenance on the VM cluster stays complete for the drained data, and you can export the final event history before decommissioning.
  • Queue behavior: Back-pressure can hold a queue open for long periods if a downstream system is slow or down, so the drain may never finish on its own.
  • Sources you cannot pause: Listening processors and external pushes need a redirect or a buffer in front of NiFi during the pause.

Confirm with your flow owners which queues feed systems that are slow to recover, which sources can be stopped without losing data, and how long a full drain has taken in your last test.

What drives the effort and cost of the move

This table lists the factors that change the size of a VM-to-Kubernetes NiFi move, with who carries the work, whether each cost is one-time or recurring, and what to ask for in a written quote. Use it to compare your own inventory against the scope a partner or internal team would price. The main takeaway is that queue depth, repository size and Kubernetes readiness usually matter more than the number of flows.

DriverWhy it changes effort or costOne-time or recurringWhat to ask for in a quote
Node count and repository sizesEach node needs its own persistent volumes for the content, flowfile, provenance and database repositories, and larger repositories slow copy and validation.One-time (migration), recurring (storage)Per-node volume plan, sizes, storage class and the copy or rebuild method for each repository.
Queue depth at cutoverQueued FlowFiles either drain before the move or must be carried across, which sets the length of the change window and the risk of duplicates.One-timeDrain approach, target queue size at cutover, and how duplicate handling is verified afterward.
Number of flows and custom processorsEach flow needs review for paths, local file references and NARs, and custom processors must be rebuilt into a container image.One-timeFlow inventory count, custom NAR list, and which items are retested against the new runtime.
NiFi version change during the moveMoving from 1.x to 2.x adds the 1.27.0 step, the flow.json.gz conversion and renamed components, which adds test cycles.One-timeTarget version, upgrade path steps, and whether the platform move and version change are split into separate cutovers.
Certificate and hostname changesNew hostnames, ingress routing and certificates change the TLS trust chain and the node addresses NiFi advertises to peers.One-time, recurring (renewal)Certificate authority used, renewal method, and the list of hostnames each node advertises.
ZooKeeper state migrationMoving from ZooKeeper to Kubernetes Leases requires permissions for the service account and a controlled move of cluster state.One-timeConfirmation of the leader election setting, service account role scope, and the state migration test.
Kubernetes cluster readinessControl plane layout, etcd backups, the chosen distribution and version skew policy determine whether NiFi can run safely before the workload is moved.One-time (setup), recurring (upgrades, backups)Distribution name, control plane topology, etcd backup schedule and restore test, and upgrade plan.
Secrets and certificate storageKeystore, truststore and sensitive properties must be stored as Secrets, which are unencrypted in etcd by default unless encryption at rest is configured.One-time, recurring (rotation)Secret store choice, encryption at rest status, and RBAC scope for listing and watching Secrets.
Change windows and rollbackThe number of approved windows, and whether the VM cluster stays available for rollback, sets how many parallel runs and rehearsals are needed.One-timeNumber of windows, rollback criteria, and how long the VM cluster is kept running after cutover.

What usually goes wrong when NiFi moves onto Kubernetes

Most failed NiFi moves onto Kubernetes trace back to a few decisions made during the lift: where repositories live, how traffic reaches each node, how the cluster elects a leader, and how credentials are stored. Each card below names one of these failure points and the check to run before cutover.

Repositories on ephemeral storage

A NiFi pod that starts with an emptyDir or the container's writable layer loses its FlowFile, content, provenance and state repositories when it is rescheduled. Queued data on that node is then gone, and the node may rejoin the cluster with a flow it cannot reconcile. Bind every repository directory to a PersistentVolumeClaim and confirm the storage class supports the volume modes and reclaim policy you intend. Test a forced pod deletion on a staging cluster before go-live.

Duplicate or lost data during drain

Moving a node by draining it while flows still hold data can either drop FlowFiles that were never persisted or deliver the same records twice when upstream sources retry. The risk grows when queues are large and the cutover is not sequenced. Stop ingest at the sources, let queues drain to zero, and only then move nodes. Compare record counts and provenance events between the VM cluster and the Kubernetes cluster for a sample window.

Missing session affinity at the ingress

The NiFi UI keeps user state on the node that served the login, so requests load-balanced to a different node fail with authorization or session errors. The problem often appears only after the first node restart. Configure session affinity at the ingress controller and confirm the cookie or hash rule survives the controller version you deploy. Test a login followed by flow editing across several browser sessions.

Leader election with too few permissions

When NiFi uses Kubernetes Leases or ConfigMaps for coordination, the pod's service account must read and write those objects in its namespace. A role that grants only read access lets nodes start but leaves no elected coordinator, so the cluster stays disconnected. Bind a Role that grants lease and ConfigMap verbs to the NiFi service account only, and check the lease name prefix when several clusters share a namespace.

Secrets unencrypted in etcd

Kubernetes stores Secrets unencrypted in etcd by default, so keystore passwords, sensitive parameter values and OIDC client secrets are readable by anyone who can read etcd backups or list Secrets. NiFi certificates mounted from these objects inherit the same exposure. Enable encryption at rest for etcd, restrict list and watch on Secrets with least-privilege RBAC, or source material through the Secrets Store CSI Driver. Confirm backups of etcd are encrypted too.

Kubernetes version outside the supported set

Running a cluster on a minor version that has passed end of life leaves the control plane without security patches, and skew rules can block the upgrades you need later. NiFi itself also has a Java runtime requirement that must match the node image. Pick a Kubernetes minor version from the currently maintained set and record its end-of-life date before building manifests. Plan upgrades one minor version at a time, control plane first.

Questions buyers ask before moving NiFi to Kubernetes

How much does moving NiFi from virtual machines to Kubernetes cost?

The cost depends on the work scope, not on a standard price. A move has one-time costs: discovery of flows, parameters and certificates, building the Kubernetes manifests or Helm values, persistent volume design, the leader election change, testing, the parallel run and the cutover. Recurring costs are the Kubernetes cluster itself, storage, monitoring, and the people who patch NiFi and Kubernetes afterward. The main drivers are the number of flows, how many processors use external credentials, how many nodes and repositories exist, whether a 2.x upgrade is included, and how many cutover windows the business accepts. Ask any provider for a quote that lists these items separately, with what is delivered and what the client must supply. DeployOpen issues quotes from a written, fixed scope.

How long does a NiFi move to Kubernetes take?

Duration varies with the same factors that drive cost, and the largest one is how well the current flows are documented. A cluster with a small number of well-owned flows and clean parameter contexts moves faster than one with undocumented processors, hand-edited flow.xml.gz files or flows that depend on a single node. Other factors include the number of change approvals required, how long the business allows a parallel run, the Kubernetes platform already in place, and whether the team is also moving from ZooKeeper to Leases in the same window. Ask for a timeline tied to phases: inventory, target build, test cutover, parallel run and production cutover. Each phase should name its exit criteria.

What must our internal team provide for the migration?

Your team provides the decisions and access that only your organization can give. The usual items are:

  • Flow owners who can confirm what each flow does, which systems it reads and writes, and what counts as a successful run
  • Cluster access, including read access to the running NiFi nodes, the flow definition, parameter contexts and the repositories
  • Kubernetes access: a cluster or namespace, service accounts with the permissions needed for Leases and ConfigMaps, and storage classes for persistent volumes
  • Certificate authority details, hostnames and trust stores
  • Change approvals for the parallel run and the cutover window

Agree in writing who signs off each flow after the parallel run, because that sign-off is what closes the project.

Can our own engineers move NiFi to Kubernetes without a partner?

Yes, if the team has NiFi administration experience, working knowledge of Kubernetes workloads, persistent volumes, Secrets and RBAC, and time to test flows properly. The work is not only deployment. It includes preserving the flow definition and parameter contexts, moving four repositories with their data, changing cluster coordination, rebuilding the ingress so each client stays on the same node, and proving that queued data is not lost or duplicated at cutover. Engineers who have not run a NiFi cluster in production often underestimate the cutover validation. A useful self-assessment is to list who owns each flow, who can restore a node, and who can explain the provenance of a sample record. If any of those answers are unclear, a partner can cover the gap.

Do we need to upgrade to NiFi 2.x before moving to Kubernetes?

Not necessarily, but the two decisions should be made together. The NiFi downloads page states that NiFi 1.28 is the last minor release of the version 1 series, that 1.28.1 reached end of support on December 8, 2024, and that users are strongly encouraged to upgrade to NiFi 2. The project explains that 1.x cannot be maintained further because it depends on Jetty 9.4, Spring Framework 5.3 and AngularJS 1.8. The migration guidance requires upgrading to NiFi 1.27.0 before moving to 2.0.0 and changing the flow configuration file from flow.xml.gz to flow.json.gz. Moving to Kubernetes on 1.x keeps the support gap open, while combining the platform move and the 2.x upgrade increases the number of changes that must be tested at once. Confirm the sequencing with your team before the plan is fixed.

How do we validate the parallel run and cutover of queued data?

Validation starts with a written definition of success for each flow: expected record counts, duplicate handling, ordering needs and the systems that confirm delivery. During the parallel run, the new cluster processes a copy or a controlled share of traffic while the old cluster continues. Compare outputs at the destination, use NiFi provenance on both sides to trace sample records, and review error and retry queues. For cutover, decide how queued FlowFiles are handled, whether they are drained on the old cluster or moved, and what the rollback point is. Repository data stored on persistent volumes must be backed up before the switch. Confirm with your engineers which provenance retention and back pressure settings apply, and rehearse the rollback at least once in a non-production environment.

Which security items should we check before cutover?

Check the items that change when NiFi moves into Kubernetes. Certificates and keystore passwords should be stored as Kubernetes Secrets, and Kubernetes stores Secrets unencrypted in etcd by default, so encryption at rest and least-privilege RBAC for list and watch access are needed. The NiFi administration guide lists the authentication options available, including LDAP, OpenID Connect, SAML 2.0 and X.509 client certificates, and notes that the Kerberos provider is deprecated. Confirm the authentication provider, the ingress path and session affinity, and that the pods run under the pod security settings your policy requires. Ask your security team to review these items against your own control framework, since DeployOpen prepares evidence but does not issue audit reports or certifications.

Whether the Kubernetes platform is ready to host NiFi

Before any NiFi pod is scheduled, the target cluster has to clear a short set of platform checks. The first is version support. Confirm that the Kubernetes minor version you plan to run is one the project still patches. The project maintains the three most recent minor releases, and each release receives about one year of patch support. Check the end-of-life dates on the Kubernetes releases page against your NiFi go-live and the planned upgrade window, because a cluster that falls out of support during the migration forces a second move. The version skew policy also governs how you upgrade the control plane and kubelets later, so ask the platform owner to show the upgrade order they will follow.

Next, confirm the control plane is highly available and recoverable. Ask whether etcd is stacked on the control plane nodes or external, and whether there are at least three control plane nodes behind a TCP load balancer on port 6443. NiFi stores its coordination state in Leases and ConfigMaps, so an unhealthy API server will stall leader election for the whole cluster. Ask for the date of the last successful etcd snapshot and a record of a restore test, not just a backup job. Kubernetes Secrets are stored unencrypted in etcd by default, so confirm that encryption at rest is enabled, because the NiFi certificates and sensitive property keys will live there.

Finally, confirm identity, access and conformance. Identify how human administrators authenticate to the cluster, whether through OpenID Connect, X.509 client certificates or another supported method, and who can read Secrets in the NiFi namespace. Confirm that Pod Security Standards are enforced on that namespace at a level that allows NiFi's required settings, and that a dedicated service account for NiFi has only the RBAC rights needed for Leases, ConfigMaps and its own pods. Last, ask the distribution vendor for its CNCF Certified Kubernetes conformance listing for the exact version you run. Conformance shows the required APIs are present, but it does not confirm your hardening, so treat it as one input to your own checks.

How DeployOpen scopes and delivers the VM-to-Kubernetes move

DeployOpen plans, builds, tests and hands over self-hosted NiFi and Kubernetes work inside infrastructure you control. For a VM-to-Kubernetes NiFi move, we start with a written scope that covers repositories, coordination, ingress, certificates and the cutover of queued data before any cluster is built.

How we scope

  • We inventory the existing NiFi nodes, repository locations, flow files, ZooKeeper dependency and the processors that assume a single node.
  • We agree in writing which flows, clusters, namespaces and ingress paths move, and which stay on the VMs during parallel running.
  • We confirm with you the target Kubernetes distribution, its version and the access model your platform team will grant for the engagement.

How we build

  • We build the Kubernetes deployment with persistent volumes for each repository, Lease-based leader election, and service accounts scoped to the needed resources.
  • We configure session affinity at the ingress and manage certificates and passwords through Kubernetes Secrets with encryption at rest enabled.
  • We test the cutover runbook in a non-production environment, including queue drain, node offload and rollback steps agreed in the scope.

How we hand over

  • You receive deployment manifests or charts, configuration records, and a runbook covering the cutover sequence and rollback decision points.
  • We document the verification checks used to confirm flow state, provenance continuity and cluster health after each migration step.
  • We walk your operations and security staff through the Kubernetes and NiFi configuration so your team can run the platform after handover.

How we support

  • Engagements are fixed-scope, so changes after kickoff are handled through a written scope amendment with its own deliverables and timeline.
  • Ongoing support, NiFi and Kubernetes upgrades, and continuous testing are available as a retainer after the migration closes.
  • Before you request a quote, confirm your Kubernetes support window and NiFi version path with the vendor or project so the plan reflects current releases.

Sources and further reading

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.