Replacing ZooKeeper with Kubernetes Leases for NiFi clustering

Moving a NiFi cluster from ZooKeeper coordination to Kubernetes Leases and ConfigMaps is a configuration and permissions change with a state migration and a cutover plan. This page sets out what maps, what does not, and what drives the effort.

On this page

What moving NiFi clustering from ZooKeeper to Kubernetes Leases involves

The coordination switch is a configuration and platform change, not a flow rewrite. On Kubernetes, the NiFi Administrator's Guide sets nifi.cluster.leader.election.implementation to KubernetesLeaderElectionManager, which creates and manages Kubernetes Leases for cluster coordination and primary node tracking. The change also needs the NiFi pods to have a service account with permission to read and write Leases and ConfigMaps in their namespace, and the cluster state moves to the Kubernetes ConfigMap state provider rather than the ZooKeeper state provider. Your flows, processor configuration and controller services generally stay as they are. What you must plan is the platform underneath: how the namespace, RBAC bindings and pod identity are created, and who owns them after handover.

The effort depends mostly on the Kubernetes platform you run NiFi on, not on NiFi itself. Teams that already operate a hardened cluster with managed storage, a tested ingress that supports session affinity and a reviewed RBAC model will find the switch narrow. Teams that are still building that platform should expect the Kubernetes work to dominate. Confirm with your platform team which Kubernetes version and distribution you will use, and check the current support window in the Kubernetes releases page before you commit to a version, since patch support for each minor release is limited. Ask whether the lease permissions can be scoped to the NiFi namespace, because broad cluster-wide Lease rights are a common finding in access reviews.

The data side needs its own cutover plan. Repositories, the flow definition and the state held in ZooKeeper do not move automatically with the coordination setting. Decide whether ZooKeeper-held state is recreated or migrated, and schedule the switch in a window where queued data can drain or be moved deliberately. Run a rehearsal in a non-production namespace and verify that exactly one node holds the primary role after startup, that all nodes rejoin, and that primary-node-only processors resume once. Keep ZooKeeper available until that verification passes, so you can roll back. Ask your NiFi vendor or partner to document which settings change, which secrets and certificates are mounted, and how each step is verified.

Which ZooKeeper coordination and state features map to Kubernetes Leases and ConfigMaps

This table compares the ZooKeeper coordination and state features of a NiFi cluster with their Kubernetes counterparts. Read each row as a question to answer for your cluster: what the NiFi property becomes, where state lives, and what still needs confirmation against your NiFi version. The main takeaway is that leader election and state move to Kubernetes objects, but lease prefixes and state provider settings need deliberate planning.

FeatureZooKeeper (embedded or external)Kubernetes (Leases and ConfigMaps)What to confirm
Leader election settingCoordination through the ZooKeeper election manager, configured with ZooKeeper connect string and root settingsSet nifi.cluster.leader.election.implementation to KubernetesLeaderElectionManagerExact property names and defaults for your NiFi 2.x release in the administrator's guide
Primary node trackingPrimary node role is elected and tracked through ZooKeeper znodesPrimary node tracked through a Kubernetes Lease object created and managed by NiFiHow failover timing behaves in your cluster size during test runs
Cluster coordinationNode membership and election state held in ZooKeeper ensembleCluster coordination held in Leases managed by NiFiWhether any ZooKeeper settings remain required for other components
Cluster state locationZooKeeper state provider stores component state in ZooKeeperKubernetes ConfigMap cluster state provider stores state in ConfigMapsWhich state provider each processor and flow component uses after migration
Several clusters in one namespaceSeparate ZooKeeper root paths keep clusters apartUnique nifi.cluster.leader.election.kubernetes.lease.prefix per clusterPrefix values are unique and documented before any cluster starts
Service account permissionsNot applicable at the Kubernetes API levelRole granting Lease and ConfigMap access in the namespace, bound to the NiFi service accountLeast-privilege verbs and resource names reviewed by your Kubernetes team
Existing state migrationState held in ZooKeeper and local repositoriesState must be rebuilt or copied into ConfigMaps, which NiFi does not copy automatically from ZooKeeperMigration path and data loss risk for stateful processors confirmed with the vendor docs
Operational visibilityOperators inspect ZooKeeper znodes with ZooKeeper toolingOperators inspect Leases and ConfigMaps with kubectl and existing cluster monitoringAlerting on lease renewal failures and ConfigMap write errors in your monitoring stack

How the NiFi leader election properties change between the two models

Keeping ZooKeeper is the lower-change option when the cluster already runs on virtual machines or when your platform team already operates a ZooKeeper ensemble. In this model the leader election settings stay on their ZooKeeper defaults, and the coordination endpoint is the connection string your nodes already use. An embedded ZooKeeper setup places the ensemble on NiFi nodes, while an external setup points every node at a separately managed ensemble.

  • Election implementation: the nifi.cluster.leader.election.implementation value stays on the ZooKeeper-based option rather than the Kubernetes option.
  • Connection string: the ZooKeeper connect string and timeouts remain in nifi.zookeeper.connect.string and related keys.
  • Lease prefix: the nifi.cluster.leader.election.kubernetes.lease.prefix property does not apply, so no per-cluster prefix is set.
  • State provider: the ZooKeeper state provider remains the cluster state store unless you change it deliberately.
  • Operations: you keep monitoring and backing up the ZooKeeper ensemble as a separate service.

Before relying on this model, confirm the property names and defaults for your NiFi version in the NiFi Administrator's Guide, since the Kubernetes options are documented in the same section.

Which Kubernetes permissions the NiFi service account needs

NiFi on Kubernetes needs its service account to read and write Leases for leader election and primary node tracking, and it may need ConfigMap access for state. These permissions are granted through a Role and RoleBinding scoped to the NiFi namespace, so the review should cover each permission area separately and confirm it against the documentation and your cluster policies.

Lease access for leader election

NiFi's KubernetesLeaderElectionManager creates and manages Kubernetes Leases for cluster coordination and primary node tracking. The Role must grant the lease verbs and resources that NiFi uses in the NiFi namespace, and nothing cluster-wide. Confirm the exact verbs against the NiFi Administrator's Guide and your cluster's policies before you write the Role, because over-broad verbs such as wildcards are the most common reason this review fails.

ConfigMap access for cluster state

If you use the ConfigMap cluster state provider instead of ZooKeeper for state, the service account needs read and write access to the ConfigMaps that hold that state. Scope these rules by resource name where your cluster supports it, so that one NiFi cluster cannot read or change another cluster's state. Confirm which ConfigMap operations the provider performs in your NiFi version before you restrict them.

Namespace-scoped Role and RoleBinding

Use a Role and RoleBinding rather than a ClusterRole, so the permissions apply only to the namespace where NiFi runs. Bind the Role to the NiFi service account that the pods actually use, and check that the pod spec does not fall back to the default service account. When several NiFi clusters share a namespace, make sure each lease prefix maps to resources the bound identity is allowed to touch.

Least-privilege review of the service account

Apply least privilege by listing every API call NiFi makes during startup, election and state writes, then granting only those. Do not add permissions for convenience or to troubleshoot, and remove any leftover bindings from earlier test deployments. Ask your platform team to record who approved the Role, when it was last reviewed and which NiFi version it was checked against.

Pod Security Standards alignment

The pods that run NiFi should meet the namespace's Pod Security Standards level, which sets limits on privileged settings, host access and capabilities. Confirm with your NiFi image and init containers that the chosen level is compatible before enforcing it. A permissive pod profile combined with broad RBAC widens what a compromised NiFi pod can reach, so review both together.

Periodic RBAC and access audit

Permissions drift as NiFi versions change and as teams add clusters to the same namespace. Schedule a review of the Role, RoleBinding and the service accounts that hold them, and check audit logs for unexpected lease or ConfigMap writes. Ask your team to confirm how the audit findings are tracked and who signs off on changes before the next NiFi upgrade.

How to plan the switch with a parallel check and a controlled cutover

This sequence moves NiFi cluster coordination from ZooKeeper to Kubernetes Leases and ConfigMaps without losing flow state or running two primaries. Each step ends with a check that the team can record before moving on.

  1. Inventory current ZooKeeper settings and state

    The platform engineer records every ZooKeeper-related property in each node's nifi.properties, including the connect string, the root node, and whether ZooKeeper is embedded or external. The team also records the state providers in use, the cluster size, and which flows depend on primary-node-only processors. The inventory is reviewed by the NiFi administrator and the owner of the ZooKeeper ensemble. It is complete when every node's configuration is accounted for and each setting has a named target in the Kubernetes design.

  2. Build the Kubernetes coordination configuration in a non-production namespace

    Create a non-production namespace and set the leader election implementation to KubernetesLeaderElectionManager. Assign a unique lease prefix if other NiFi clusters share the namespace, and bind a service account with only the permissions needed to manage Leases and ConfigMaps. Confirm the cluster state provider target in the NiFi administration guide before writing the values. The check is that a test cluster starts, forms membership and writes its Leases without errors in the logs.

  3. Rehearse the state migration on a copy

    Take a copy of the flow definition, the cluster state, and the repositories from the production-like source, then run the state migration in the non-production namespace. Confirm the flow file format matches what the running NiFi version expects, and keep the original ZooKeeper data untouched. The engineer records how long the copy took and which steps needed manual intervention. The rehearsal is complete when the migrated copy starts with the same flows, parameters, and component states as the source.

  4. Run a parallel validation window

    Keep the ZooKeeper-based cluster serving production traffic while the Kubernetes-coordinated cluster runs against representative data or a mirrored input. Compare flow outputs, queue depths, and provenance records between the two. Watch for duplicate processing, since two clusters reading the same source can double-deliver. The window closes when outputs match the agreed acceptance criteria and the team has reviewed the findings with the data owners.

  5. Cut over with a documented rollback path

    Stop intake to the ZooKeeper-based cluster, drain or hand off queued data according to the cutover plan, and start the Kubernetes-coordinated cluster. Confirm that exactly one node reports the primary role and that every other node reports as connected. Verify flow state after restart, including running status, controller service states, and parameter values. Keep the ZooKeeper cluster stopped but intact until the rollback window in the change plan has passed.

  6. Confirm a single primary and flow state after restarts

    Restart individual pods and the whole cluster in a controlled order, then check after each restart that a single primary node holds the lease and that primary-only processors run on that node alone. Confirm that flow state, component states, and queues survive the restart without loss. Record the results against the acceptance criteria. This check is repeated after any change to the service account or the lease prefix.

  7. Decommission ZooKeeper after verification

    Remove ZooKeeper only after the cutover has passed every check and the rollback window has closed. Archive the final ZooKeeper data and configuration, remove the connect string from all nodes, and delete the ZooKeeper workloads and their storage according to the retention policy. The change record should show who approved the decommission and where the archive is stored. The step is complete when no NiFi node references ZooKeeper and a fresh restart still forms one healthy cluster.

Moving existing cluster state without losing flow data

Switching coordination does not move flow data. The ZooKeeper-to-Lease change affects cluster coordination: which node is primary, cluster membership and the state that NiFi keeps for components that need it. Coordination data is small and can be rebuilt once the new election manager starts. The flow itself lives elsewhere. The flow configuration file (flow.xml.gz on 1.x, or flow.json.gz on 2.x) defines the canvas, and the FlowFile, content and provenance repositories hold data in transit and its history. Those repositories sit on each node's local disk or volume, so a pod rescheduled without its persistent volume starts with an empty repository. Before any change, confirm which state provider each cluster uses, whether components store state in ZooKeeper, and where the ConfigMap cluster state provider will hold it instead.

Queued FlowFiles are the main data risk. Data waiting in connection queues is not replicated by the leader election switch and is not moved by a rolling restart unless the nodes keep their repositories. Plan a drain before cutover: stop sources, let downstream processors empty queues, and confirm queue counts are zero in the NiFi UI. Where a queue cannot drain, decide whether the repository volumes move with the nodes or whether the data is replayed from its source. Provenance can show what was delivered and what is still pending, which is useful evidence for the change record. Confirm the exact state migration path for your NiFi version in the NiFi Administrator's Guide and the NiFi migration guidance before scheduling any production change, and rehearse the full sequence on a copy of production data.

What drives the effort and cost of the coordination switch

This table lists the factors that change the effort and cost of moving NiFi coordination from ZooKeeper to Kubernetes Leases, with whether each driver is one-time or recurring and what to request in a written quote. Read it before scoping, because the first two rows often decide how much design work the switch needs.

DriverWhy it changes effort or costOne-time or recurringWhat to request in the quote
Number of clusters in one namespaceEach NiFi cluster needs a unique lease prefix, and each prefix needs its own RBAC scope and naming review. More clusters mean more configuration and more test cases.One-time for setup, recurring when new clusters are addedCount of clusters, planned prefixes, and whether the scope covers future clusters
Kubernetes platform maturity and versionA cluster on a supported minor version with GitOps delivery and monitoring is faster to change than one with manual deployments. Version skew rules and patch windows affect the test plan.One-time, with ongoing patch workCurrent Kubernetes version, upgrade cadence, and who owns the control plane
RBAC and security reviewThe NiFi service account needs permission to manage Leases and ConfigMaps. Reviewers will check that these permissions are limited to the NiFi namespace and resource names.One-time, plus review at each permission changeRole and binding definitions, review sign-off steps, and the evidence your reviewers expect
TLS and secrets handlingCertificates and keystore material move into Kubernetes Secrets. Secrets are stored unencrypted in etcd by default, so encryption at rest and access limits must be confirmed for the switch.One-time, with rotation work every certificate cycleSecret store design, encryption at rest status, and rotation procedure
State migration complexityMoving cluster state from the ZooKeeper state provider to the ConfigMap state provider requires a planned cutover and a rollback path. Flows with custom state or many components take longer to verify.One-timeInventory of components that store state, cutover steps, and rollback steps
Size of queued dataLarge queues on existing nodes raise the risk of a disruptive rolling change. Draining or moving repositories adds hours of planning and monitoring.One-timeCurrent queue depth per node, drain strategy, and the maintenance window assumed
Test environment availabilityA staging cluster that matches production topology lets the team rehearse leader election, failover and state migration. Without one, testing happens in production windows.One-time, plus a standing environment if keptWhether staging is included, its topology, and who provisions and tears it down

What usually goes wrong when NiFi moves off ZooKeeper

Most failed cutovers from ZooKeeper to Kubernetes Leases come from configuration that looks correct in isolation but collides with something else in the cluster. The cards below cover the failures we see most often, from lease naming and RBAC to Kubernetes versions, secrets exposure and queued data, each with a prevention step to settle before the cutover window.

Duplicate lease prefixes across clusters

When two NiFi clusters share a Kubernetes namespace and leave nifi.cluster.leader.election.kubernetes.lease.prefix unset or identical, both clusters compete for the same Lease objects. Nodes from one cluster can then appear to hold primary status for another, and coordination becomes unpredictable. Before cutover, list every NiFi cluster in each namespace and assign each one a unique prefix. Check the prefix in every node's configuration, not only the first cluster you set up.

Incomplete RBAC blocks leader election

NiFi nodes create and update Lease objects and read or write ConfigMaps, and the service account running them needs those permissions in the target namespace. If a Role or RoleBinding omits a verb or resource, election can fail quietly, leaving nodes unable to agree on a primary. Test the service account with a permissions check for each required verb before starting nodes. Keep the Role scoped to the namespace and to the Lease and ConfigMap resources NiFi uses.

Two primary nodes during cutover

During a move from ZooKeeper to Leases, some nodes may still run the old election code while others use the new one. Mixed configurations can produce two nodes that each believe they hold the primary role, which matters for processors scheduled on the primary node only. Plan the cutover as a full cluster restart with a single configuration change, and keep the old ZooKeeper ensemble reachable until every node has joined. Confirm exactly one primary in the UI and logs before resuming flows.

Kubernetes version and distribution gaps

Lease behavior depends on the Kubernetes API your distribution exposes, and managed or customized distributions may differ in default admission rules, API server flags or upgrade timing. An upgrade of the control plane can also move the cluster outside the version range your NiFi release was tested against. Record the distribution, Kubernetes minor version and planned upgrade dates in the design. Confirm with your distribution vendor that the Lease and ConfigMap APIs behave as your NiFi version expects.

etcd and Secrets exposure

Kubernetes stores Secrets unencrypted in etcd by default, and NiFi deployments often place keystore passwords, sensitive property values and the flow encryption key there. Coordination objects and ConfigMaps can also reveal cluster topology to anyone with broad read access. Enable encryption at rest for etcd, restrict list and watch access on Secrets with least-privilege RBAC, and review who can read ConfigMaps in the NiFi namespace. Confirm your etcd backup copies are encrypted as well.

Unplanned loss of queued data

Moving a cluster to new pods, volumes or nodes can strand FlowFiles in content and provenance repositories if the old nodes are removed before their queues drain. Lease coordination does not move data, so a node restarted with empty or unmounted repositories starts without the queued work it held. Before cutover, drain queues where the flow allows it, confirm each node's repository volume is attached to the new pod, and record queue counts before and after. Ask the team to rehearse a restore from a repository backup in a test namespace.

Questions buyers ask before moving NiFi coordination to Kubernetes

What does a move from ZooKeeper to Kubernetes Leases cost, and what drives it?

The cost is mostly the labor to change coordination in a running cluster, test it, and prove the cluster behaves the same afterward. Work is quoted from a written scope, so the main drivers are the number of NiFi clusters, how many flows and parameter contexts need review, whether state is migrated or rebuilt, how much of the Kubernetes setup (service accounts, RBAC, namespaces) is already in place, and how many test cycles you require. Ask for the quote to list one-time items (assessment, configuration change, cutover, validation) separately from recurring items such as retainers for upgrades or ongoing support. DeployOpen does not publish prices, so the quote follows from the scope you agree in writing.

How long does the change take, and what makes it longer?

Duration depends on the drivers that set the amount of verification work: how many clusters share a namespace, how many nodes each cluster runs, how large the repositories and queues are, how often flows change, and how many environments (development, test, production) must be exercised before cutover. A single small cluster with a clean test environment is the simplest case. Multiple clusters in one namespace, custom processors that depend on coordination behavior, or a required maintenance window add time. The timeline is agreed in the written scope, so ask for a plan that names each phase and the test evidence it produces.

What must our team provide before the work starts?

You need to provide access and decisions that only your organization can give. The list usually includes:

  • Access to the NiFi clusters, their configuration files and a non-production copy to test against
  • A named owner for the Kubernetes cluster, including who approves namespace, RBAC and service account changes
  • Flow owners who can confirm which flows depend on primary-node-only behavior and which can pause during cutover
  • Change approval and a maintenance window for each environment
  • Decisions on the lease prefix naming scheme if several clusters share a namespace

Missing any of these typically moves the schedule rather than the technical work.

Can our internal team make this change without outside help?

Yes, if your team already runs NiFi and Kubernetes in production and can test in a non-production environment that mirrors the cluster. The work is configuration-driven, but the risk sits in the details: the ZooKeeper to Lease change, RBAC permissions the NiFi pods need to read and write Leases and ConfigMaps, and the state migration. Teams without recent NiFi cluster operations experience often find the validation step, not the configuration, takes the most effort. Compare your team's capacity and experience against the written scope before deciding.

Can ZooKeeper stay in place while we test the Kubernetes setup?

Yes, in a non-production environment you can run a test cluster on Kubernetes Leases while production continues to use ZooKeeper, because the two environments are separate clusters. Keep them apart. Do not point a production NiFi node at a mixed configuration, and do not let two coordination mechanisms manage the same cluster at once. Confirm with NiFi's administration guide how your NiFi version handles the leader election implementation setting before you plan a parallel run.

What happens to data already queued in NiFi during the switch?

Queued FlowFiles live in the content and FlowFile repositories on each node, not in ZooKeeper or Kubernetes Leases, so changing the coordination mechanism does not by itself delete them. The risk is in the cutover: restarting nodes, moving repositories to new persistent volumes, or changing node identity can strand or duplicate work. Plan a drain or a controlled stop, confirm queue counts before and after, and validate with provenance that no data was lost or processed twice. Ask for the cutover runbook to state how queued data is handled for each cluster.

What does the state migration involve, and how do we verify it?

State held by NiFi components, including cluster state, has to be moved or rebuilt when the coordination backend changes. NiFi provides a Kubernetes ConfigMap cluster state provider alongside the ZooKeeper state provider, so the choice of provider and the migration path should be decided before cutover. Verification means confirming that the primary node is elected as expected, that each node reports into the cluster, and that stateful processors resume from their last known position. Record these checks as acceptance criteria in the scope.

How DeployOpen scopes and delivers a NiFi coordination migration

DeployOpen approaches the move from ZooKeeper to Kubernetes Leases as a written, fixed-scope engagement covering the NiFi cluster, its Kubernetes namespace and its state. Each step is agreed before work starts, tested in an environment you control, and handed over with documentation your team can operate.

How we scope

  • Inventory the current ZooKeeper dependencies, state providers and cluster count before agreeing the change.
  • Confirm the Kubernetes version, namespace layout and service account permissions your platform team allows.
  • Agree in writing which clusters, lease prefixes, state migration steps and validation tests are in scope.

How we build

  • Configure the Kubernetes leader election implementation and unique lease prefixes in a non-production cluster first.
  • Create the RBAC role and binding for Lease and ConfigMap access with the least permissions the cluster needs.
  • Rehearse state migration and failover tests, then record results so each step can be repeated during cutover.

How we hand over

  • Deliver configuration files, RBAC manifests and a runbook covering election, failover and state recovery.
  • Walk your platform and NiFi administrators through the cutover steps and the rollback plan.
  • Document the open items, such as settings to confirm with your Kubernetes distribution, before the engagement closes.

How we support

  • Engagements are fixed-scope, and the written scope defines what is included after handover.
  • Ongoing support, NiFi and Kubernetes upgrade planning and repeat testing are available as a retainer.
  • Ask for a quote based on the written scope, which sets the cost of each phase and any recurring support.

Sources and further reading

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.