Why securing ZooKeeper matters for a production NiFi cluster
A clustered Apache NiFi depends on a coordination layer to decide which node is the cluster coordinator and primary node, to share cluster membership and to keep state that every node must agree on. In a ZooKeeper-based cluster, that layer is the ZooKeeper ensemble. Whoever can read or write its znodes can influence which nodes NiFi believes are in the cluster, who holds leadership and what state the nodes load on startup. This page covers how to lock that ensemble down: access control lists, TLS between NiFi and ZooKeeper, the choice between embedded and external ZooKeeper, and when Kubernetes ConfigMaps or Leases remove the need for ZooKeeper altogether. An unsecured ensemble is a direct path to cluster state, so it should be treated as part of the NiFi trust boundary, not as internal plumbing.
The NiFi administration guide describes three ways a cluster can coordinate. It can use ZooKeeper, either the embedded instance that NiFi can start on its own nodes or an external ensemble that your platform team operates. On Kubernetes it can instead use ConfigMaps or Leases for state and leader election. Each option moves the security question to a different place. With an embedded or external ensemble, you own the ZooKeeper hosts, their network exposure, their authentication and their patching. With Kubernetes objects, the question shifts to role-based access control on the namespace and who can edit those ConfigMaps or Lease objects. Confirm which mode your version and deployment support before you design controls, because the supported configuration keys and defaults differ between releases.
Teams usually need this work when a NiFi cluster moves from a lab or proof of concept into production, when a shared ZooKeeper already serves other systems, or when a security review asks where cluster state lives and who can change it. Ask your NiFi and platform owners to confirm three things before the design starts: which coordination mode the cluster uses, which hosts or namespaces hold the state, and which people or service accounts can reach them. Those answers decide how much of the work below applies to your environment and whether a shared ensemble needs its own access boundary.
Embedded, external or Kubernetes Leases: which coordination model fits
Embedded ZooKeeper runs inside the NiFi nodes themselves, so no separate ensemble has to be provisioned. The NiFi team operates it as part of the NiFi installation, and its security boundary is the same as the NiFi nodes it runs on. Anyone who can reach its client and peer ports on a node can attempt to interact with coordination state, so those ports must be restricted to the other cluster nodes and approved hosts. This model suits small clusters, lab or pre-production environments and teams that want the fewest moving parts. The operational burden is lower at the start, but coordination shares resources and failure domains with NiFi, which makes capacity planning and node maintenance more sensitive to load.
- Operator: the NiFi platform team, with no separate ZooKeeper administration role
- Security boundary: the NiFi node hosts, so host hardening and firewall rules around the client and peer ports carry the weight
- Operational burden: low to set up, but upgrades and node restarts affect coordination directly
- Fit: small or non-critical clusters where a separate ensemble is not justified
- Limits: coordination is tied to NiFi node availability, so review the administration guide before relying on it for production
Before you choose this model, confirm the embedded configuration, port settings and any authentication options for your NiFi version in the administration guide.
An external ZooKeeper ensemble runs on its own hosts, separate from the NiFi nodes. This separates coordination from NiFi processing and lets the security team set access control lists, TLS and network policy on a service with a narrow purpose. The security boundary becomes two systems: the ensemble and the connection from each NiFi node to it. This model suits production clusters that already run ZooKeeper for other services, or organizations that want coordination to survive individual NiFi node failures and maintenance windows. The cost is ongoing work: ensemble sizing, quorum management, patching, backups of coordination data and monitoring of the ensemble itself.
- Operator: a platform or middleware team that administers the ensemble separately from NiFi
- Security boundary: ZooKeeper ACLs, TLS on client connections and firewall rules limiting access to NiFi nodes
- Operational burden: recurring work for quorum health, version updates and certificate rotation
- Fit: production clusters, multi-team platforms and environments with an existing ZooKeeper service
- Ownership: agree in writing which team patches the ensemble and which team responds to coordination alerts
Confirm the supported ZooKeeper versions, ACL and TLS settings, and client configuration for your NiFi release in the administration guide.
When NiFi runs on Kubernetes, the administration guide describes using ConfigMaps or Leases for cluster state and leader election, so a separate ZooKeeper ensemble may not be needed. The security boundary moves to the Kubernetes API: RBAC roles, service accounts and namespace isolation decide which workloads can read or write the coordination objects. Leases remove the need for a separate ZooKeeper only in this Kubernetes setting and only where the guide supports the mode you deploy. Your platform team operates the coordination objects through the cluster, which reduces the number of systems to patch but ties NiFi availability more closely to the Kubernetes control plane.
- Operator: the Kubernetes platform team, with NiFi configuration owned by the NiFi team
- Security boundary: RBAC on ConfigMaps and Leases, service account scope and namespace isolation
- Operational burden: lower for ZooKeeper, but dependent on control-plane health and API access policy
- Fit: NiFi already deployed on Kubernetes with a platform team that manages cluster RBAC
- Check: that your Kubernetes version and NiFi release combination is documented, and that no other ZooKeeper dependency remains in your design
Confirm the exact Kubernetes configuration, required permissions and supported topology in the administration guide before removing ZooKeeper from the plan.
ZooKeeper security options and when each fits
This table compares the main ZooKeeper hardening controls for a NiFi cluster, from znode access control to network isolation and the Kubernetes Lease alternative. Read each row as a decision: what the control protects, what it costs in operational effort, and the deployment shape where it makes sense. The main takeaway is that embedded versus external ZooKeeper and Lease-based coordination change which controls you own at all.
| Option | Trade-off | When it fits |
|---|---|---|
| ACL enforcement on znodes | Restricts which identities can read or change the coordination state NiFi stores in ZooKeeper. Misconfigured ACLs can block nodes from joining the cluster, so changes need a test cluster first. | External ZooKeeper shared with other services, or any environment where an assessor expects access to coordination data to be limited to NiFi. |
| TLS client connections from NiFi to ZooKeeper | Encrypts coordination traffic in transit. Requires certificate issuance, distribution to every node and a rotation process that does not interrupt the cluster. | Any ensemble that crosses a network segment, a shared data center network or a cloud VPC boundary. Confirm the exact property names against the current NiFi administration guide. |
| Mutual authentication (client certificates) | Lets ZooKeeper verify NiFi nodes by certificate, not only by network location. Adds a certificate authority to operate and more failure modes during renewal. | Regulated environments that require each service identity to be proven, or ensembles reachable from more than one team's network. |
| Ensemble network isolation | Limits which hosts can reach the client and peer ports. Relies on firewalls, security groups or network policies that someone must keep accurate as hosts change. | Most production ensembles. Usually the first control to apply because it is low cost to configure and reduces exposure even when other controls are deferred. |
| Embedded ZooKeeper | Runs ZooKeeper inside the NiFi nodes, so there is no separate ensemble to harden. Ties coordination availability to NiFi node availability and gives you less independent control over ZooKeeper settings. | Small or non-critical clusters where a separate ensemble adds more operational risk than it removes. Confirm the supported node count and failure tolerance for your design. |
| External ZooKeeper ensemble | Gives you a separately hardened, separately patched service with its own access controls. Adds a second system to back up, monitor and upgrade. | Production clusters where security teams want coordination state isolated from NiFi nodes and where a platform team already runs ZooKeeper. |
| Kubernetes Leases instead of ZooKeeper | Uses Kubernetes API objects for state and leader election, which removes the ZooKeeper ensemble from the design. Shifts the security question to Kubernetes RBAC and API server access. | NiFi deployed on Kubernetes where the cluster already has strong RBAC, audited API access and a supported NiFi configuration for Leases. Confirm current support in the NiFi administration guide before choosing it. |
How to secure the ZooKeeper ensemble step by step
This sequence moves a NiFi cluster's coordination layer from default access to a deliberate design: a chosen coordination model, a restricted network path, encrypted connections, and znode permissions that only the NiFi principal can write. Each step ends with a check that shows whether it worked.
Decide the coordination model
The platform or architecture owner chooses between embedded ZooKeeper, an external ensemble, or Kubernetes using ConfigMaps or Leases for state and leader election, as described in the NiFi administration guide. Record the choice, the ensemble members, who operates them, and who owns the ZooKeeper ACL credentials. The step is complete when the decision is written down and every later step refers to the same model. If Kubernetes Leases are chosen, ZooKeeper steps 2 to 5 do not apply, but the Kubernetes RBAC and network controls must be reviewed instead. Confirm which coordination options your NiFi version supports before you commit.
Isolate the ensemble network
The network team restricts access so that only NiFi nodes and other ensemble members can reach the ZooKeeper client and quorum ports. Administrative hosts and general user subnets should have no route to these ports. Use firewall rules or security groups and record them as configuration. The check is a connection test from a host outside the allowed list, which should fail, and a test from each NiFi node, which should succeed. Confirm the exact ports your ensemble version uses against the ZooKeeper documentation, not against defaults you remember.
Enable TLS on client and quorum paths
The team issues server certificates for each ensemble member with subject alternative names that match the hostnames NiFi will use to connect. TLS is enabled for client connections and, where your ZooKeeper version supports it, for quorum traffic between members. Keystores and truststores are stored with restricted file permissions. The check is an
openssl s_clienthandshake against each member that shows the expected certificate chain, plus a confirmation that a plaintext client connection is refused. Confirm the TLS configuration keys and certificate requirements for your exact ZooKeeper release in its official documentation.Configure NiFi's connect string and TLS settings
The administrator updates
nifi.zookeeper.connect.stringandnifi.zookeeper.root.nodeinnifi.propertieson every node, then sets the secure client options and the keystore and truststore paths that match the ensemble's certificates. The values must be identical across nodes, and the NiFi principal is the one that will hold the ACL rights in step 5. The check is a review of the rendered file on each node, followed by the NiFi log entries showing a connection to the ensemble without certificate or handshake errors. Confirm each property name and its accepted values in the administration guide for your NiFi version before editing.Apply znode ACLs for the NiFi principal
The ZooKeeper administrator sets an ACL on the NiFi root node so the NiFi authenticated identity has full rights to create, read, write and delete its paths, and other identities have no write access. The ACL is applied to any existing NiFi znodes and to the parent path so that new children inherit the restriction. The check has two parts: a ZooKeeper client connected as an unauthenticated or other identity should be denied when it tries to write under the NiFi path, and the NiFi principal should still be able to read and write. Confirm the auth scheme and ACL syntax against the ZooKeeper security documentation for your version, because a wrong scheme locks NiFi out.
Restart and verify cluster formation
The administrator restarts ensemble members one at a time so that quorum is kept throughout, then restarts NiFi nodes in a rolling sequence. Each restarted node should rejoin the cluster and the cluster should elect a coordinator and primary node. Check the NiFi cluster view in the UI or the cluster API, confirm every node shows as connected, and search the logs for ZooKeeper authentication, ACL or TLS errors. The step is complete when all nodes are connected and the logs are clean. If a node fails to join, roll back the most recent change before continuing. Confirm the restart order and any cluster-specific steps in the administration guide for your release.
Test failover and access denial
The operations team runs controlled failure tests in a non-production cluster first. They stop one ensemble member and confirm the ensemble keeps quorum and NiFi stays connected. They then stop the NiFi coordinator and check that a new coordinator is elected and that flows keep running. They repeat the unauthorized write test from step 5 to confirm the ACLs still hold after restarts. The check is a written test record with timestamps, the observed behavior, and any flow or cluster state that needed manual recovery. Losing quorum on the ensemble will stop coordination, so document the recovery path and confirm the expected behavior with your ZooKeeper and NiFi documentation before running the test in production.
Security and operational risks to control around ZooKeeper
ZooKeeper holds the coordination state that a NiFi cluster depends on for leader election and cluster membership. The risks below share one root cause: anyone who can reach or alter the ensemble can change how the cluster behaves, so each card pairs a risk with the control that closes it.
Unauthenticated or world-readable znodes
Znodes hold cluster coordination data that NiFi nodes read and write. If ACLs are missing or left open, any client that reaches the ensemble may read or modify that state. Check that NiFi connects with credentials and that ZooKeeper ACLs restrict znode access to the NiFi principal, then confirm with a test client that unauthenticated reads are refused.
Unencrypted coordination traffic
Traffic between NiFi nodes and ZooKeeper carries cluster metadata and leader election messages in clear text when TLS is not enabled. An attacker on the network path could observe or interfere with that exchange. Enable TLS on the client connection, use certificates issued for the ZooKeeper hostnames, and verify that no plaintext client port is open to NiFi.
Ensemble exposed beyond the NiFi network
ZooKeeper's client and peer ports should be reachable only from NiFi nodes and, for the ensemble itself, from other ZooKeeper servers. A broad firewall rule or a public load balancer widens the attack surface considerably. Review network policies and security groups, limit the client port to NiFi subnets, and confirm the peer and leader election ports are reachable only inside the ensemble.
Shared ZooKeeper with other applications
Running NiFi on an ensemble also used by other applications means one team's change, load spike or credential leak can affect NiFi's cluster state. Shared use also complicates ACL design, since other clients need their own paths and permissions. Decide whether NiFi gets a dedicated ensemble, and if sharing is unavoidable, isolate NiFi under its own root path with restrictive ACLs.
Quorum loss and recovery
ZooKeeper needs a majority of its servers to stay available. If too many members fail at once, NiFi loses leader election and coordination, and the cluster can stall. Plan the ensemble size with failure domains in mind, monitor member health, and rehearse restoring a quorum from a known-good state so recovery does not depend on improvised steps during an incident.
ZooKeeper versions and credentials under change control
Upgrades to ZooKeeper, rotations of TLS certificates and changes to ACLs or passwords all alter how NiFi connects. Without a change record, a rotation can break cluster coordination or leave an old credential active. Track versions, certificate expiry and credential owners in your change process, test each change in a non-production cluster first, and keep evidence of who approved it.
Questions teams ask before securing ZooKeeper for NiFi
Is embedded ZooKeeper acceptable for a production NiFi cluster?
Embedded ZooKeeper can run in production, but only when the cluster is small, the NiFi nodes are the only clients, and your team accepts that ZooKeeper shares the lifecycle, JVM and failure domain of NiFi. Apache NiFi supports both embedded and external ZooKeeper, so the choice is about operational boundaries rather than a product limitation. External ZooKeeper suits teams that want independent upgrades, separate access control, monitoring and backups for coordination state. Whichever you pick, confirm the ensemble size, the quorum behavior during node loss and the procedure for restoring state. The NiFi administration guide describes the supported options and is the right starting reference for your architecture review.
Do Kubernetes Leases remove the need for ZooKeeper?
Yes, for clusters running on Kubernetes, NiFi can use ConfigMaps or Leases for cluster state and leader election instead of ZooKeeper, according to the NiFi administration guide. That removes one stateful service from the design, along with its ACLs, TLS material and quorum monitoring. It does not remove the need for Kubernetes RBAC, which becomes the control that limits who can read or change coordination objects. Confirm that your NiFi version and deployment method support this mode, then review the Role and RoleBinding objects that grant access to the namespace. If you run NiFi outside Kubernetes, ZooKeeper remains the coordination path.
What does the work need from our internal team?
Your team needs to provide access to the NiFi and ZooKeeper hosts or namespaces, the certificate authority or PKI process that issues server and client certificates, and a named owner for change approval. You also need someone who can confirm the network paths between NiFi nodes and ZooKeeper servers, and who understands how the cluster is monitored today. Plan for reviewers who can test the ACL model against real NiFi behavior, since mistakes usually appear during a restart or a leader election. Ask the implementation partner to list these dependencies in the written scope so the inputs are agreed before work starts.
How long does securing ZooKeeper take, and what drives the effort?
The duration depends on the topology and the number of environments involved, and no fixed figure applies to every cluster. The main drivers are whether you run embedded or external ZooKeeper, how many ensembles and NiFi clusters need changes, how mature your PKI and secrets handling are, and how many environments (development, test, production) must be validated. Effort also rises when you must migrate existing unsecured znodes or coordinate a maintenance window for a live cluster. Ask for a timeline that lists each environment, the change windows and the validation steps, rather than a single number.
Can our own team secure ZooKeeper without a partner?
Yes, a team with ZooKeeper, NiFi and PKI experience can complete this work internally, provided it follows the vendor documentation and tests each step in a non-production cluster first. The risk lies in incomplete ACLs, mismatched certificates or a restart order that breaks quorum, and these problems often surface only after a rolling restart. Teams with limited ZooKeeper experience usually benefit from an outside review of the design and test plan. Decide based on the skills you have in-house and the change risk you can accept on a live coordination service.
What does a fixed-scope quote cover for this work?
A fixed-scope quote follows from a written scope that names the ZooKeeper topology in each environment, the NiFi clusters it serves, the security controls to implement and the deliverables. Typical items include the ACL design, TLS configuration for NiFi to ZooKeeper connections, certificate and key handling, a test and rollback plan, runbook updates and knowledge transfer to your operators. Ask the quote to state what is one-time (design, configuration, validation) and what recurs (monitoring, certificate renewal, upgrades), and which items depend on your environment. DeployOpen does not publish a price list, so the quote will reflect the written scope.
How do we validate that ZooKeeper security works?
Validation should prove both that unauthorized clients are refused and that NiFi still starts, elects a coordinator and replicates state after a controlled restart. Test anonymous and wrong-credential connections to the ensemble, confirm that each NiFi node authenticates with its expected identity, and check the ACLs on the znodes NiFi uses. Then perform a rolling restart of ZooKeeper and NiFi nodes, followed by a node failure test that confirms quorum behavior. Record the results as evidence for your change record. Confirm the exact client properties and ACL syntax for your NiFi and ZooKeeper versions against the NiFi administration guide before you write the test cases.
How ACLs and TLS protect NiFi coordination state
ZooKeeper access control lists limit who can read or change the znodes NiFi uses for coordination, such as cluster membership, leader election and component state. Each znode carries an ACL that lists the identities and permissions allowed on it. Without an ACL, any client that can reach the ensemble port can read cluster state or write to a path NiFi trusts, which can disrupt elections or corrupt the state a node relies on after a restart. Confirm in the NiFi administration guide which ZooKeeper settings control the root path and ACL behavior for your version, and check that the ZooKeeper server is configured to require authenticated clients rather than accepting anonymous connections. Then verify with a ZooKeeper client from a host outside the NiFi nodes that the coordination paths cannot be read or modified.
TLS protects the traffic between NiFi nodes and the ZooKeeper ensemble so that coordination data cannot be read or altered in transit. ACLs decide who may act, but they do not encrypt the session, so a credential or state update sent in clear text can be captured on the network path between the two tiers. Enabling TLS on the client connection requires a certificate on each ZooKeeper server, a trust store on the NiFi side, and matching hostnames in the certificates. Treat the certificate and key lifecycle as an operational task: decide who issues the certificates, how renewal is scheduled, and how a rotation is rehearsed on a non-production cluster before it reaches production.
Both controls are needed because each one covers a gap the other leaves open. An ACL without TLS still exposes authenticated traffic to anyone who can observe the network, while TLS without an ACL still lets any authenticated client change the state. Plan the two together, record the exact settings used on each NiFi node and ZooKeeper server, and confirm the values against the current administration guide before go-live, since property names and defaults change between releases.
How DeployOpen works on this
DeployOpen approaches ZooKeeper hardening for an Apache NiFi cluster as a scoped piece of work inside your own infrastructure, starting from how your cluster coordinates today. We agree the written scope before any change is made, then build, test and document the ensemble access controls and TLS settings, and hand over the result with the evidence your team needs to run it.
How we scope
- Inventory the current NiFi cluster, embedded or external ZooKeeper topology, and the hosts that reach it.
- Confirm whether your platform should keep ZooKeeper or use Kubernetes Leases and ConfigMaps for state and leader election.
- Agree in writing which ACL, TLS and network controls are in scope, and what is excluded.
How we build
- Configure ZooKeeper ACLs and TLS between NiFi nodes and the ensemble in a lab copy first.
- Apply changes to your environment in a sequence that keeps the cluster coordinating throughout.
- Test node restarts, leader election and access denial for unauthorized clients before sign-off.
How we hand over
- Deliver configuration files, certificate locations and ACL settings with a record of each change.
- Document the ensemble operations runbook, including rotation steps and how to verify the controls.
- Walk your operations team through checks they can repeat after the engagement ends.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after handover.
- Review ZooKeeper and NiFi configuration when either platform is upgraded or the cluster topology changes.
- Repeat the verification tests on request so your evidence stays current with the running system.
