Apache NiFi on Kubernetes

Run four checks, each tied to one way a NiFi node can leave the cluster. Start by identifying which pod holds the Primary Node and Cluster Coordinator roles. The NiFi cluster view shows node status and the roles assigned to each node, and nifi-app.log records role changes. Before you delete anything, record the pod names, the node status and the queued flowfile counts on a test process group. Then delete the pod holding the Primary Node role with kubectl delete pod and watch whether another node takes over both roles. The expected result is that leadership moves within the cluster, the flow keeps moving data through the nodes still connected, and the deleted pod returns to the cluster. Write down how long the handover took so you can compare results across reruns.

Next, delete a node that holds no leadership role. A StatefulSet recreates the pod under the same ordinal name and reattaches the PersistentVolumeClaim with the same name, so the question is whether the new pod picks up the old repositories. Before deletion, note the PVC name, the volume it is bound to and the contents of the flowfile, content and provenance repository directories. After the pod returns, confirm that it reconnects to the cluster, that the claim is still bound to the same volume and that queued flowfiles are processed again. An empty repository after reattachment is a failure, even if the pod shows Ready. Also check the destination for duplicate deliveries of the test data.

The third check restarts the whole StatefulSet, which is the closest test to a planned maintenance window or a node pool replacement. Confirm that the flow.json.gz file and the flow state survive the restart, and that the cluster returns to a state where all nodes are connected and a single Primary Node is elected. Query provenance for events recorded before the restart to confirm the provenance repository still answers queries. Then isolate one pod from the others, for example with a Kubernetes NetworkPolicy that blocks cluster traffic to it, and observe whether the node disconnects and stops processing as part of the cluster. Remove the isolation and confirm the node rejoins. Check the exact disconnection and reconnection behavior in the documentation for your NiFi version before you set expectations for this step.

A failed test shows up in a few places. Nodes that stay in a disconnected or connecting state after the pod reports Ready, leadership that does not move, queues that stop draining, or repositories that appear empty after reattachment are all failures. Check the NiFi cluster view and node heartbeats, nifi-app.log and nifi-user.log on the affected pod and on the surviving pods, and startup output for configuration errors. On the Kubernetes side, run kubectl describe on the pod to read its events, check the PVC binding, and inspect the Lease or ConfigMap your deployment uses for leader election, since its holder should change during a handover. Save logs from before and after each deletion so the timeline can be reconstructed.

This test checks recovery. It is not a load test, and passing it does not show throughput, latency under pressure or how the cluster behaves at the data volumes you expect in production. Those need a separate performance test with your own flows and data. Run the sequence on a non-production cluster that uses the same NiFi version, storage class and resource settings as production, and repeat it after each upgrade or change to the StatefulSet. Ask any implementation partner to list these four checks, the evidence each one produces and the acceptance criteria in the written scope before work starts.

After go-live, the platform team should watch four signals that show whether the NiFi cluster is healthy before flows start failing. The first is pod restarts and volume health. Repeated restarts of a NiFi pod often point to memory pressure, a failing readiness check or a problem with the node it runs on. A PersistentVolumeClaim that stays pending, or a volume that reports I/O errors, puts repository data at risk, so alert on both. The second is cluster membership. Confirm that every expected node is connected and that exactly one node holds the coordinator role, using the cluster view in the UI or the NiFi REST API. Alert when a node drops out and does not rejoin within the window your team agrees. The third is repository disk usage. The content, FlowFile and provenance repositories grow with traffic and with the retention settings you chose, so alert early enough to add capacity or adjust retention before a volume fills. The fourth is ingress certificate expiry. Track the expiry dates of the ingress certificate and the NiFi node certificates, and agree who renews them and how far ahead.

Back up what the container image cannot rebuild. The first item is the flow definition. In NiFi 2 this is the flow.json.gz file on the persistent volume. If you also version flows in Git through a Flow Registry Client, the repository is a second copy, but it does not capture every running setting, so keep both. The second item is nifi.properties and the key material held outside the image, delivered through Kubernetes Secrets. This includes the sensitive properties key, the TLS keystores and truststores, and the login identity and authorizer configuration. Store the sensitive properties key separately from the flow backup, with access limited to the people who may restore the cluster. The third item is the state needed to rebuild: the Lease or ConfigMap objects used for leader election and cluster state, parameter context values, and the list of controller services and their settings. For each repository, decide with the business owner whether its contents must be restored or whether losing them is acceptable, because provenance and content data can be very large.

A backup that has never been restored is an assumption. Rehearse a restore at least once before go-live and again after any major change to the cluster. Run the rehearsal in a separate namespace with its own volumes. Restore the manifests from version control, restore the Secrets, restore the flow definition onto a fresh volume, then start one node before scaling to the full node count. Check that the flows load, that controller services start, and that sensitive parameters decrypt. Without the original sensitive properties key, encrypted values in the flow cannot be read, so this check is the most important one. Record which steps needed manual work, which items were missing from the backup and how long each step took in your environment. Treat each gap as a defect to fix before the next rehearsal. Ask any partner you engage to confirm in writing which restore steps they will perform and which ones your team must perform.

The handover should leave the next operator with three documents. The first is the manifests: the Helm values or Kustomize overlays, the pinned container image tag, resource requests and limits, storage classes, and the ingress definition, all kept in version control with a record of who approved each change. The second is the upgrade runbook for the Kubernetes layer. It should describe how to perform rolling cluster changes such as node pool upgrades or pod disruption events: drain one node at a time, wait for the NiFi node to rejoin the cluster and for its queues to clear, then continue. The runbook covers only the Kubernetes layer. Changes to the NiFi version, including the steps for moving between releases, belong to the upgrade and support topic and should be planned separately. The third document is the list of integrations. For each external system NiFi connects to, record the endpoint, where its credentials are stored, the team that owns it, the certificate expiry date, and which flows depend on it. This list drives both the monitoring checks and the restore rehearsal.