Apache NiFi backup and recovery

Which NiFi artifacts to back up and which to rebuild

A NiFi node holds a mix of durable definitions, secrets, runtime state and transient data. Some of it must be restored from a backup, some can be regenerated from code or the registry, and some should not be restored at all. Decide each artifact's category before you pick backup tooling, because the answer changes what you store, where, and who can read it.

How backup scope changes between standalone, clustered and Kubernetes NiFi

How to restore a NiFi node or cluster without duplicating in-flight data

This sequence brings a lost NiFi node or cluster back in a known state, with the configuration, keys and flow definition matching what ran before the loss. It also settles what happens to FlowFiles that were queued at the time of failure, so upstream data is not processed twice.

1. Confirm the backup set and NiFi version

   Before touching the new host, confirm that the backup set came from the same NiFi version you are about to run, and that it contains the flow configuration, the configuration files, the sensitive properties key and the state directory. Check the flow file name too: NiFi 2 uses flow.json.gz, while older 1.x backups may hold flow.xml.gz, and the migration path from 1.x to 2.x is a separate change that should not be mixed into a restore. Confirm the exact version with the owner of the backup job and the release notes for that version. The step is done when someone other than the person who took the backup has signed off on the version and file list.

2. Restore configuration and keys before first start

   Place the configuration files, including nifi.properties, the bootstrap configuration, TLS keystores and truststores, and the sensitive properties key, on the new node before NiFi starts for the first time. The sensitive properties key is the most important item in this step: without it, encrypted passwords and sensitive parameter values in the flow cannot be decrypted, and the flow will load with values that fail at runtime. Restore file ownership and permissions so the NiFi service account can read the keys but other users cannot. The step is done when the files exist at their configured paths and a checksum comparison against the backup manifest matches.

3. Restore the flow definition or pull it from Git

   Put back the flow definition from the backup, or, if your flows are version controlled, retrieve the exact committed revision from the Git repository used by the Flow Registry Client. Record the commit identifier and the flow version so the restored state can be traced later. Where a flow is imported from Git, check that the registry client settings and any stored credentials point to the repository and branch you expect. The step is done when the flow on disk matches the recorded revision, and the commit or checksum is written into your recovery record.

4. Start the node in isolation before rejoining

   Start the restored node with clustering disabled or with it kept away from the cluster coordination service, so it cannot take part in live flow traffic yet. Watch the startup logs for errors about missing keys, unresolved parameter contexts, controller services that fail to enable, and processors that cannot find their referenced resources. Keep source connections that feed this node paused or pointed elsewhere while it is isolated. The step is done when the node reports a clean startup, and every controller service and processor in the restored flow has been reviewed against the expected state.

5. Verify the restored flow against the cluster

   Compare the restored flow with the state the cluster had before the loss. Check the component count, connection layout, controller service states, parameter context values (without copying secret values into tickets), and the version of the flow. If the node will rejoin a cluster, the flow it presents must match the cluster's flow, because a mismatch blocks the node from joining. Review the provenance and queue state that the cluster reports for the connections this node owned. The step is done when the diff is empty or every difference is documented and accepted by the flow owner.

6. Decide what to do with queued FlowFiles

   FlowFiles that were sitting in queues or in the content repository when the node was lost cannot be recovered from a flow definition backup. Your options are to accept that those records were lost and re-ingest them from source, to rely on the upstream system to resend from its own checkpoint, or to read the surviving content repository if it exists on other storage. Each option has a different duplication risk. Make the decision per flow, record it, and assign an owner for any replay. Do not replay everything from the beginning of a source without first checking what downstream systems already received. The step is done when every affected flow has a written decision.

7. Replay sources with idempotency keys or checkpoints

   Replaying data from upstream systems can duplicate records when the destination has already accepted some of them before the loss. A replay that restarts from an earlier offset will write the same events again unless the destination can detect repeats. Reduce that risk with source checkpoints that record the last confirmed offset or timestamp, and with idempotency keys on writes so that a repeated record updates instead of inserting. Confirm which of these the destination supports before you start the replay. For an illustrative case, a database writer with a unique event identifier can skip a record it has already stored, while an append-only file target cannot. The step is done when a test replay of a small window shows no duplicate rows at the destination.

8. Rejoin the cluster and confirm steady state

   Once the node is verified, allow it to join the cluster and re-enable the source connections you paused. Watch the cluster view for the node reaching a connected state, and check that its queues drain and that back pressure does not build up on the connections it handles. Reconcile the destination counts against the source for the replay window, and record the final flow version and backup identifier in the recovery record. The step is done when the node shows as connected, the flows it handles are processing normally, and the reconciliation for the replay window is signed off.

Handling in-flight data and duplicate records after recovery

Recovery options and when each fits

This table compares four ways to recover a NiFi deployment, from restoring everything on disk to rebuilding nodes and replaying data from upstream sources. Read each row for what it protects, what it costs in storage and restore time, and the conditions under which it fits. The main takeaway is that the more of the flow you can rebuild from Git and configuration, the less you need to copy from the repositories, but in-flight data is only protected by the repositories or by upstream replay.

How to set up a NiFi backup and recovery routine

This sequence turns a running NiFi into a recoverable system by listing what each node holds, protecting keys and sensitive values, versioning flows outside the server, automating encrypted backups, and proving the restore works in a separate environment before you need it.

1. Inventory artifacts on every node

   Walk each node and list what it holds: the flow configuration file (flow.json.gz on NiFi 2), conf files such as nifi.properties, authorizers and login identity providers, the keystore and truststore, the sensitive properties key, bootstrap and logging configuration, custom NARs, and the content, provenance, flowfile and state repositories. Mark each item as back up, rebuild, or regenerate, and record its location. The step is done when a reviewer can point to every item on the list for every node, and when the cluster-wide items (such as the shared flow and cluster state) are identified separately from per-node items. Confirm the repository locations and file names against the NiFi System Administrator's Guide for your exact version, since layout differs between 1.x and 2.x.

2. Move keys and sensitive values into a vault

   Identify every secret the flow depends on, including the sensitive properties key, keystore passwords, and the sensitive parameter values used by controller services and processors. Store them in a controlled secrets manager such as OpenBao or another vault your organization operates, and reference them through NiFi parameters or providers rather than plain text in the flow. Restrict who can read them and log access. Check that a copy of the sensitive properties key exists outside the NiFi host, because a flow backup without that key cannot decrypt its sensitive values. Done when you can rebuild a node's configuration from the vault alone and the key copy has been verified to decrypt a test value.

3. Version flows in a Git-based registry

   Connect NiFi to a Git-backed Flow Registry Client, such as the GitHub or GitLab clients documented for NiFi 2, and commit each production process group as a versioned flow with its parameter contexts handled deliberately. Keep the repository under your own control with branch protection and review for changes. Flow versions give you a readable history of what the flow was, which the flow.json.gz file alone does not. Note that NiFi Registry is deprecated and planned for removal in NiFi 3.0, so avoid building a new routine that depends on it. The step is complete when a changed process group can be committed, pulled into a second environment, and shows the same version identifier.

4. Schedule and encrypt the backups

   Schedule a job for each node that copies the configuration set and the flow configuration file, and a separate cluster-level copy of shared state where your deployment uses it. Stop or quiesce writes where the copy would otherwise capture a half-written file, or use a snapshot of the underlying volume taken at a consistent point. Encrypt the archives before they leave the host and store them in a location with separate credentials and retention rules. Content and provenance repositories are large and may be better rebuilt or accepted as loss, which is a business decision to record. Verify the job runs and that the encrypted archive can be listed and decrypted. Finish when the job log, the archive checksum and a decryption check are all recorded.

5. Document the restore runbook

   Write a runbook covering the order of restoration: install the same NiFi version and Java runtime, restore configuration and keys, recover the flow from backup or pull it from Git, restore cluster coordination (ZooKeeper or Kubernetes state), and start nodes one at a time. Include the decisions the operator must make, such as whether in-flight data from the content repository is replayed, discarded, or reconciled with the source systems, and how to avoid duplicate writes to downstream systems for items already processed. Name the owner of each action and the evidence to collect. The runbook is finished when a second engineer who did not write it can follow it on paper and identify each check.

6. Run a restore test in isolation

   Restore the backup into a separate environment that cannot reach production sinks or sources, using test credentials and a copy of the vault entries. Confirm that the flow loads without errors, that every controller service enables, and that restored sensitive values decrypt. Start the nodes, then confirm they join the cluster without flow mismatches and that the cluster shows the expected node count. Review queues and back pressure, and check the provenance and bulletins for errors that appeared during startup. Record which checks passed, which failed, and the time each took, then fix the runbook where it was unclear. The test is done when the isolated cluster matches the production flow version and all checks are recorded.

7. Repeat the restore test on a schedule

   Set a recurring restore test in your change calendar and assign an owner who is not the person who built the backup job. Each run repeats the isolated restore, uses the current backup and the current runbook, and updates the record of results. Treat any new NiFi version, processor bundle, or vault change as a trigger for an extra run before it reaches production. Review the findings with the operations and security owners, and confirm that access to backups and keys is revoked for people who have left the team. The routine is working when recent test results are on file, the owner has signed them off, and the runbook matches the live configuration.

Protecting backups that contain keys and flow secrets

A NiFi backup is only as safe as the material inside it. The flow definition, sensitive property values, security configuration and certificates can each expose the systems NiFi connects to, so they need controls that match their sensitivity. The cards below cover where that material sits, how to separate it, and how to prove who touched it.

Questions teams ask before restoring or hiring help for NiFi recovery