Why collect Apache NiFi diagnostics before a restart
When a NiFi node or cluster is hung, slow or out of memory, the evidence should be captured before anyone restarts it. A restart clears the threads, queue state in memory and heap contents that would explain the failure, and the same problem often returns within hours. Spend a few minutes collecting a diagnostics snapshot, one or more thread dumps and, where memory is the suspect, a heap dump, then restart. Confirm the exact commands, options and output locations against the NiFi version you run, since the administration guide documents them per release and the names change between versions.
The three artifacts answer different questions. A diagnostics report is a point-in-time snapshot of flow status, connection queues, processor states, bulletins and system information, and it is the first thing to collect because it is cheap and shows which part of the flow is backed up. A thread dump shows what every JVM thread is doing at one moment, which is the right tool when processors appear stuck or the UI stops responding. Take several dumps spaced apart, because a single dump cannot distinguish a thread that is briefly waiting from one that has been blocked for minutes. A heap dump is large and pauses or slows the JVM while it is written, so use it when you suspect memory exhaustion or a leak, and plan disk space and transfer for the file before you trigger it.
Ask your support team, or your own platform engineers, which of these they need and in what form before the incident. Agree where the files are stored, how they are transferred off the node, and who may receive heap dumps, since they can contain flow data, parameter values and credentials held in memory. Keeping these steps in a runbook means the evidence is collected the same way each time, whichever engineer is on call.
Which evidence to collect for which symptom
This table compares the four kinds of evidence a team can collect from a struggling NiFi node, with the symptom each one fits, what it captures, and what it costs the running node. Read it before restarting anything, starting with the lightest option that answers your question. The main takeaway is that heap dumps carry the highest risk on a live production system and should be a deliberate, approved step.
| Evidence | Symptom it fits | Data captured | Cost and risk on a production node |
|---|---|---|---|
| Diagnostics command (default) | Node slow, unresponsive or misconfigured, with no clear cause yet | Node and flow configuration summary, system state and runtime details (confirm contents for your version) | Low cost. Read-only collection. Confirm what it writes and where before running it on a production host. |
| Diagnostics command with verbose mode | Same symptoms, when the default output does not explain the cause | Everything in the default output plus more detailed component and runtime data (confirm scope in the admin guide) | Larger output and more collection time. Review the file for sensitive values before sharing it outside your team. |
| Automatic diagnostics on shutdown | Repeated restarts or a node that fails in a way you want to explain afterward | Diagnostic snapshot written as the node stops (confirm the setting and file location for your version) | Captures evidence only when a shutdown is clean. A forced kill or crash may write nothing, so do not rely on it alone. |
| Thread dump | Processors or the UI appear hung, threads seem blocked, or the Long-Running Task Monitor flags a task | Stack traces of all JVM threads at one moment, showing what each thread is waiting on | Short pause in activity while captured, and usually taken in a series. Output can reveal flow names and remote hostnames. |
| Heap dump | Suspected memory leak, rising heap use or out-of-memory errors that a thread dump cannot explain | Full snapshot of objects in memory, which can include flow data, credentials or other sensitive values | Highest risk. Can pause the JVM, needs disk space in proportion to heap size, and the file is sensitive. Get change approval first. |
| Logs and bulletins (for comparison) | Errors that name a component, or a timeline of when symptoms began | Application log lines and component bulletins already written by the node | No added load. Often the fastest first read, and it should be gathered before any other evidence. |
How to capture diagnostics safely on a running node
This sequence collects the evidence a support team or your own engineers need before anyone restarts a struggling NiFi node. It runs from a reachability check through bulletins, the diagnostics command, thread and heap dumps, and ends with an archive that can be read later without the live node.
Confirm the node is still reachable
Before touching anything, the operator checks that the node answers on its management interface and that the NiFi API or UI responds from the jump host or bastion used for administration. Record which node you are on, its hostname, the NiFi version and the time in UTC, because every later file depends on those identifiers. If the node does not respond at all, note that as the first finding and move to host-level checks such as process status and disk space. Done when you have a short written note with node name, version and timestamp, and a successful status response from the node.
Record bulletins and current state
Export the bulletin board and the cluster view while the symptoms are still visible. Capture the bulletins with their levels, source components and timestamps, and note which connections show back pressure, which processors are running, stopped or invalid, and whether any node in a cluster is disconnected or offloading. Save these as plain files named with the node identifier and time. Done when the bulletin export and state summary exist and are not empty, and you can count the bulletins you saved against what the UI shows.
Run the diagnostics command
Run the NiFi diagnostics command from the administration guide on the affected node, using the syntax for your installed version, which the guide documents. Add the verbose option only when the first pass lacks detail, because verbose output is larger and takes longer to produce. Confirm the exact command and output location in the administration guide before running it on production. Done when the diagnostics output file exists in the expected location, has a size greater than zero, and opens as readable text with the expected sections.
Capture thread dumps where justified
Take thread dumps when processors or the framework appear hung, for example when a processor shows a long-running task or a thread count keeps climbing. Take several dumps a short interval apart so you can compare them, since one snapshot rarely proves a thread is stuck. Use the JDK tooling available on the host and record the process identifier and interval with each file. Done when each dump file exists, is not empty, and includes thread entries for the NiFi process. Confirm the dump method and any permissions required in your environment before running it.
Capture a heap dump only when justified
A heap dump is large, can pause the JVM and may contain sensitive flow data, so take one only when memory pressure or an out-of-memory signal is the suspected cause and the support team has asked for it. Confirm the storage location has room for the file, and restrict access to it because it can hold credentials, payloads and configuration values. Done when the dump file exists, its size is consistent with the heap size, and the file permissions limit reads to the operators who need it.
Store evidence with timestamps and node identifiers
Move all outputs into one directory per incident, named with the cluster, node, NiFi version and UTC time. Write a short manifest listing each file, its size, its checksum and the command that produced it. Keep the bundle separate from the live node's data and logs so a later restart or disk cleanup cannot remove it. Done when the manifest matches the files on disk, every checksum is recorded, and the bundle has been copied to storage that survives the node restart.
Prepare the bundle for the support team
Send the support team the manifest, bulletins, state summary, diagnostics output, thread dumps and, where taken, the heap dump through the channel your agreement specifies. Remove or mask secrets in anything you share unless the team has asked for unmasked data, and say which files contain sensitive content. Include the symptom timeline and the action you plan next, such as a restart. Done when the support team confirms receipt and you have a record of what was shared and when.
How diagnostics, thread dumps and heap dumps differ
Each artifact answers a different question, so collect the one that matches the symptom. A NiFi diagnostics bundle is a point-in-time record of the node's state: flow status, queue sizes, system diagnostics and component details that support staff can read alongside the logs. A thread dump lists what every JVM thread is doing at one instant, which shows where a processor is blocked but not whether it has been blocked for a long time. A heap dump copies the objects in memory, which shows what is consuming the heap but not why the garbage collector is struggling right now. Before you restart anything, confirm with your support channel which of these they need, because a thread dump alone rarely explains a full disk or a disconnected node.
A single thread dump proves very little, so plan for a series of them. A thread that sits in the same method across several dumps taken a short interval apart is a candidate for a stuck remote call. A thread that appears once is usually ordinary work. Ask your team to capture at least three dumps spaced far enough apart to see whether the same stack persists, and record the time of each. Keep in mind that the dump itself briefly affects the JVM, so repeated captures on a node under heavy load add to the problem you are investigating. Use the Long-Running Task Monitor to identify which threads have exceeded a threshold, and use the dumps as the evidence for that thread's stack.
Heap dumps and shutdown diagnostics have strict prerequisites. A heap dump can be many times the size of the live heap and writing it can pause the JVM long enough for the node to miss heartbeats, so confirm free disk space on the target volume and agree on a window with the operations team before taking one. Automatic diagnostics on shutdown only help when NiFi can still stop cleanly, because a node that is hung or out of memory may never reach that point, so do not rely on it as your only evidence. Bulletins and their alerting are covered separately, and they often point you to the processor whose thread dump is worth examining.
Security risks in diagnostic files
A diagnostics bundle or thread dump is a snapshot of a running NiFi, and it can carry the same sensitive detail as the flow itself. The cards below cover what the files contain, who can reach them, how they are protected in storage and transit, and how to redact them before anyone outside the team sees them.
Flow configuration in the bundle
A diagnostics bundle describes processors, connections, their properties and the topology of your dataflows. An outsider with that picture knows where data enters, where it lands and which systems NiFi talks to. Before you collect, confirm with your team which flows are sensitive and whether the bundle scope is acceptable for the people who will read it.
Parameter values and property material
Parameter values and processor properties can include database URLs, usernames, API endpoints and, in some setups, secret material. NiFi marks sensitive properties so they are not displayed in the UI, but you should verify in your own bundle how values are handled rather than assume every field is masked. Check the bundle for literal values before it leaves your environment.
Hostnames and internal addresses
Thread dumps and bundles name hosts, ports and cluster node addresses. That map shows an attacker where the nodes, ZooKeeper members or external services sit inside your network. Treat internal addresses as sensitive and replace them with consistent placeholders when a vendor or partner needs the evidence.
User names and identities in logs
Diagnostics and the surrounding logs can record user identities, client certificate subjects and the accounts behind REST calls. Those names help an adversary target specific operators. Confirm which identity fields appear in your bundle and remove or pseudonymize them before sharing outside the operations team.
Access to run and read diagnostics
Generating a diagnostics bundle and taking a thread dump need elevated permissions in NiFi, and reading the output should be limited to the same group. Review which policies grant access to the diagnostics and system-level operations, and use the admin guide to confirm the policy names in your version. Restrict who can download the files, not just who can trigger them.
Storage and transfer of the files
Bundles and heap dumps sit on the node's disk and may be copied through email, tickets or file shares. Store them on an encrypted volume with restricted ownership, delete them when the case closes, and send them only over encrypted channels. Ask your support provider which transfer method they accept before you collect anything.
Heap dumps hold live data
A heap dump captures the full memory of the JVM, including flowfile content and attribute values in flight. It is far more sensitive than a thread dump and can be large. Decide in advance whether a heap dump is allowed under your data handling rules, and redact or exclude it when the trigger is only a thread-level question.
What to send a support team and what to keep internal
Send a support team a dated evidence package rather than a description of the symptom. The package should state the NiFi version exactly as the UI and nifi.properties report it, the cluster topology (node count, whether ZooKeeper is embedded or external, and which node was the primary or coordinator at the time), and a timeline with the moment the slowdown began, the moment someone first noticed, and every restart or configuration change in between. Attach the bulletins that fired in that window, the nifi-app.log and nifi-bootstrap.log excerpts covering it, the diagnostics output produced by the diagnostics command with verbose mode enabled, and at least two thread dumps taken minutes apart so that a stuck thread can be told apart from a busy one. If the flow is versioned in Git through a Flow Registry Client, include the commit history for the affected process group, because a change that looks unrelated to the symptom often explains it.
Scrub before anything leaves your network. Remove or replace hostnames, IP addresses, user names, DNs in certificates, and any values of sensitive parameters, controller service properties, and processor properties. Diagnostics output and thread dumps can contain file paths, SQL text, endpoint URLs with embedded tokens, and flow component names that reveal business logic, so review them line by line rather than relying on a search and replace. Withhold keystore and truststore files, sensitive.props.key, flow.json.gz contents unless the vendor asks for them under an agreed channel, provenance records that contain payload attributes, and any content repository data. Keep a copy of the unredacted originals under your own control and record which redactions were made, so the support team can ask for a specific value without you sending the whole file.
DeployOpen performs diagnosis within an agreed written scope and does not replace the Apache NiFi project's community channels or a vendor support agreement where one applies. Before engaging us, confirm with your own team which channel you will use for upstream questions, who on your side approves disclosure of configuration, and how long collected evidence will be retained. Ask us to state in the scope which evidence we will review, which environments we may access, and what we will hand back. Any finding that points to a defect in NiFi itself should be reported through the project's own process, and we will help you prepare that report from the same package.
Questions teams ask about NiFi diagnostics support
Can our own team collect NiFi diagnostics without outside help?
Yes, in most cases. Collecting diagnostics is a supported administrator task, and the work is mostly running the right commands on the right node at the right moment. The harder part is knowing what to capture before a restart erases the evidence. Teams with a Linux administrator who can run the NiFi toolkit, read JVM output and handle files securely can usually collect a first set themselves. Outside help adds the most value when the team cannot tell whether a symptom is a flow design problem, a repository problem or a JVM problem, or when the evidence has to be interpreted quickly for a vendor or internal review. Confirm your own staff's experience with the JVM before deciding.
How long does a NiFi diagnosis take, and what drives the time?
The time depends on how many nodes are affected, how much evidence exists, and whether the team can reproduce the symptom. Collection itself is usually short once the procedure is agreed. Analysis takes longer. The main drivers are the cluster size, the number of flows and processors involved, whether logs and bulletins are retained long enough to cover the incident, and how quickly access to the affected environment is granted. Heap dumps and large thread dumps add transfer and analysis time. Ask for the expected steps and the dependencies in writing so the timeline reflects your environment rather than a generic estimate.
What does a diagnostic engagement need from our team?
The engagement needs named people and access, not much of their time. Typical inputs include:
- An administrator who can run commands on each NiFi node and copy files out of the environment.
- Read access to the NiFi UI, bulletins, logs and the current flow definition.
- A decision-maker who approves capturing heap dumps, since they contain sensitive data.
- A secure transfer path for evidence, and an agreed owner for the incident timeline.
Also confirm who can authorize a restart, because the diagnostic steps should happen before any restart is planned.
Is a heap dump safe to take on a production NiFi node?
A heap dump is not free of risk. Capturing one usually pauses the JVM while the memory is written, so a node can stall for a period that depends on heap size and disk speed, and it may be unresponsive during that time. The file can also contain flow data, credentials held in memory, and other sensitive values, so it must be stored and transferred under your data handling rules. Plan the capture for a window your operations team accepts, confirm that the target disk has enough free space, and check whether the node is in a cluster where the pause could trigger a disconnect. Confirm these points with your NiFi and JVM documentation for your exact version before the first capture.
How does a consultant scope a NiFi diagnostics engagement?
A consultant scopes the work by asking what is failing, how it appears, and what evidence already exists. The scope usually covers the NiFi version and deployment type, the number of nodes, the flows and components suspected, the logs and bulletins available, and the access model. It also records what is out of scope, such as changing flow logic or repairing the environment. The output is a written scope with deliverables, assumptions and the access each step needs. Ask the consultant to list which decisions remain with your team, for example restarts, data replay and production changes.
What does a fixed-scope diagnostic review include?
A fixed-scope diagnostic review is an agreed set of steps with defined deliverables, and it is usually written before work starts. Expect the review to cover a collection procedure for diagnostics, verbose output and thread dumps, a review of bulletins and logs for the same period, and a written findings report that separates facts from hypotheses. It may include recommendations for monitoring or for a follow-up fix, but fixes are normally separate work. Ask for the exact list of artifacts, the handling rules for sensitive files, and what happens if the evidence shows a problem outside the agreed scope.
What makes a diagnostics quote cost more or less?
Cost is driven by the amount of work and the access it requires. The main cost drivers are the number of nodes and environments, the number of flows and custom components, the availability of retained logs and evidence, the time needed for analysis, and whether the work must happen during an out-of-hours window. A one-time diagnostic review differs from an ongoing support retainer, which recurs. When requesting a quote, ask for the cost split by collection, analysis, reporting and any follow-up, and ask which items are one-time and which recur. DeployOpen scopes work in writing, and the quote follows from that scope.
How DeployOpen approaches NiFi diagnostics
DeployOpen starts a diagnostic review with a written scope before anyone touches the running NiFi. The scope names the symptom under review, the nodes and environments involved, the evidence sources that are in bounds (diagnostics output, thread dumps, heap dumps, logs and bulletins), and who on the client side approves each collection step. Because a thread dump or heap dump pauses or slows a JVM, and a heap dump can contain sensitive flow content, the client's change owner should agree on timing and storage location before capture begins. The written scope also states what is out of bounds, such as restarts or node offloads, so that evidence collection does not turn into an unplanned change.
Collection happens with the client's operators, running commands from their own hosts under their change controls and privileged access process. DeployOpen can supply a capture checklist covering the diagnostics command with verbose output, the relevant thread dump intervals, the nifi-app.log and bootstrap log windows, and the node state at the time of capture. Before any artifact leaves the client environment, it is reviewed and redacted: flow definitions, parameter values, hostnames, IP addresses, user identities and any sensitive property values are removed or replaced. The client keeps the unredacted originals. Findings are then handed back as a written report that separates observed facts from likely causes, followed by a remediation plan with owners, change steps and a verification check for each item. Ongoing support, upgrades and continuous testing are available as retainers, but a diagnostic review does not require one. DeployOpen is independent of the Apache NiFi project and is not an official partner of it.
