Hyperledger Besu QBFT validator operations

Adding or removing a QBFT validator is a consensus event, not a restart. Existing validators propose a change and cast votes. Once the required majority is reached, the new validator set takes effect, and the set is recorded in the chain so every node applies the same membership from the same point onward. Block production continues while the vote runs, which is why operators can change the set on a live network. Your job before the first vote is to make sure every participant understands what the chain will accept and what it will refuse, because the vote is only as trustworthy as the rules behind it.

Write the governance rules down and get them signed before anyone submits a proposal. The document should answer five questions. Who may propose a change? Who may vote, and does each member hold one vote? How is a candidate vetted for operational competence, legal standing and data handling? How is the new node's identity verified, including who confirms that the candidate controls the node key? And how must a removal be justified, and who can contest it? Ambiguity on any of these will surface at the worst moment, typically when a member wants out during an incident. Confirm the vote mechanics against the current Besu QBFT documentation so the written rules match what the software actually does.

Identity verification deserves its own procedure. A validator is identified by the public key of its node key, and Besu's documentation states that the local node private key file is not encrypted. That means the custody of each candidate's key file is part of the vetting decision, not an afterthought. Ask candidates how the key is stored, who can read it and whether a security module plugin such as the Besu HSM plugin is in use. Check the key fingerprint out of band, through a channel separate from the proposal itself, before any vote. Node permissioning, which restricts connections to known participants, should be updated in the same change window so the new node can actually reach the network.

Treat fault tolerance as the main risk of every membership change. Adding validators changes the margin of failures the network can absorb, and removing one can reduce it, sometimes to a point where a single additional outage stops block production. Before any proposal, calculate the threshold for the exact validator count you will have after the change, and confirm that arithmetic in the Besu documentation rather than relying on general BFT rules of thumb. Record the before and after numbers in the change ticket so the consortium approves the change knowing its effect on resilience.

Rehearse every add and remove on a test network first. Build a network with the same consensus settings and validator count, run the full proposal and vote sequence, and confirm that every node reports the same validator set afterward. Then remove a validator on purpose and watch whether blocks keep being produced at the expected cadence. Enable metrics on the test nodes so you can see peer counts and block production during each step. Keep a written runbook from the rehearsal, including the rollback path, and repeat the rehearsal after each Besu upgrade, because the behavior you tested is tied to the release you tested.

When a QBFT network stops producing blocks, the first task is to prove the stall is real and identify its cause before any validator is changed. Query the latest block number from every validator's JSON-RPC endpoint and compare the results. Check the peer count on each node as well. A validator that is behind but still connected to peers points to a different problem than a set of nodes that have lost contact with each other. Record timestamps for each check, because a single snapshot rarely shows whether the chain is halted or only slow. Keep these observations in the incident log so operators work from the same evidence.

Once the stall is confirmed, sort it into one of three causes. The first is too few validators online to reach agreement. Confirm the fault tolerance of your validator set and the agreement threshold for your Besu version in the documentation, then compare that figure with the number of validators actually reachable. The second is a clock or network problem affecting consensus, such as validators whose system clocks have drifted or peer-to-peer traffic blocked between sites. The third is a bad configuration change, including edits to genesis or consensus settings, block time or timeout values, or validator vote changes. Assign one owner to each hypothesis so the team tests them in order.

Changes to a stalled network carry their own risk. Restarting many validators at once, or replacing validator keys during the incident, can produce conflicting state and makes it hard to establish which node signed what. Every change should follow a written recovery plan that operators agreed before work began. The plan should name the order of restarts, the person who approves each step, the first node to touch, the criteria for stopping and rolling back, and who communicates with the consortium. Leave node key files unchanged unless the plan explicitly calls for it. The local node private key file is not encrypted, so any copy made during the incident must be controlled and removed afterward.

Take a backup before any action that touches a node's database, including a restart with a new binary, a resync, or a configuration change that alters storage. The Besu 26.10.0 release notes state that the Bonsai database upgrades automatically on first start and that older Besu versions cannot open an upgraded database, so rollback depends entirely on a backup taken beforehand. Stop the node cleanly before copying its data directory, and check that the copy is complete. Where possible, test a restore on a separate host so the team knows the backup works before it is needed.

Recovery should restore liveness by returning validators to agreement on the chain that already exists. Committed blocks must not be rewritten, and any plan that appears to require editing history should be stopped and escalated. Before an incident occurs, confirm in the Besu documentation for your exact version how finality behaves under QBFT and what the recommended recovery procedure is. Rehearse the sequence on a non-production network that mirrors your validator count, key custody and monitoring, so the on-call team has practiced each step before a live stall.

Every validator change should be proven in a test environment that matches production, both before and after the change. Start with a baseline. Confirm that all validators are producing or agreeing on blocks, meaning every node reports the same head hash at the same height. Record the validator set at that height using the QBFT validator query available through your node's RPC interface, so that later comparisons have a fixed reference. Keep the baseline with the change record. A vote without a baseline gives you no way to show later whether the network behaved differently because of the change.

Next, test adding and removing a validator. Propose a vote to add a test validator, then confirm that the new set appears on every node, not only on the node that proposed the vote, once the vote takes effect. Check that the test validator joins, takes part in block production, and that the chain keeps producing blocks. Then vote the test validator out and repeat the agreement check. Confirm the exact vote procedure and the majority threshold in the Besu private network documentation for the release you run, because the procedure depends on version and your team should not rely on memory.

Then test failure behavior. In the test environment, stop validators one at a time and confirm that blocks continue and all remaining nodes agree while the fault tolerance of your validator count still holds. Stop validators until the threshold is reached, then stop one more. The expected result is that the chain halts. Confirm that it halts without producing blocks that some nodes accept and others reject, and that no node reports a different head. Restart the stopped validators and confirm that each resyncs from its peers and reaches the same head without forking. Rehearse restarts on a copy of the data directory so that production state never becomes the experiment.

Ongoing health checks should run continuously in production. Compare block height and head hash across validators, and alert on any sustained disagreement. Track peer counts per node, since a validator with few peers may lag. Confirm that the metrics endpoint is reachable from your Prometheus server, using the scrape setup in the Besu metrics documentation. Check key file integrity as well: the node private key file is not encrypted by Besu, so verify its owner, permissions and a stored hash, and alert on any change. Finally, stop a validator in the test environment and confirm that the missing-validator alert fires. Alert design is covered on the monitoring page in this series, so use it for thresholds and routing.