How to upgrade Besu validators one at a time and keep the Besu upgrade path safe
If your permissioned QBFT network must move onto newer Besu calendar releases, such as 26.7.0 through 26.10.0, you can do it without halting block production by upgrading one validator at a time, and only after you have checked the upgrade path for each target version. The rolling sequence is what keeps the chain live: each validator is stopped, upgraded, restarted and confirmed as peered and producing or following blocks before the next one is touched, so the network keeps enough validators online to reach QBFT agreement throughout. Plan the order, the pause between nodes and the abort conditions before the first node is touched.
The supported path depends on what each release changes. Read the release notes for every version between your current build and your target, because they list breaking changes. Besu 26.6.0 requires JDK 25 to build and run, so hosts, container images and any custom plugin builds must move with it. Besu 26.4.0 removed Clique consensus, so a network that still depends on Clique cannot start or produce blocks on that release. Besu 26.10.0 auto-upgrades the Bonsai database on first start, and older versions cannot open the upgraded database, so a verified backup of each node's data must exist before that node is upgraded. Confirm each of these points against the official upgrade documentation for your exact target version, starting with the Besu upgrade guide, because release behavior can change between builds.
Which Besu release changes affect a rolling validator upgrade
This table compares the supplied Besu release changes that touch validators during a rolling upgrade, one row per change. Read each row as a gate: the action column describes what must be true on every validator before it is restarted, and the impact column explains why a skipped step can stall the network or make a node unrecoverable.
| Release change | Impact on validators | Action before the rolling restart |
|---|---|---|
| 26.6.0: JDK 25 required | Nodes built or run on an older JDK will not build or start on 26.6.0 or later, so a validator can fail on restart while its peers keep running. | Confirm the JDK 25 runtime on every host or image, and rebuild any custom plugin against it before the first restart. |
| 26.4.0: Clique removed | Besu can no longer start or produce blocks on pure Clique networks, so a Clique-based validator set cannot be carried forward by a rolling restart. | Confirm the chain uses QBFT or IBFT 2.0 and no node still expects Clique. Treat any Clique network as a migration project, not a restart. |
| 26.10.0: Bonsai database auto-upgrades | The database upgrades on first start and older Besu versions cannot open it, so a restarted validator cannot be rolled back by switching binaries. | Take and verify a full node data backup for each validator before its restart, and record the backup location against the change. |
| 25.6.0: Tessera, onchain permissioning removed | Privacy and smart-contract-based permissioning no longer run in Besu, so validators that depend on them will not behave as before after upgrade. | Identify any node still using Tessera or onchain permissioning, and move it to local permissioning or a plugin before the rolling restart. |
| 25.7.0: EEA and PRIV RPC groups removed | Applications or scripts calling EEA or PRIV methods will fail after the node restarts, which can look like a network fault. | Search client code and monitoring for EEA and PRIV calls, and confirm each caller is updated in staging. |
| Announced: fast sync and proof of work sunsetting | Nodes configured for fast sync or proof of work may not behave as expected once these features are removed, and a validator config may carry settings that no longer apply. | Review each validator configuration file for fast sync or proof of work settings, and confirm the change plan against the release notes for the version you target. |
Why keeping two-thirds of validators online keeps QBFT producing blocks
QBFT finalizes a block only when a Byzantine fault tolerant quorum of validators agrees on it, so a rolling upgrade must keep at least two-thirds of the validator set running at every step. The rule is about the validators, not the RPC nodes or the bootnodes. A validator that is stopped for its upgrade is one of the faulty or absent members the consensus must tolerate, and the operator decides how many of those the network can absorb at one moment. Before the first restart, write down the total number of validators in the active validator set, calculate two-thirds of that figure and round up to a whole validator, then confirm the result against the validator list in the genesis file and the current on-chain set if your network uses contract-based validator selection. The gap between the total and the quorum threshold is the number of validators you may restart at once, which is often one on a small set.
The risk comes from restarting a second validator before the first has rejoined consensus and caught up to the chain head. A node that has just been upgraded may take time to reconnect to peers, sync missing blocks and resume voting, so during that window it counts as offline even though its process is running. If the operator starts the next validator on a timer rather than on a health check, the network can drop below the threshold and stop producing blocks until one of them returns. Gate each step on the restarted validator reporting that it is in sync and participating, and on its peers being connected, before moving to the next host. Use the Besu metrics endpoint described in the Besu metrics documentation to check these conditions, and enable it with the --metrics-enabled option before the rollout begins.
Confirm the exact quorum behavior, the round timing and any validator set changes for your version in the Besu QBFT documentation and the release notes for the target release before you write the runbook. Check whether your network has any scheduled validator set changes that would move the threshold during the upgrade window. If a validator set change is pending, the calculation must use the set that will be active at each step, not the set you started with. Record the arithmetic, the threshold you agreed and the health checks you used in the change record, so that operators on call can apply the same rule under pressure without recalculating it.
What to prepare before the first validator restarts
Four preparation items should be finished before any validator restarts on a new client version. Each one protects a different way the upgrade can fail: lost data, untracked configuration, broken plugins or images, and untested behavior. Complete them in parallel where you can, and treat the backup as the gate that everything else depends on.
Verified backup of each node
Take a backup of every node's data directory before the first restart, and confirm that it can be restored. The Bonsai database upgrades automatically on first start, and older Besu versions cannot open an upgraded database, so this backup is your only rollback path. Record the Besu version and the backup timestamp next to each node so an operator can find the right copy during an incident. Test a restore on a spare host before relying on it.
Configuration under version control
Keep every node's configuration file, service unit, container definition or Kubernetes manifest in a Git repository before the upgrade starts. The Besu upgrade guide recommends preserving node data and configuration and keeping configuration under version control. A tagged commit gives you an exact before and after diff for each node and a fast way to restore settings if a flag changed meaning between releases. Review the release notes for flags that were renamed or removed.
Plugin and image inventory with JDK 25
List every plugin, custom image and JVM option in use, and record the Besu version each one was built against. Besu 26.6.0 and later require JDK 25 to build and run, so any custom image, base layer or plugin build that still targets an older JDK will fail after the swap. Confirm with each plugin's maintainer which Besu versions it supports. Pin the tested combination in your pipeline so that a rebuild cannot pull in an untested version.
Staging network from the same configuration
Build a staging network from the same configuration repository, with the same validator count, genesis settings and consensus parameters as production but on isolated hosts. Run the upgrade there first, in the same order you plan for production, and confirm that blocks keep being produced and that peers reconnect after each restart. Staging proves the procedure and the configuration; it does not prove production capacity, so keep those two questions separate in your test plan.
Permissions and key file inventory
Record the permissions configuration files, the JSON-RPC credentials or JWT key settings, and the location of each node's private key. The local node private key file is not encrypted, so confirm who can read it and where copies exist. If your nodes use a hardware security module through the Besu HSM plugin, check that the plugin version matches the Besu version in your plan. Restore keys and permission files alongside the data backup so a rebuilt node joins with the expected identity and allowlists.
How the rolling validator upgrade runs node by node
The sequence keeps a QBFT network producing blocks while each validator moves to the new release, with a check and a rollback point before every restart.
Freeze changes and record baseline health
The network operator and the change owner agree a freeze on configuration and deployment changes for the upgrade window, then record the current state of every validator: client version, peer count, chain head, and whether each node is signing blocks. Keep this record with the change ticket so the team can compare later. The step is done when every validator reports a healthy baseline and the freeze is announced to all operators who run nodes on the network.
Rehearse the upgrade on a staging network
The platform team reads the release notes for every version between the current and target release, looking for breaking changes, removed features and new runtime requirements. They then apply the same binary or image, configuration and data layout to a staging network that mirrors the production validator set. The rehearsal is complete when the staging network produces blocks on the new version, the restart procedure has been timed by the people who will run it, and any configuration fix is written into the production runbook.
Confirm the validator set and quorum margin
Before touching production, the operator lists the active validators from the chain and checks how many can be offline at once while the network still reaches the two-thirds threshold that QBFT needs to finalize blocks. The team also confirms which validators belong to which organization and who must be available for each restart. The step closes when the team has a written table of validators, owners and the number of nodes that may be down at any moment, and that number is agreed by everyone who signs off the window.
Choose the first validator to upgrade
The operator picks a validator whose loss is least disruptive, often a spare or non-critical node that is still part of the validator set, and confirms it is not the only node a counterparty depends on for access. The choice is recorded with the reason. The step is done when the selected node, its owner and its restart window are named in the change record and the owner has acknowledged the plan.
Stop the node and back up its data
The operator stops the Besu process or container cleanly, then takes a backup of the node data directory, the configuration files, the permissions files and the node key file, stored outside the host. For releases that migrate the database on first start, this backup is the only rollback path. The step is complete when the backup is written, its checksum or size is recorded, and a restore has been confirmed to be possible on another host or path.
Replace the binary or image and start the node
On Linux, the operator replaces the Besu installation with the target release. On Docker, the image tag is changed in the compose or run configuration. On Kubernetes, the StatefulSet image is updated so the pod restarts with the same volume. Configuration is kept under version control and applied unchanged, except for any key that the release notes require to change. The node is then started. The step is done when the process runs the new version and logs show the expected release identifier and no configuration errors.
Confirm peer count and block import
The operator checks that the upgraded node reconnects to its peers, and that its peer count matches the other validators, using the node's peer and sync information through the JSON-RPC interface and the logs. They then confirm that the node imports new blocks at the same height as the rest of the network. The step closes when the node has rejoined the peer mesh and its chain head tracks the network without falling behind, with the values recorded in the change log.
Confirm the node signs again and repeat
The operator checks that the upgraded validator proposes or signs blocks in the QBFT rounds in which it is selected, and that the network keeps finalizing blocks throughout. Only after this check passes does the team move to the next validator, repeating the same steps in the agreed order. The sequence stops and the team rolls back to the backup if the network loses the two-thirds margin, a node fails to rejoin peers, blocks stop importing, or the release notes conflict with the configuration in a way that cannot be fixed within the window. Operators confirm the exact steps in the Besu upgrade docs before the first production restart, which is the next read in this series: Besu docs: Upgrade your node.
How the steps change on Linux, Docker, Kubernetes and Ansible
On a Linux host, Besu usually runs as a process under a service manager such as systemd. Each validator upgrade is a stop, a binary replacement and a start on the same machine. The data directory and configuration stay in place, so the operator controls every step directly. Confirm your own service definition and file paths before you change anything.
- Restart mechanics: Stop the service, replace the Besu distribution, then start the service again. Check that the service definition points to the new binary and to a JDK that meets the release requirement.
- Configuration and data: Keep the configuration file and the data directory outside the install directory so swapping the binary does not touch them. Keep configuration under version control, as the upgrade guide recommends.
- Java runtime: Besu 26.6.0 and later require JDK 25 according to its release notes. Confirm the runtime version on the host before the restart.
- Backups: Take a backup of the data directory before each upgrade if the release changes the database format, since older versions cannot open an upgraded database.
- Verification: After the start, confirm the service is active, the node reconnects to its peers, new blocks import, and the validator signs in its expected turn.
Before moving to the next validator, check the service logs and peer count on the node you just upgraded.
In Docker, the Besu process runs inside a container built from an image. An upgrade replaces the container with one created from a new image tag while reusing the same data volume. The image carries the runtime, so the JDK version is set by the image you pin. Pinning exact tags, rather than floating ones, is the step that matters most here.
- Restart mechanics: Stop the running container, create a new container from the pinned new image, and start it with the same volume and configuration mounts. Recreating the container is the upgrade, so the container's writable layer should hold no node state.
- Configuration and data: Mount the configuration and the data directory as volumes. Keep the node key and data volume attached to the same validator identity so the node does not come back under a new key.
- Image contents: If you build custom images for plugins or JVM options, rebuild them on the JDK version the release requires and retest them before the rollout.
- Rollback: Keep the previous image tag and a backup of the data volume, so a container can be recreated from the older image if the new one fails to open the database.
- Verification: Check the container status, the container logs for block import, and the peer count and signing behavior on the node before the next container is stopped.
Upgrade one validator container at a time and wait for it to rejoin consensus before stopping the next one.
On Kubernetes, each validator runs as its own pod in a StatefulSet, so each pod keeps a stable identity and its own persistent volume. The controller applies the new image pod by pod in a defined order, which fits the one-validator-at-a-time rule. The operator decides how far the rollout proceeds, and the cluster enforces the ordering. Confirm how your controller version handles partitions and disruption budgets before you run it.
- Restart mechanics: Update the image in the StatefulSet and let the controller replace one pod at a time. Use a controlled rollout that pauses after each pod, so the rollout does not continue while a validator is still syncing.
- Configuration and data: Store configuration in ConfigMaps and keys in Secrets, and keep each pod bound to its own PersistentVolumeClaim so the data survives pod replacement.
- Availability guardrails: Use a PodDisruptionBudget sized so voluntary disruptions cannot take the validator set below the two-thirds threshold that QBFT needs.
- Backups: Confirm that volume snapshots or file backups exist for each validator's data before the rollout starts, especially for releases with one-way database changes.
- Verification: Check readiness, the peer count reported by each pod, block import in the pod logs and signing activity before the controller moves to the next ordinal.
If a pod fails its checks, pause the rollout and leave the remaining pods on the previous image.
With Ansible, the upgrade is a playbook that applies the same steps to each validator host. The playbook makes the sequence repeatable across hosts, and the inventory defines which validators are upgraded together. Besu's documentation describes Ansible as one of the supported upgrade methods, so the playbook should follow the same order as a manual upgrade. The control node holds the logic, so the playbook is the record of what happened.
- Restart mechanics: Use a serial setting so the play upgrades a small batch of hosts, then waits for the checks to pass before the next batch starts. Choose the batch size so the remaining validators still hold the two-thirds threshold.
- Configuration and data: Render configuration from templates kept in version control. Leave data directories untouched in the role, and make backups a required task before the binary changes.
- Failure handling: Stop the play on the first failed assertion so a bad host does not pull the others into the same failure.
- Verification: Add assertions for the service state, peer count, recent block import and the validator's signing activity, and fail the play if any of them is missing.
- Dry runs: Run the playbook against a staging network first, and review the check-mode output before any production run.
Keep the playbook output with the change record, so the operator can show which hosts were upgraded, when, and what the checks returned.
How to confirm peering after each validator returns
An upgraded validator is not done until it has reconnected to the other validators and to its bootnodes. A restart can succeed at the process level while the node comes back with no peers, a stale peer list, or connections that the permissioning rules now reject. Before you move to the next node in the upgrade order, confirm that the returning validator has rejoined the peer mesh and is following the chain head. Do this for every node, including the last one in the sequence, because a node that stays isolated still counts against the two-thirds of validators that must remain online.
Permissioning sits directly in this path. Besu's local permissioning uses a node permissions configuration file to restrict connections to known participants, and the node allowlist in that file must still contain the enode identities of the other validators and bootnodes after the upgrade. Check that the upgrade did not change the node keys or the enode URLs in your configuration, because a changed identity will be refused by peers that still allow only the old one. Keep the permissions file under version control so you can compare it with the running copy. Confirm the exact flags and file behavior for your release in the Besu permissioning documentation.
Use metrics and logs to verify the result. Start Besu with --metrics-enabled, which exposes Prometheus metrics on port 9545 by default, and scrape the peer count and sync-related metrics for each validator. Compare the returning node's peer count and block height with the rest of the set, and watch the logs for repeated disconnects or permission rejections. Metric names differ between releases, so confirm the exact peer and sync metric names against the version you run before you write alerts or gating checks. Record the results in the change record for each node.
What makes a Besu validator upgrade take longer
This table lists the factors that stretch a rolling Besu validator upgrade and what each one does to effort. Read each row against your own network, then use the quote column to make sure the partner or internal team prices the same work. The main takeaway is that the validator count matters less than the custom code, host runtime, database size and the number of organizations that must agree on restart windows.
| Driver | Effect on effort | What to ask for in a quote |
|---|---|---|
| Validator count and margin above two-thirds | A larger set gives more room to restart nodes one at a time, but each extra validator is another restart, health check and peering check. A small set leaves little margin, so any unplanned outage can stop block production. | Confirm the number of validators, how many can be offline at once while the network keeps producing blocks, and the restart sequence written for that margin. |
| Custom plugins and rebuilds | Plugins built against an earlier Besu API must be rebuilt, retested and pinned to the target version. Plugin API changes in the 26.x line add work that a stock node does not need. | Ask for a plugin inventory, the build and test steps for each plugin, and whether compatibility testing against the target release is included. |
| JDK 25 host and image changes | Besu 26.6.0 requires JDK 25 to build and run, so hosts, base images, CI pipelines and JVM options must change before or with the upgrade. Each host image that is not updated becomes a blocker at restart. | Ask which hosts, container images and pipelines will be changed, who rebuilds them, and how the JDK change is tested on staging nodes. |
| Database size and backup time | The Bonsai database upgrades automatically on first start, and older versions cannot open it. Backup and restore time grows with data size, and rollback depends on a backup taken before the upgrade. | Ask for current database size per node, the measured backup and restore time, storage capacity for copies, and who verifies restores. |
| Permissioning and key file setup | Node permissioning files and account permissioning rules must match on every validator and be reloaded correctly after the restart. Unencrypted node key files or HSM-backed keys each need a different check during the change. | Ask for a review of permissions files across all nodes, confirmation that node keys are preserved, and the steps for any HSM plugin version pinning. |
| Organizations coordinating restart windows | Each operator controls its own validators and maintenance calendar. Every organization that adds a restart window adds scheduling, approval and communication work, and a late operator can slow the whole sequence. | Ask who approves each window, how change records are kept for each operator, and what the plan says if one operator misses its slot. |
| Staging fidelity | A staging network that mirrors validator count, peering, data size and host runtime catches problems before production. A thin staging setup shifts those problems into the live network, where rollback is harder. | Ask how closely staging matches production, how long a full rehearsal runs, and what is excluded from the rehearsal. |
| Release notes and breaking changes | Each calendar release can remove features or change configuration. Reviewing notes across every release between the current and target version adds time, especially if several releases are skipped. | Ask for a written release note review covering each version between current and target, with a list of required configuration changes. |
Questions buyers ask before upgrading Besu validators
How much does a rolling Besu validator upgrade cost?
The cost depends on the written scope, so no single figure applies. A fixed-scope engagement covers agreed deliverables such as release note review, a staging rehearsal, the upgrade runbook and the rollout across validators. The cost components are the engineering time to review changes, build the runbook, rehearse on a staging network, and carry out the rolling restarts, plus the time your own staff spend on approvals and change windows. Retainers for ongoing upgrades and support are recurring and separate from the one-time work. Ask for the quote to list each deliverable, which environments are covered, and which items are optional.
How long does a rolling Besu upgrade take?
Duration is driven by the number of validators, how many hosts each one runs on, how much staging testing the change needs, and how many approval steps sit in your change process. The sequence itself is slow by design: each validator is stopped, upgraded, restarted and checked before the next one moves, so the network keeps quorum throughout. Database size matters too, because Bonsai upgrades on first start and a backup must finish before the change begins. Ask for a timeline that separates staging, backup, each validator restart and the post-upgrade watch period.
What does the client's team need to provide for the upgrade?
Your team provides access and decisions. Typically this means access to the hosts, container platform or Kubernetes cluster that run each node, the configuration files and keys locations, a staging network that resembles production, and named owners who can approve the change. Someone must also confirm the monitoring and log pipelines that will show peer counts, block production and validator health during the rollout. Keeping node configuration under version control makes this faster, because the team can compare the current settings with what the new release expects.
Can our internal team perform a rolling Besu upgrade alone?
Yes, many teams can, provided they have engineers who understand Besu's configuration, the consensus setup, and the infrastructure that runs the validators. The work is manageable when the team reads the release notes carefully, rehearses on staging, and follows the upgrade guide. The risk comes from skipped steps: a missed breaking change, a configuration key that was removed, or a validator restarted before its peers are healthy. Assess your team against that list honestly. A partner can take the planning, rehearsal and runbook work while your staff keep ownership of the production change.
Can we roll back a Besu node after the Bonsai database upgrade?
Only if you took a backup before the upgrade. According to the Besu 26.10.0 release notes, the Bonsai database upgrades automatically on first start, and older Besu versions cannot open an upgraded database. That makes a verified copy of the node data the rollback path. Your plan should state where the backup is stored, how long a restore takes on your storage, and who decides whether to roll back. Test the restore on staging before production, because an untested backup gives no assurance during an incident.
Does a validator upgrade need a protocol milestone agreed with other operators?
It depends on what changes. A client upgrade that keeps consensus rules and the genesis unchanged can usually be rolled out by each operator on its own schedule, as long as every node stays on a compatible version and peers stay connected. A change that alters protocol behavior, such as a consensus or fork change, needs a block or time agreed across all operators so that every node switches at the same point. Read the release notes to classify the change, then confirm with the other operators in writing before scheduling.
How do we confirm nodes still peer after each validator restarts?
Check peer counts and connection status on each restarted node before moving to the next one, and confirm that the remaining validators still hold quorum. Besu exposes metrics for Prometheus when metrics are enabled, so peer counts and block production can be graphed during the rollout. Also verify that the restarted validator is producing and importing blocks at the same height as the others. Define a clear stop condition, such as a node that fails to peer, so the rollout halts rather than continues with a degraded network.
How DeployOpen scopes and delivers a rolling Besu validator upgrade
DeployOpen plans, implements, tests and documents rolling Besu validator upgrades inside the infrastructure you control, whether that runs on Linux, Docker or Kubernetes. Each engagement starts with a written scope that ties the release notes, the upgrade order and the quorum checks to your own network and change process.
How we scope
- Inventory every validator, boot node and RPC node with its Besu version, JDK version and deployment method.
- Review the release notes of each target Besu version for breaking changes, removed features and database upgrade effects.
- Agree in writing the upgrade order, the minimum validator count to keep online and the acceptance checks for each node.
How we build
- Version your node configuration, permissions files and deployment definitions under version control before the first change.
- Rehearse the full rolling sequence on a staging network that matches your validator count and peering layout.
- Verify peer connections, block production and consensus participation after each node restart before moving to the next.
How we hand over
- Deliver a runbook with the exact upgrade order, rollback steps tied to a verified data backup, and go or no-go criteria.
- Document the peering and quorum checks in a form your operators can repeat during later upgrades without DeployOpen.
- Walk your operations and change teams through the runbook and record the results for your change management files.
How we support
- Support for future Besu releases is available as a retainer, scoped to the versions and environments you agree.
- Retainers can cover release note review, staging rehearsals and execution of each calendar release upgrade.
- Ask any quote to state which items are one-time work and which recur under a retainer.
