Why a private Besu protocol upgrade is a coordination problem before it is a software task
On a private Besu network, a protocol upgrade is agreed by every participant before it is installed. The Besu upgrade documentation is direct on this point: in private networks all participants must agree on protocol upgrades and coordinate them, with the milestone block written into the genesis file. Each node adds that agreed block to its genesis file and restarts before the milestone is reached. Any operator that misses the restart risks a forked network, so the work that decides success is the agreement, the timetable and the restart record, not the binary.
This differs from upgrading one client binary on a network you run alone. When a single team controls every node, it can stop, replace and restart them in an order it chooses. When several organizations share one permissioned network, each operator has its own change board, maintenance windows, hosting provider and sign-off rules, and each node's genesis file has to carry the same milestone. A node that is patched but still holds the old genesis, or that restarts after the milestone, behaves differently from its peers. Before the first node is touched, the consortium should confirm in writing who owns the genesis file, who approves the milestone block, and how each operator will prove its restart happened in time.
The supported path is the one the Besu documentation describes: agree the upgrade with all participants, set the milestone block in every genesis file, restart each node before that block, and verify that all nodes continue on one chain. Confirm with your network operators which Besu release and protocol changes apply, because release timing and the list of changes differ between releases and should be read from the Besu network and protocol upgrades documentation and the release notes for the version you plan to run. Treat that document as the reference for the sequence, and record every decision as it is made, since those records become the evidence a regulator or auditor will ask to see.
Which release changes affect a protocol milestone
This table lists the Besu release changes that touch a coordinated protocol upgrade, what each one means for a synchronized restart, and what each operator should confirm before the milestone block. Read each row against the Besu version your nodes run today and the version you plan to reach at the milestone. Changes that alter the runtime or the database carry the most risk because they can block a node from starting at the agreed block.
| Release change | What it means for a coordinated restart | What each operator must confirm before the milestone |
|---|---|---|
| Calendar cadence (26.7.0, 26.8.0, 26.9.0, 26.10.0 listed) | Besu ships roughly monthly, so a milestone can fall between releases. The target version must be fixed before the genesis change is distributed. | Confirm the exact release tag every node will run at the milestone and that no operator moves to a later release mid-window. |
| JDK 25 required from 26.6.0 | Nodes on 26.6.0 or later need JDK 25 to build and run. A host still on an older JDK will fail to start at the restart. | Confirm the JDK 25 install on every host, container image and plugin build, and test a start on each node type before the restart window. |
| Clique removed in 26.4.0 | Besu can no longer start or produce blocks on pure Clique networks. Any network still on Clique cannot cross this version without a consensus migration. | Confirm the network runs QBFT or IBFT 2.0 and that no node config still references Clique before setting the milestone. |
| Tessera and onchain permissioning removed in 25.6.0 | Nodes that depend on Tessera privacy or smart-contract permissioning cannot run the release. Permissioning must be local files on every node. | Confirm each operator has migrated permissions to local configuration and that no privacy feature is still wired into the node config. |
| EEA and PRIV privacy RPC groups removed in 25.7.0 | Clients or scripts calling EEA or PRIV methods will fail after the upgrade. Restarts can look healthy while integrations break. | Confirm that applications, monitoring checks and operator scripts do not call EEA or PRIV methods, and test them against the target version. |
| Bonsai database auto-upgrades in 26.10.0 | The database upgrades on first start, and older Besu versions cannot open it. A node that fails mid-restart cannot simply be rolled back. | Confirm each operator has a verified backup taken before the restart, a documented rollback path and a rehearsal on a non-validator node. |
What each operator and the coordinator must prepare before the milestone
Every participant needs the same inputs before a protocol milestone can be agreed and executed. The six items below connect because the milestone block depends on an accurate node inventory, and the restart schedule depends on a tested staging run.
Agreed milestone block and change calendar
Agree one milestone block number across all operators before anyone edits a genesis file. The calendar should set the dates for genesis distribution, the restart window for each node, a freeze on unrelated changes, and a go or no-go decision ahead of the milestone. Confirm that every operator can hit each date, since a node that restarts late or not at all risks forking the network.
Inventory of nodes, clients and JDK
List every node the consortium runs, with its role, host, Besu version, and the JDK it runs on. Mixed versions across operators are a common source of confusion during a rollout. Besu 26.6.0 requires JDK 25 to build and run, so check each host and any custom image against that requirement before the milestone date.
Current permissioning files and node keys
Collect each node's local permissions configuration file, covering node permissioning and account permissioning, and confirm that every operator holds the same allowlists the protocol change assumes. Also record where each node private key file sits and who can read it. The key file is not encrypted by default, so access controls around it need to be confirmed in writing.
Backups before any database-changing upgrade
Take a full backup of each node's data directory before an upgrade that changes the database. Bonsai auto-upgrades on first start, and older Besu versions cannot open an upgraded database, so the backup is the only practical rollback. Store copies outside the node host and test a restore on staging before the milestone window opens.
Staging network that mirrors the genesis
Build a staging network with the same consortium genesis settings, validator count, and milestone block as production, using the candidate Besu version. Run the full sequence there first: genesis change, restart, block production, and peering. Staging results only help if the genesis file and node topology match production, so compare hashes of the genesis files before you rely on any test.
Named coordinator and change record template
Name one coordinator with authority to call a go or no-go and to hold the change window. The coordinator keeps a change record template covering each EIP in scope, the agreed milestone block, the genesis version each operator applied, restart times, verification results, and rollback decisions. Regulated operators should confirm with their own compliance and audit teams which fields the record must hold.
Sequence for agreeing, distributing and activating a milestone
This sequence moves a protocol change from release note review to a verified, running network with every operator on the same milestone block. Each step ends with a checkpoint, and the change record is closed only after cross-operator checks pass.
Review release notes and protocol changes
The network coordinator reads the Besu release notes for the target version and lists each EIP or protocol change that alters consensus, genesis handling or database format. Each operator's platform lead confirms which changes affect their build, including the JDK 25 requirement that applies from 26.6.0 and the Clique removal in 26.4.0. The step closes with a written impact list that every operator has acknowledged. Before any Bonsai database upgrade in this release line, confirm that a backup procedure exists, because the Bonsai database auto-upgrades on first start and older versions cannot open it.
Agree the milestone block across operators
Operators jointly choose a future block number at which the new rules activate, and record the agreed value in the change record along with the release version. Every operator confirms the number in writing, because a single operator working from a different value creates a split between nodes. The check is a signed or minuted confirmation from each operator, with the block height reviewed against current chain height so that the milestone is still in the future with room for the restart window. Ask each operator to confirm which genesis file version they will hold.
Distribute the genesis change through the agreed channel
The coordinator issues the updated genesis file, with the milestone block added at the agreed value, through the channel the consortium has approved for controlled distribution. Operators compare the file checksum on receipt against the checksum the coordinator published, so that any copy altered in transit is caught. Only the changed genesis content is distributed, and the file is stored under version control with its checksum. The step is complete when every operator reports a matching checksum and the old file is retained for reference.
Confirm every operator has the file and a restart window
Each operator confirms in writing that the updated genesis file is installed on every node they run and that a restart window is booked before the milestone block. The coordinator tracks each node by name, operator, host and planned restart time, and flags any node without a confirmed window. Operators with validator nodes confirm the order in which their validators will restart, so that the set stays above the quorum needed for block production. The check is a complete node register with no unconfirmed entries. If any operator cannot confirm, the milestone block should be revisited before proceeding.
Restart nodes before the milestone
Each operator adds the agreed milestone block to its genesis configuration, then restarts each node before the milestone block is reached. Restarts follow the agreed order, and the coordinator watches block height across the network during the window. An operator that does not complete its restart before the milestone risks forking the network, so the coordinator treats any missed node as a stop condition rather than a delay to absorb. The checkpoint is a log or dashboard entry on every node showing the upgraded release running with the milestone recorded before the agreed block height.
Verify peering and block production after the milestone
Once the milestone block is passed, the coordinator checks that every node is peered with the expected participants and that blocks are being produced and imported across all operators. Operators compare their chain head and block hashes at a shared height to confirm they agree on the same chain. Metrics and logs from each node are reviewed for rejected blocks or repeated peer disconnects. The checkpoint is a joint sign-off that all nodes report the same chain at a shared height. If the chains diverge, the change is paused and the rollback decision is taken from the backups held from the pre-upgrade step.
Close the change record
The coordinator completes the change record with the release version, the agreed milestone block, the genesis file checksum, the restart times for each node, the verification results and any deviations. Each operator signs off its own entries, and the record is stored where auditors and supervisors can retrieve it. Backups taken before the upgrade are referenced in the record with their location and retention status. The step is complete when the record is approved by all operators and no open items remain, and any follow-up such as plugin compatibility checks is logged as a separate change.
What the coordinator and each operator own
The coordinator runs the upgrade as a shared project across every operator on the network. Besu's upgrade guidance states that in private networks all participants must agree on protocol upgrades and coordinate them, with the milestone block written into the genesis file. The coordinator is the party that turns a set of release notes into one agreed plan that every operator can execute on the same day.
- Milestone agreement: propose the milestone block and record each operator's written acceptance before any node is changed.
- Genesis distribution: issue one signed copy of the genesis change so every operator edits from the same source.
- Change calendar: publish a single schedule of release review, staging tests, restart windows and the milestone itself.
- Release review: confirm which Besu release and which EIPs are in scope, and that every operator is running a version that supports them.
- Exception handling: decide who is contacted, and how, when an operator reports it cannot restart in time.
Check that every operator has acknowledged the same milestone block number and genesis file hash before the change calendar is locked.
Each node operator changes and restarts its own node, and is responsible for having the agreed milestone in place before the network reaches it. Besu's documentation says each node adds the agreed milestone block to its genesis file and restarts before the milestone, and that any participant that does not risks a forked network. The operator's work is small in steps but has no tolerance for skipped ones.
- Genesis edit: add the agreed milestone block to the local genesis file exactly as distributed, and keep the previous file.
- Binary or image change: move the node to the approved Besu release, including any runtime it requires.
- Backup: take a backup of node data before the restart, because some database upgrades cannot be reversed by an older version.
- Restart: restart the node before the milestone, and confirm it starts cleanly with the new configuration.
- Peer checks: confirm the node is connected to the other validators and is following the chain after restart.
Before the milestone, check that each node reports the expected chain configuration and peer count, and record the result against the operator's name.
The security or compliance lead decides whether the change is authorized and makes sure the evidence of what happened is kept. This role does not run the nodes, but it sets the conditions under which each operator may proceed. The lead's approval should be recorded before the coordinator locks the calendar, so that the change record shows who authorized what and when.
- Change approval: review the release notes, the EIP list and the rollback plan, then approve or reject the change in writing.
- Access and key review: confirm that changes to node keys, RPC authentication and permissioning files follow the same approval path.
- Evidence retention: keep the approved change request, genesis files and hashes, backup records, restart logs and peer check results.
- Deviation handling: require a documented decision for any operator that misses a step, including a rollback or a delayed milestone.
- Post-change review: confirm that the network ran past the milestone without divergence and that the records are filed.
Ask your auditor or compliance function which change records they expect, and in what format, before the first upgrade starts.
What makes a coordinated upgrade take longer
This table lists the factors that stretch a coordinated Besu protocol upgrade across consortium operators and shows what each one changes. Read each row as a question to answer for every member before you agree a milestone block. The main takeaway is that the slowest operator's approval cycle and restart windows usually set the pace, not the software change itself.
| Factor | Why it changes effort | How to keep it manageable |
|---|---|---|
| Number of operators and nodes | Every operator must add the agreed milestone block to its genesis file and restart before the milestone, so more participants mean more coordination points. | Agree one named coordinator, a single milestone block and a shared checklist before any operator edits a file. |
| Deployment model | Bare metal, Docker and Kubernetes each need different rollout steps, image changes and restart procedures, and mixed estates need separate runbooks. | Document the genesis change and restart steps per deployment model so each operator follows the same sequence. |
| Custom plugins | Plugins built against the previous release may need rebuilding, retesting and re-signing before the new Besu version can load them. | Ask each operator to list its plugins and the Besu versions they were tested against, and confirm compatibility with the release notes. |
| JDK changes on hosts | Hosts, base images and build pipelines may need a different Java runtime before the new release can start, which can add work outside the Besu team. | Confirm the required JDK version from the release notes early and check it on every host and image in the staging path. |
| Database size and backup time | Backups must be taken and verified before upgrades that change the database format, and larger databases take longer to copy and restore. | Measure backup and restore time on a staging copy of a production-sized node and plan restart windows around it. |
| Change approval cycles | Each regulated member may need its own change record, risk assessment and sign-off, and those cycles rarely run in parallel with other operators. | Ask each member for its approval calendar and required evidence at the start, and set the milestone block only after the slowest approval is known. |
| Restart windows per operator | A milestone only works if every node restarts in time, so the number of windows each operator can offer limits how many nodes can move before the milestone. | Ask each operator for its available windows in writing, and keep enough validators online during each restart to preserve consensus. |
| Staging and rollback evidence | Testing the upgrade on a network that mirrors production and proving a backup can be restored both add steps that each operator must repeat. | Share one staging test plan and one rollback procedure so each operator can reuse the same evidence in its own change record. |
Questions operators ask before coordinating a Besu upgrade
Who pays for upgrade planning and who covers the internal effort?
Planning work is usually a fixed-scope engagement, while internal effort stays with each operator's own staff. Each operator pays its own engineers to review release notes, test staging nodes, update genesis files and perform restarts on its nodes. A partner's fee covers the coordination work: reviewing the EIPs in a protocol upgrade, drafting the milestone plan, preparing the genesis change set, and writing the runbook and change records. Ask any quote to split these components by party, so you can see which costs recur and which are one-time. The main drivers are the number of operators, the number of nodes each one runs, the deployment model (Linux, Docker or Kubernetes), and whether custom plugins or an HSM must be retested.
What drives the price of a fixed-scope upgrade quote?
The written scope sets the price, so the quote follows from the scope rather than a fixed rate card. The largest drivers are the number of participating organizations, the number of validator and non-validator nodes, the release gap being crossed, and whether the upgrade touches the database, consensus settings or plugins. Scope also grows with the number of environments that need staging rehearsals and with any regulator-facing evidence you need produced. DeployOpen does not publish a price list. Ask for deliverables, assumptions about your environment, and what the quote excludes, such as work by your own change board.
Can a consortium coordinate a Besu upgrade without outside help?
Yes, a consortium can coordinate an upgrade with its own staff, provided every operator has engineers who understand Besu and who can follow the same plan. The Besu documentation states that in private networks all participants must agree on protocol upgrades and coordinate them, with the milestone block specified in the genesis file. The hard parts are agreement, timing and evidence: getting every operator to review the same release notes, choosing a milestone that everyone can meet, and keeping records that a regulator can follow. Outside help is most useful when operators run different tooling, have uneven skills, or lack a neutral person to hold the schedule.
How long does a Besu protocol upgrade take to coordinate?
No duration is given here because the timeline depends on the agreed scope, the number of operators, and how quickly each one can complete its own change approvals. Work usually runs in phases: release review, staging rehearsal, milestone agreement, genesis distribution, node restarts, and post-milestone checks. Your operators can shorten the calendar by preparing change approvals in parallel and by agreeing the milestone early enough to allow a full staging run. Ask for a written schedule with dependencies for each operator before any work starts.
What does each operator need to provide for a coordinated upgrade?
Each operator needs to supply named engineers, access to its nodes and deployment tooling, a staging environment that mirrors production, and a decision-maker who can approve the change inside its own organization. Operators also need to confirm which Besu version and JDK their hosts run, whether they run custom plugins or an HSM plugin, and whether they hold a backup they can restore. Each operator must agree to the milestone block, add it to its genesis file and restart before the milestone. Confirm these requirements in writing with each operator before the plan is final.
What happens if one operator misses the milestone block?
If one operator misses the milestone, its nodes can follow a different set of rules from the rest of the network, which can split the chain. The Besu documentation says each node must add the agreed milestone block to its genesis file and restart before the milestone, and that any participant that does not risks a forked network. Recovery depends on the state of each node, so the plan should define who decides on rollback, how nodes are stopped and restored from backup, and how the forked nodes are brought back onto the agreed chain. Test this path in staging before the milestone, and record who approved each decision.
Which change records do regulators commonly ask to see for a network upgrade?
Regulators and internal audit teams usually ask for evidence that the change was planned, approved, tested, executed and reviewed. Common records include the scope and risk assessment, the list of release notes and EIPs reviewed, the agreed milestone block and genesis change set with its approvals, staging test results, the rollback plan with proof of a verified backup, the restart log for each node, and a post-milestone verification report. Your own compliance team must decide which records your supervisor expects. DeployOpen prepares systems, controls and evidence for compliance assessments but does not issue audit reports, attestations or certifications, so confirm the evidence requirements with your assessors early.
Which change records a regulated consortium should keep
A regulated consortium should keep a change record for every milestone, and the record should show who agreed the milestone block, when that agreement was made, and what each operator did in response. The core entry is the signed decision naming the milestone block number, the protocol changes it activates, and the operators who approved it. Attach to it the release notes reviewed for the Besu version in use and for the target version, with the date each was read and by whom. Without this entry, an examiner has no way to tell whether the consortium agreed to the change or one operator acted alone.
The genesis file is the artifact most likely to be questioned after the fact, so record each version distributed to operators along with its hash, the date it was sent, and the recipient list. Each operator should then record the time it added the milestone block to its own genesis file, the time its node was restarted, and the restart log excerpt showing the node came up on the new configuration. Keep the backup record for each node as well: where the backup was stored, when it was taken, and whether a restore was tested. For Bonsai database upgrades, the backup is the only rollback path, so its record carries real weight. Confirm with your regulator and auditor how long these records must be retained and in what format they must be produced.
Close each milestone with post-milestone checks: peer counts per node, confirmation that every validator is producing and importing blocks past the milestone, and a comparison of block hashes across operators at a fixed height. Record any node that missed the restart window, any delayed peering, and any deviation from the plan as a named exception with its cause, the remediation, and the approver. DeployOpen can prepare the templates, collect the evidence, and organize it for review, but it does not issue audit reports or attestations, so ask your auditor which evidence they will sample and how they expect exceptions to be presented.
How DeployOpen scopes and runs a consortium protocol upgrade
DeployOpen approaches coordinated protocol upgrades on a private Besu network as a written, operator-wide change, not a single-node task. We scope the upgrade with every participating operator, build and test the change in a staging network, and hand over the runbooks and change records your governance and auditors will ask for. Engagements are fixed-scope, with ongoing support available as a retainer.
How we scope
- Written scope lists every operator, node and validator affected by the protocol upgrade and its milestone block.
- We review the EIPs included in the target release and record which ones change behavior on your network.
- Scope names who approves the milestone block, the genesis change and the restart window for each operator.
How we build
- We reproduce the planned genesis change and milestone block on a staging network that mirrors your validator set.
- Test plans check that every node restarts before the milestone and that peering and block production continue afterward.
- Build records capture genesis file versions, node configuration, client versions and the rollback path for each operator.
How we hand over
- Runbooks give each operator the exact genesis edit, restart order, verification commands and escalation contacts.
- Change records document approvals, test results, timestamps and the evidence your compliance and audit teams request.
- Handover sessions walk operator teams through the upgrade sequence, with open questions logged against named owners.
How we support
- Ongoing support covers follow-up releases, upgrade planning for the next calendar release and operator onboarding changes.
- Retainers can include recurring staging tests of upcoming Besu releases before they reach your production network.
- Support scope and response arrangements are confirmed in writing before any retainer work begins.
