Why Hyperledger Besu high availability starts with the validator set
The first availability decision for a permissioned Besu network is the validator set, because the set of nodes allowed to seal blocks determines what can fail and still leave the chain producing blocks. Besu does not provide high availability as a product feature. Availability comes from how validators, RPC nodes, node keys and data are placed across sites and operators, and from how each of those is recovered when something fails. Teams should treat the validator set as a governance and consensus decision before it is an infrastructure decision. Besu supports QBFT and IBFT 2.0 as its proof-of-authority options for private networks. Clique is no longer supported: Besu 26.4.0 removed it, so a network that still depends on Clique cannot be moved onto current releases without a planned migration.
The validator set is hard to change later because membership is part of consensus. Adding or removing a validator is a network decision that the participants agree on, not a server change an operator makes alone. Before go-live, ask for written answers to a few questions: which organizations hold validators, how many sites host them, who can propose or vote a validator in or out, and how the group responds when a validator is lost for an extended period. Confirm the fault tolerance arithmetic for your chosen consensus option in the current Besu documentation before you fix the validator count. The count you choose now sets how many simultaneous failures the network can absorb, so it should be tested against your site and operator plan, not assumed.
Three details make this harder than it looks. First, the local node private key file is not encrypted by default. Anyone who copies that file from a validator host can impersonate the validator, so key custody must be designed up front. Deployments that need hardware-backed storage can use a security module plugin, such as the Besu HSM plugin through the --security-module option, and that choice is much easier to make at design time than to retrofit on a live validator. Second, the Bonsai database upgrades automatically on first start, and older Besu versions cannot open an upgraded database. A backup taken before each upgrade is the only rollback path, so the backup must be verified by restoring it on a copy before any production upgrade.
Turn these points into a short set of decisions. Map each validator to a site and an operator, and record where its key is stored and who can access it. Keep RPC nodes separate from validators so that application traffic and API exposure do not run on the hosts that seal blocks. Write a recovery runbook that states which node data can be rebuilt from peers and which must come from a verified backup. Enable metrics with the --metrics-enabled option (default endpoint port 9545) so that a struggling validator is visible before the network loses the agreement it needs to keep producing blocks. Confirm each of these against the Besu version you plan to run, because the release cadence is roughly monthly and details change between versions.
Which Besu components carry availability
Each component below carries a different part of availability, from consensus and application access to connectivity, identity, stored state and failure detection. Read them together, because a node can look healthy on its own and still fail the network if its peers, keys or data are not in order.
Validator nodes
Validator nodes sign and propose blocks under QBFT or IBFT 2.0, so they carry the consensus weight of the network. If enough validators go offline, block production stops, and a validator that stays online but falls out of step can still disrupt finality. Confirm the validator count and fault tolerance in your sizing before go-live, and check whether your design requires each validator to run on separate infrastructure or with a separate operator.
RPC nodes outside the validator set
RPC nodes serve application reads and writes, and they should run outside the validator set. Keeping them separate reduces the risk that heavy application traffic or a misbehaving client affects block signing. If an RPC node fails, applications lose access while the chain keeps running, so place RPC nodes behind a load balancer across more than one node. Confirm how each application handles endpoint failover.
Bootnodes, static peers and permissions
Bootnodes and static peer lists define which nodes find each other at startup, and the local permissions configuration file restricts connections to known participants. A wrong or missing peer entry can leave a node running but isolated from the network. If the permissions file is too strict, validators or RPC nodes may be unable to reconnect after a restart. Keep both files under version control and test reconnection after every change.
Node private key file
The node private key file gives each node its network identity. Besu documentation states that this file is not encrypted, so anyone who copies it can impersonate the node. If the file is lost, the node comes back under a new identity, and peer lists and permissions must be updated to match. Confirm where each key is stored, who can read it, and whether a security module plugin such as the Besu HSM plugin fits your key custody model.
Data directory and Bonsai database
The data directory holds chain state in the Bonsai database. If it is lost or corrupted, the node must resync from peers before it can serve or sign again. The Besu 26.10.0 release notes state that the Bonsai database auto-upgrades on first start and that older Besu versions cannot open an upgraded database, so a backup taken before any upgrade is the only practical rollback path. Confirm backup frequency and test the restore procedure on a non-production node.
Metrics endpoint for failure detection
The metrics endpoint, enabled with the --metrics-enabled option and served on port 9545 by default, lets Prometheus detect a failing node. Without it, a node that stops producing blocks or falls behind may go unnoticed until applications report errors. Confirm alerts on peer count, block production and sync status, and make sure the scrape job reaches every validator and RPC node.
Decisions that set the availability ceiling
This table compares the five choices that set how much failure a Besu QBFT network can absorb. Read each row as a decision: the option, when it fits, and what it costs your operations team over time. The main takeaway is that more validators and more sites raise fault tolerance only if operators, keys and upgrade routines can support them.
| Decision | Option | When it fits | Operational cost |
|---|---|---|---|
| Validator count | Small fixed set (for example, four validators) sized to the faults you must survive | Single-operator networks where losing one validator is acceptable and tolerance needs are modest | Lower node count, but a second failure can halt block production; confirm the QBFT fault threshold with your engineers before choosing |
| Validator count | Larger set sized for a higher fault tolerance target | Consortia or regulated networks that must keep producing blocks through several simultaneous outages | More nodes to patch, monitor and key-manage; every validator change goes through on-chain voting, so set size growth is slow to adjust |
| Site placement | All validators in one data center or cloud region | Lowest latency and simplest operations during a pilot or non-production environment | A site outage stops the network; this is the main ceiling on availability, regardless of validator count |
| Site placement | Validators spread across two or more sites or regions | Production networks where a single facility or region loss must not halt consensus | Cross-site latency affects block propagation and timeouts; you must test timeout settings and network links under load before go-live |
| Operator model | One operator runs all validators | One accountable team, fast coordinated upgrades and key rotation | Operator failure or staffing gaps affect the whole network; trust in one party may not satisfy consortium members |
| Operator model | Several organizations each run one or more validators | Consortia where no single member should control block production | Each operator must agree on Besu release windows, JDK version, database upgrade timing and key rotation dates, which adds coordination time and governance work |
| RPC separation | Dedicated RPC nodes that do not validate | Applications and integrations need API load that should not disturb consensus | Extra nodes to size, monitor and secure; JSON-RPC authentication and permissioning must be configured on each RPC node |
| Key custody | Node key in an encrypted-at-rest file on a protected host | Lower-risk environments or early phases where hardware is not yet budgeted | Besu documents that the node private key file is not encrypted, so host hardening and access control carry the security burden |
| Key custody | Security module plugin such as the Besu HSM plugin with the --security-module option | Production validators where keys must be hardware-backed and access must be auditable | Hardware, HSM licensing and support, plugin testing across Besu releases; confirm the plugin version matches your Besu release |
How to build a highly available Besu validator set
This sequence takes a permissioned Besu network from a consensus decision to a validator set that keeps producing blocks when a node, site or operator fails. Each step ends with a check you can record, and the final gate is a restored node that rejoins its peers while the chain keeps producing blocks.
Confirm consensus and size the validator set
Confirm whether the network will use QBFT or IBFT 2.0, then read the fault-tolerance rule for your validator count in the Besu private networks documentation. Decide how many validator failures the network must survive and which scenarios count, such as one node lost, one site down, or one operator unavailable during a planned upgrade. Owner: the platform architect with the consortium's technical lead. Closing check: a design record that names the consensus, the validator count, the rule it satisfies and the Besu release you will target, checked against the current release notes, including the JDK requirement.
Assign validators to sites and operators
Place each validator on a site and with an operator so that the failure scenarios from the previous step do not take the network below its fault-tolerance threshold. Avoid concentrating enough validators in one data center, cloud account or operator team to halt the chain if that domain fails. Each operator confirms in writing which nodes it runs, who holds on-call responsibility, and how it reaches the other operators during an incident. Closing check: a placement map signed by each operator, tested against every scenario with the number of validators remaining after each failure.
Generate and custody node keys first
Generate each node key in a controlled environment before any node starts, and record the public identity of every node in the design record. The Besu documentation notes that the node private key file is not encrypted, so decide now whether keys stay on disk under strict file permissions or move to hardware-backed storage through a security module plugin, such as the Besu HSM plugin enabled with the --security-module option. Assign a named custodian for each key and for each backup copy. Closing check: a key inventory that matches the placement map, with custody owners recorded and no key existing outside that record.
Set up RPC nodes outside the validator set
Deploy separate RPC nodes that sync from the network but are not in the validator set, so application traffic and API load never touch validator hosts. Configure JSON-RPC authentication on these nodes using a credentials file or JWTs validated against a public key, and terminate HTTPS in front of them, since Besu does not terminate HTTPS natively. Restrict validator hosts to connections from peers and approved operator networks. Closing check: each RPC node reaches the same head as the validators, and validator hosts accept connections only from peers and approved operator networks.
Configure static peers and permissions
Set static peers on each node so connections are predictable, then write the node permissions file so only known participants can connect. Use account permissioning only where the consortium needs onboarding rules or suspension controls. Keep the permissions file under version control with change approval from every operator, because local permissioning is set per node and each operator must apply the same allowlist. Closing check: an allowlisted node connects on the live network, and a node outside the list is refused.
Enable metrics before go-live
Enable metrics on every validator and RPC node with the --metrics-enabled option, which exposes an endpoint for Prometheus on port 9545 by default. Point your scrape configuration at each node and build alerts on the signals that matter for a permissioned network: peer count, block production and sync status. Use the sample Grafana dashboard as a starting point and adjust it to your environment. Closing check: every node appears as a target in Prometheus, and a test alert fires when one node is stopped in a non-production environment.
Restore a full backup onto a separate host
Restore a full backup of node data and keys onto a separate host that is not running a live validator, then start it as a non-validating test node. Run this rehearsal in a separate environment so two running nodes never share one validator key. Take a backup before every upgrade as well: the Besu 26.10.0 release notes state that the Bonsai database upgrades automatically on first start and older versions cannot open it, so rollback depends on that backup. Closing check: the restored node syncs to the same head as its peers, and its key matches the inventory.
Start the network and confirm block production
Start validators in the order set by the placement map, then the RPC nodes, and watch peer counts rise as static peers and permissions take effect. Confirm that blocks are produced and that validators agree on the head. Then stop one validator in a controlled test and check that the chain keeps producing blocks within the fault tolerance you designed for. Record the test results, the Besu release and the JDK version in the runbook. Owner: the platform team with each operator on call. Closing check: a block production log across all validators for the test window, showing the chain continued after the controlled stop and the stopped validator rejoined after restart.
How to test that the network survives a failure
Prove failover before go-live by running a scripted set of failure tests against a staging copy of the production topology, and record the expected and observed result of each one. The first test stops one validator at a time. While the set is within its tolerated fault count, blocks should keep being produced and every surviving validator should keep importing them. Confirm this by watching block height on at least two surviving nodes, including nodes in different hosting locations, so you do not rely on the node you stopped to report its own health. Restart each validator afterward and confirm it catches up. Repeat this for every validator, because a host with a stale peer list or a misconfigured clock can pass one test and fail another.
Simulate the loss of a site by stopping every validator in that site at the same time. This test checks your placement decisions, not just the software. Before running it, work out on paper how many validators sit in each site and confirm that the survivors still hold enough validators for the design to keep producing blocks. If the test passes, expect block production to continue with the remaining set. If it fails, you have found a design fault, and the result should go back into the validator placement review before go-live. Also check which RPC endpoints your applications use in the failed site, since application access is a separate dependency from consensus and can fail even when the chain keeps running.
Restore a validator from backup onto a clean host to test the recovery path that teams most often leave untested. Build the new host using the same steps you would follow during a real incident, restore the data directory and the node key file through your approved secure channel, and start Besu. The node should connect to its peers and sync to the chain head. Check whether the restored node uses the same identity as the original, and confirm that the original host is powered off or isolated before the restored node joins, because two live nodes with the same key is a fault you must rule out. For RPC, stop one RPC node and confirm that applications reconnect to another endpoint, including any JSON-RPC credentials or tokens they present, without manual changes on their side.
The most important test is losing more validators than the design tolerates. In that state, QBFT cannot gather the agreement it needs, so the network stalls and stops producing blocks. A stall halts the chain but does not by itself create a fork, and the priority is to keep it that way. Do not start replacement validators with keys that are already in use, do not change genesis or validator membership on only part of the set, and do not let separate groups of validators continue on their own copies of the chain. Recover by bringing the missing validators back under their original identities until quorum returns, then confirm that every node reports the same block hashes at the same heights. Repeat the full test set after every Besu upgrade, because release notes can change consensus behavior, peer handling and database format.
How the design changes by deployment model
All validators run in one cloud account or data center. This is the simplest model to build and operate, but availability is bounded by one failure domain. A power, network, or control plane event that affects the site can stop block production for the whole validator set. Spreading validators across availability zones or racks inside the site reduces that exposure, but it does not remove the shared risk.
- Availability trade-off: Site-level events stop the network. Zone or rack separation narrows the blast radius but keeps one operator and one site.
- Key handling: Node key files are not encrypted by default, so file permissions, storage locations and backup copies of each validator key need explicit controls. An HSM security module plugin is an option where hardware-backed storage is required.
- Connectivity: Validators peer on internal networks. Keep RPC nodes on a separate subnet so only validators and approved clients can reach the peer-to-peer port.
- Permissioning: One team maintains the local permissions files and applies changes in a single change window, which keeps the small validator set easy to reconcile.
- Backups: Keep snapshots and key copies in a separate account or facility so one incident cannot remove both live data and recovery material.
Confirm which zones or racks your site actually separates, and whether a single cloud control plane could affect every validator at once.
Both sites are run by the same team. The trade-off is resilience to the loss of one site against the added complexity of cross-site traffic, split key custody and synchronized administration. Latency and packet loss between sites affect block timing and validator health, so the split has to be designed around measured link quality, not assumptions. Choose the validator distribution so that the surviving site still holds enough validators to keep the network live after a site loss. Confirm the exact threshold for your validator count with the Besu QBFT documentation and your own testing.
- Availability trade-off: A site loss removes part of the validator set. Test whether the surviving validators can still reach finality under the failure scenarios you plan for.
- Key handling: Each validator's key stays at the site that hosts it. Document how a validator is restored at the other site, and who authorizes that restore.
- Connectivity: Use dedicated, monitored links between sites. Track latency, jitter and packet loss, and rehearse partition scenarios.
- Permissioning: One administrative body controls the permissions files at both sites. Apply changes in a fixed order, verify them at each site, and record the version deployed at each one.
- Backups: Replicate backups to the second site and keep at least one copy outside both sites. Measure how long a restore takes from each location.
Confirm the measured latency between sites and the failure domains each site truly has before you commit to a validator split.
Each member runs its own validators under its own operations. Availability depends on each member staying online and on shared agreements about how the network is changed and recovered. No single operator controls recovery, so governance and key custody become the hardest parts of the design. A member outage affects the network only to the extent that it removes validators the quorum needs, which is why the validator distribution should be agreed in advance.
- Availability trade-off: Uptime depends on each member's operations and response times. Agree on minimum service expectations and escalation paths in writing.
- Key handling: Each member generates and holds its own keys. Agree on key generation, rotation and the steps required to bring a replacement validator into the set.
- Connectivity: Peer-to-peer endpoints cross organizational boundaries. Agree on firewall rules, allowlists, bootnode ownership and how changes to endpoints are announced.
- Permissioning: Besu removed onchain permissioning in version 25.6.0, so consortium rules live in local permissions files that each member applies. Changes need a multi-party approval process and a shared record of the file versions in force.
- Backups: Each member backs up its own node data and key material. Agree on a joint recovery runbook, backup retention, and how a new node can be seeded from peer data.
Confirm each member's operating model, response times and key custody arrangements before the validator distribution is finalized.
An air-gapped network has no outbound connectivity, so software, configuration, monitoring exports and backups all move by controlled transfer. Availability depends on the internal topology and on how quickly the team can recover from hardware or media failures without external services. There is no remote help path during an incident, so spares, local runbooks and tested restores carry more weight than in connected models.
- Availability trade-off: The network does not depend on outside services, but recovery depends entirely on local spares, skills and documented procedures.
- Key handling: Keys stay inside the enclave. Hardware-backed storage, such as an HSM security module plugin, matters more here because remote recovery is not available.
- Connectivity: Only internal peering is allowed. Besu requires JDK 25 to build and run, so the JDK, Besu binaries and plugin artifacts must pass through a verified transfer process before they are installed.
- Permissioning: Permissions files are managed locally and changed through the internal change process. Every change needs a recorded version, approval and a verified copy on each node.
- Backups: Write backups to media held inside the enclave and test restores there. Take a backup before any upgrade, because the Bonsai database upgrades automatically on first start and older Besu versions cannot open an upgraded database.
Confirm your transfer, media and change-control procedures with your security team before you finalize the recovery design.
What drives the effort and cost of high availability
This table lists the factors that move the work and cost of a highly available Besu design, with a way to keep each one down. Read it before scoping, because most drivers are set by decisions made in the first weeks, not during the build. The main takeaway is that agreeing topology, custody and restore procedures early costs far less than changing them after validators are running.
| Driver | Why it changes effort | How to keep it down |
|---|---|---|
| Validator count | Each QBFT validator adds a node to build, key to custody, peer relationship and monitoring target, and fault tolerance depends on how many can fail at once. | Size the set to the fault tolerance the consortium agrees, and confirm the count with every operator before build. |
| Number of sites and operators | Each site adds network paths, firewall rules, time synchronization and a party who must answer for their nodes during an incident. | Limit sites to the failure domains that matter, and write down who operates which validator and who approves changes. |
| Key custody approach | Node keys sit in unencrypted files by default, so hardware-backed storage through a security module plugin adds procurement, integration and ceremony work. | Agree custody model with each operator before build, including who holds backups and how key loss is recovered. |
| Number of environments | Development, test and production each need the same validator and RPC topology to be useful, so every extra environment repeats the build and the operations. | Keep fewer environments and run the same topology in each, so tests reflect production behavior. |
| Node data volume and restore time | Bonsai data grows with chain history, and a restored node must catch up before it can serve or validate, which affects how long a site runs with reduced capacity. | Take backups on a schedule, keep a documented restore runbook, and rehearse a restore on a non-production node. |
| RPC load and consuming applications | Each application calling JSON-RPC adds request volume, authentication requirements and separate RPC nodes that must stay in sync with validators. | Separate RPC nodes from validators, and list the consuming applications and their call patterns before sizing. |
| Monitoring and alerting coverage | Besu metrics must be scraped, dashboards built and alerts tuned for peer count, block production and sync status at every node and site. | Start with a fixed set of alerts agreed with the operators, and extend coverage only when an incident shows a gap. |
| Compliance evidence requirements | Assessors may ask for key custody records, change logs, backup test results and access reviews, which must be produced from the running system. | Define the evidence list at scoping and build collection into the runbooks, so records are made during operations, not reconstructed later. |
| Upgrade cadence across sites | Besu ships roughly monthly, and the Bonsai database upgrades on first start, so each site must be rolled in the same planned order with a backup taken first. | Choose a release policy and rollout order up front, and confirm the rollback path with each operator. |
Protecting keys, peers and recovery copies
Several security decisions directly determine whether a validator stays online and can recover. Key files, peer admission, RPC exposure and backups each create a failure mode that high availability design does not solve on its own. The cards below cover each control and point to the sibling topics for the detailed configuration.
Node private key file protection
Besu documents that the local node private key file is not encrypted. A copied or altered key file can let someone impersonate a validator, and a lost key file means rebuilding that node's identity. Decide file ownership and permissions for the key path, restrict read access to the Besu service account, and confirm the key location is excluded from shared volumes and logs.
Security module for hardware-backed keys
Where policy requires keys to stay in hardware, evaluate a security module plugin such as the Besu HSM plugin, enabled with the --security-module option. Confirm the plugin version matches your Besu release, that the HSM vendor is supported with it, and who operates the HSM for failover and key recovery. Add the HSM's own availability to the validator design, since a validator that cannot sign blocks is down.
Host isolation for validators
Run validators on hosts that accept only the traffic they need: peer connections from other validators and their sentries, plus administrative access from a bastion. Restrict outbound destinations as well, so a compromised host cannot reach arbitrary networks. Confirm which host firewall, cloud security group or network policy enforces this, and test it after each infrastructure change.
Peer admission with node permissioning
Use the permissions configuration to let validators accept connections only from known peers. Each node lists the enode identities it trusts, so an unknown node cannot join the peer set or disrupt gossip. Plan how new validators or replacement nodes are added to every node's list before a failover, because a missing entry can isolate a healthy node. The sibling security hardening topic covers the full permissioning model.
Keeping RPC off the validator network
Serve JSON-RPC and WebSocket traffic from RPC nodes on a separate network segment, so application load and API abuse cannot starve validator peering or block production. Validators should expose no public API. Confirm the RPC tier can be scaled or replaced without touching validator configuration, and review which methods each exposed endpoint allows. Authentication for these endpoints is covered in the JSON-RPC authentication topic.
Encrypted and controlled backups
Backups of node data and key material must be encrypted at rest and readable only by named recovery roles. A backup without its matching key can be useless, and an unprotected key copy becomes a security incident. Record where each backup is stored, who can restore it, and how a restore is tested. Take a backup before any Besu upgrade, because newer releases can auto-upgrade the Bonsai database on first start and older versions cannot reopen it.
Backing up node data, keys and upgrades without losing a validator
The backup plan for a Besu validator should name every item that cannot be rebuilt from the network, decide how often each one changes, and assign an owner to each. The node private key is the most important item. Besu documentation states that the local node private key file is not encrypted, so the backup copies need the same access controls as the live key, and custody should be recorded in a register that shows where each copy sits and who can open it. Beyond the key, back up the permissions configuration files that govern node and account permissioning, the genesis file, the node configuration, and the data directory. Genesis and configuration files change rarely, so a backup taken at each change is enough. The data directory changes constantly, so decide how far back you can accept losing it and how long a resync would take on your hardware before you set its schedule.
A restored validator must never run in two places at once with the same key. If a copy starts while the original is still running, two signers hold one identity, and the network may see conflicting behavior from that identity. The restore runbook therefore has to begin with confirming the original is stopped, not with starting the copy. Write the checks down: the process has exited on the original host, the node is no longer appearing as a peer or validator in monitoring, and the operator on call has confirmed the change in the incident log. Only then should the key be moved or copied to the replacement host. Test this sequence in a non-production network before go-live, and confirm with your consensus and key management procedures how a stopped validator is detected and how a stale key is withdrawn from service.
The Besu 26.10.0 release notes state that the Bonsai database upgrades automatically on first start, and that older Besu versions cannot open an upgraded database. This changes rollback. Once a node has started on 26.10.0 with Bonsai, reverting to an earlier binary against that data directory will fail, so the only route back is a backup taken before the upgrade. Take that backup before every version change, not only major ones, and store it where a rollback can reach it without relying on the failing node. Confirm that your restore test works with the exact older version you would roll back to, because a backup that opens only with the newer release gives no protection. The same discipline applies to the JDK requirements and validator sequencing covered in the upgrade topic, which should be followed for each change.
Backups protect data but do not tell an on-call engineer how to rebuild the network. Keep a current topology record that lists each validator and RPC node, the site and operator responsible for it, the host and key custody location, and the monitoring endpoints. Add an operator contact list with primary and backup names, escalation paths and the decision maker who can declare a validator lost. Write the recovery steps as a numbered handover document that a second team could follow without the original authors, and review it after each validator change. Store these records outside the nodes they describe so they remain available when a site is down. For day-to-day procedures after a restore, such as validator voting, monitoring thresholds and stalled-network handling, follow the sibling topics on QBFT validator operations and monitoring.
Questions teams ask before building a highly available Besu network
How many validators does a QBFT network need to tolerate failures?
Start with the number of validator failures the network must survive, then size the validator set so the remaining online validators can still reach QBFT agreement. The threshold follows from the consensus protocol, and Besu implements QBFT as proof-of-authority consensus for private networks. Model each candidate validator count against the failure scenarios you care about, including losing a site, an operator or a cloud account. Where to confirm: the Besu private network documentation at docs.besu-eth.org, the QBFT specification your team adopts, and a failure test on your own validator set before go-live. Treat the result as a design input that you verify, not a number to assume.
Does Besu provide high availability automatically?
No. Besu is an execution client, and availability comes from how you design and operate the network. That includes validator placement across independent infrastructure, separate RPC nodes, monitoring, backups of node data and keys, and recovery procedures that your team has tested. Besu gives you useful inputs for this work, such as Prometheus metrics enabled with the metrics option (default endpoint port 9545) and a plugin framework. Those features help you detect and recover from problems, but they do not decide your fault tolerance. The design and the runbooks do.
Are two sites enough for a highly available Besu network?
Two sites are often not enough by themselves, because the loss of one site can remove a large share of validators at once. The right answer depends on how many validators sit at each site and how many failures the network must survive. Plan the placement so that no single site, cloud account or operator holds enough validators to stop block production if it fails. Test that assumption by simulating the loss of each site before you rely on it. Your architecture review should confirm the site count, since the placement rule matters more than the number of sites alone.
What does the client team need to provide for this work?
The work needs decisions and access from your side, starting with infrastructure access in your cloud accounts, data centers or air-gapped networks. You also need a key custody decision. The Besu node private key file is not encrypted, so you must decide whether to protect it with a hardware-backed security module plugin, such as the Besu HSM plugin enabled with the security module option, or with another controlled approach. Consortium members also need operator agreements covering who runs each validator, who approves changes and how incidents are escalated. Expect to provide network routing decisions, monitoring access and named approvers for change windows.
How long does a highly available Besu build take, and what drives it?
Scope and timelines are agreed in writing before work starts, so the duration comes from the signed scope rather than a standard estimate. The main drivers are the number of validators, sites and operators, the key custody approach and any hardware procurement it requires, and the number of environments you need before production. Partner agreements and approval cycles often take longer than the technical build. Your Besu version also matters: Besu ships roughly monthly, and the 26.6.0 release notes state that Besu requires JDK 25 to build and run, so confirm your platform baseline early. The implementation plan and cost topic describes how these drivers affect budgeting.
Can an internal team build a highly available Besu network alone?
Yes, an internal team can build it alone if it already has engineers who understand Besu, QBFT consensus, Linux and Java operations, networking, key management and observability. Most gaps appear in three places: failure testing that covers site and operator loss, key custody decisions, and recovery drills that restore node data and keys from backup. Besu also changes often, so someone on the team must keep track of monthly releases and their breaking changes. Many teams take a hybrid route, using an outside partner for design review and recovery testing while their staff run day-to-day operations. Decide based on which of these skills you can staff and sustain.
What does a fixed-scope DeployOpen engagement cover for a highly available Besu network?
A fixed-scope engagement covers the work listed in a written scope, with deliverables and timelines agreed before work starts. For high availability, that usually includes validator layout across sites and operators, separation of RPC nodes from validators, backup and restore procedures, monitoring, runbooks and handover to your team. DeployOpen plans, implements, hardens, backs up, documents and supports self-hosted open-source platforms inside your own infrastructure. It prepares systems and evidence for compliance assessments but does not issue audit reports or certifications, since independent auditors and certification bodies do that. DeployOpen does not publish a price list, so a quote follows from the written scope. Ongoing support is available as a retainer.
How DeployOpen scopes, builds and hands over Besu high availability
DeployOpen scopes Hyperledger Besu high availability as a fixed-scope engagement, with the validator layout, site and operator spread, RPC separation and backup plan agreed in writing before build starts. Work runs inside your own infrastructure, and each step ends with documented evidence your team can review. Ongoing support is available as a retainer after go-live.
How we scope
- Written scope lists which validators, RPC nodes and sites are in and out of the engagement.
- Scope records the QBFT validator count, operator split and site placement your fault tolerance target requires.
- Discovery confirms node key custody, backup targets and recovery time expectations before any design is approved.
How we build
- Design separates validator nodes from RPC nodes so application traffic never reaches signing validators directly.
- Node key storage uses an HSM security module plugin where needed, because the local key file is not encrypted.
- Prometheus metrics and RPC authentication are enabled and tested on every node before go-live sign-off.
How we hand over
- Runbooks cover validator restart, failover between sites, and rebuilding a node from backup.
- Backup and restore test results are recorded, including a restore into a non-production node.
- Upgrade procedure requires a backup before Besu version changes, because upgraded Bonsai databases cannot be opened by older versions.
How we support
- Ongoing support is available as a retainer covering monitoring review, patching and Besu release reviews.
- Confirm with your team who owns validator availability alerts and how on-call handoffs will work.
- Retainer scope and response expectations are written into the agreement before support starts.
