Hyperledger Fabric on Kubernetes for production networks

The first decision for a multi-member Fabric network on Kubernetes is who owns the cluster that hosts each member's CA, peers and keys. A cluster per member organization puts the Fabric CA, peer MSP keys and TLS keys under that member's administrative control, so the member decides who can read Kubernetes Secrets, who can exec into pods and how the cluster is patched. A shared cluster, operated by one party or a hosting provider, reduces the number of control planes to run but moves that control to whoever holds cluster-admin rights. Neither option is a purely technical choice. It follows from governance and trust between members, and the Fabric deployment guide makes the same point when it says the network structure must fit the business use case and the relevant laws and regulations of the industry.

Network design changes with the model. In a per-organization setup, each member exposes its peers and any orderer nodes it hosts through its own ingress or load balancer, and every peer-to-peer and peer-to-orderer connection crosses a boundary between clusters. Each endpoint needs a TLS certificate issued by a CA that the other members trust, and the hostnames, ports and certificate SANs must be agreed in writing before the first deployment. In a shared cluster, members can use internal service addresses for co-located components, but the design still needs an external endpoint for every node that other members reach, and namespace, network policy and RBAC boundaries must keep one member's administrators out of another member's keys. Confirm with each member whether it will accept a shared control plane at all, because that answer decides the rest of the topology.

Orderer placement is where governance shows most clearly. Fabric recommends Raft, which keeps working while a majority of nodes are available, and a five-node cluster tolerates the loss of two nodes. Ask each member how many orderer nodes it will host, in which cluster and in which region, and whether any member will refuse to host orderers. Placing all orderers in one member's cluster concentrates both operational risk and control of channel configuration. Spreading them across clusters means each host operates its own etcd-backed Kubernetes control plane and carries the upgrade and recovery work that comes with it. Document the Raft consenter set, the endpoint for each orderer and the process for adding or removing a consenter before the network goes live.

Upgrades, incident ownership and audit evidence follow the cluster boundary. With one cluster per organization, each member schedules Kubernetes upgrades (the version skew policy requires the API server to move one minor version at a time), and Fabric binary upgrades must still be coordinated across members in the order the Fabric upgrade guide describes: back up the ledger and MSPs, upgrade orderers, upgrade peers, then update channel capabilities. A shared cluster means one maintenance window can affect several members, so the change process must give each member a veto or a defined notice period. For incidents, decide in advance who pages whom when a peer on another member's cluster stops responding, and who can read that member's operations service logs. For audit evidence, each member should be able to show its own key custody, RBAC bindings and backup records without depending on another member's cluster. Ask each member which evidence it must hold itself and which it can accept from a partner.

The first decision for Fabric keys on Kubernetes is which material can move into a hardware security module and which must stay file-based. Fabric uses the PKCS#11 standard to keep peer and orderer MSP keys in an HSM. TLS keys must remain file-based, so the TLS private keys for peers, orderers and the operations service still need to be mounted from a Kubernetes Secret or an external store. Plan for both paths in the same cluster, because a production peer typically carries signing keys and TLS keys that are handled differently. Confirm the exact keystore configuration for your Fabric release against the Fabric HSM documentation before you write the Helm values or operator manifests.

The official prebuilt Fabric Docker images are not built with PKCS#11 support, so an HSM setup requires custom images. Your platform team should build an image that includes the PKCS#11 library supplied by your HSM vendor, pin its version, and run it through the same image scanning and signing process as the rest of your supply chain. Ask the HSM vendor whether the client library is licensed for container redistribution and whether it needs a network path to a licensed appliance or cloud HSM. Each image rebuild also needs a retest of signing, because a library change can alter how keys are located by label or slot.

Kubernetes Secrets are stored unencrypted in etcd by default, so anyone who can read etcd snapshots or the datastore can read the MSP and TLS material. At minimum, enable encryption at rest for Secrets in the API server configuration and verify it by reading a test Secret directly from etcd. Restrict get, list and watch on Secrets with namespace-scoped roles, and do not grant cluster-wide list or watch to operators who only need to deploy a peer. The Secrets Store CSI Driver is an alternative when your organization already runs a central vault: keys are fetched at pod start rather than stored as Secret objects, but you must confirm how rotation and pod restarts are handled for Fabric components that read key files only at startup.

Encrypted volumes and HSMs answer different risks, and the trade-off is mainly operational burden, recovery and audit. Encrypted persistent volumes protect data at rest on disk, are simpler to run, and keep signing keys readable by the peer process, so a compromised pod can still use them. An HSM keeps the private key inside a device that performs signing, which strengthens key custody, but it adds a dependency on device availability, slot and label configuration, HSM backup and key ceremonies, and vendor support. Recovery of an HSM-backed identity depends on the device backup, not on the volume snapshot. Ask the HSM vendor how keys are backed up and restored, what the audit log records, whether a failed device blocks signing, and what the licensing model is for each partition or cluster.

Before you choose, confirm the design with your HSM vendor and your security team. With the vendor, verify the PKCS#11 library version, the supported key types for your Fabric identities, the behavior when the HSM is unreachable, the process for rotating keys and the retention of audit logs. With the security team, agree which identities must be HSM-backed, who holds the HSM administrator and crypto officer roles, how Secret access is logged and reviewed, and which Kubernetes encryption-at-rest provider is approved. Put these answers in the written scope before work starts, because they change the image build, the RBAC model and the disaster recovery runbook.

A Fabric upgrade on Kubernetes is safe only when the prerequisites are cleared before any image changes. Fabric v3.0 removed Solo and Kafka consensus and the v1.x legacy chaincode lifecycle, so a network must migrate Kafka ordering to Raft and move every channel to the v2.x lifecycle before it upgrades to 3.x. Start with an inventory of every channel, its chaincode definitions and its orderer type. Any channel still on the legacy lifecycle or any cluster still on Kafka should be treated as a blocking item with its own change plan, not as a note for later.

The Fabric upgrade guide sets four steps in order: back up the ledger and MSPs, upgrade orderer binaries in a rolling fashion, upgrade peer binaries in a rolling fashion, then update application channel capability levels. On Kubernetes, the backup step means capturing the persistent volumes that hold ledgers and copying MSP material and certificates held in secrets. Roll one orderer at a time and confirm Raft quorum is healthy before moving to the next. Treat the capability level change as a separate decision point. Confirm with the Fabric upgrade documentation how to reverse it before you enable it on a production channel.

The Kubernetes layer has its own order. Under the version skew policy, the kube-apiserver must be upgraded first, one minor version at a time, followed by kube-controller-manager, kube-scheduler and the kubelets. Kubelets may run up to three minor versions behind the apiserver, which gives you room to finish node upgrades without rushing them. Take an etcd snapshot with etcdctl snapshot save before each control plane change. Rehearse a restore with etcdutl in a separate environment, since a backup that has never been restored is an unverified control. Confirm that your etcd cluster has an odd member count, with five members recommended for production.

Day-two monitoring relies on the Fabric operations service on each peer and orderer. It exposes /healthz for health checks, /logspec for runtime log levels, /version for the running release and a Prometheus /metrics endpoint or StatsD push. These endpoints are secured with mutual TLS client certificates, so your Prometheus scrape configuration needs a client identity that is issued and rotated under your certificate process. Fabric Gateway client APIs work with both v2.5 and v3.0, so applications can move on their own schedule. Confirm which release line you will run, since v2.5 is the long-term support release and the 3.x and 2.5 lines both receive patch releases.