How Besu metrics, traces and logs reach Splunk through OpenTelemetry
A permissioned Besu network has to feed the monitoring stack your organization already runs, not a parallel one. Besu supports OpenTelemetry export: starting the node with --metrics-enabled and --metrics-protocol=opentelemetry sends node metrics and traces to an OpenTelemetry Collector, and the Collector forwards them to Splunk. The Besu OpenTelemetry documentation describes this path and notes that the Collector can run in Kubernetes as a daemonset. Logs follow a separate route, covered in the Splunk section below.
OpenTelemetry is a framework, not a backend. It defines how telemetry is generated, exported and collected, and it stays vendor-agnostic. Nothing in the OpenTelemetry stack stores or displays the data. Splunk, or another tool your operations team already runs, does that job. This split matters for scope: the Besu node, the Collector and the Splunk deployment each need their own owner, retention settings and access controls. Confirm with your Splunk administrators which index, HEC endpoint and token model they will allow before the Collector configuration is written, since those choices shape how the exporter is set up.
Before go-live, confirm the Collector distribution and version you will deploy against the OpenTelemetry Collector releases page, and confirm that each Collector component in your configuration is at a stability level your change process accepts, since stability varies by component. Treat the Collector as part of the control plane for telemetry: it receives data from validators and RPC nodes, so its own authentication, network placement and configuration review belong in the same change record as the Besu node settings.
What you need before connecting Besu to the observability stack
Five things need to be settled before any Besu telemetry leaves the node: the release and runtime, the Collector build, the Splunk deployment, the network path and certificates, and the people who own what is built. Each one fails in a different way, so each has its own check.
Besu release and JDK 25
Besu 26.6.0 (June 4, 2026) requires JDK 25 to build and run, so the runtime image or host must be upgraded before the node is pointed at a new telemetry configuration. Confirm the Besu version you are pinning against the GitHub releases page, since newer calendar releases ship roughly monthly and change flags over time. Record the exact Besu version in the same change request as the telemetry settings so a rollback restores both together.
Collector build with the components you need
Choose a Collector distribution that includes the receivers and exporters your design uses, such as the OTLP receiver and a Splunk HEC exporter. The contrib distribution carries most community components, and each release ships signed artifacts, SHA256 checksums and SBOM files that your supply chain review can verify. Component stability varies, so read each component's README before you depend on it in production.
Production Splunk deployment
The Splunk Enterprise trial Docker image is not suitable for production and limits daily log volume, so plan a licensed, sized Splunk deployment before go-live. Decide who owns the HTTP Event Collector token, the index names, retention and access roles. Ask your Splunk administrator to confirm ingest capacity against the log volume of validators and RPC nodes during a load test.
Network paths and certificates
Nodes need an allowed network path to the Collector, and the Collector needs a path to Splunk, with each hop permitted by your segmentation rules. Issue certificates for any TLS listener you enable and confirm the trust chain on every node. Validators should reach the Collector over a management or monitoring segment rather than the peer-to-peer network, so telemetry traffic does not compete with consensus traffic.
Authentication on the telemetry path
Telemetry endpoints must not be open to every host in the network. Collector authenticator extensions can validate incoming connections, for example with OIDC, and can attach credentials to outgoing exporter requests. Decide which identity each node presents, where secrets are stored, and how tokens rotate before the first node sends data. Confirm the exact extension names and their options in the current configuration documentation.
Named owner for dashboards and alerts
Assign one accountable owner for dashboards, alert rules and on-call routing before the pipeline goes live, and name a backup. Without an owner, metrics arrive in Splunk but no one reviews validator missed-block signals or RPC latency. The owner should also approve changes to the Collector configuration, since a single misconfigured exporter can silently drop data.
How to configure Besu, the Collector and Splunk in sequence
This sequence moves Besu metrics, traces and logs from one test node into Splunk, then widens the setup to validators and RPC nodes once each link has been checked. Each step ends with a check so that a failure is found at the point where it happens.
Confirm the Besu build and Java runtime
The platform engineer confirms which Besu release the node runs and that the host meets the runtime requirement for that release. Besu 26.6.0 release notes state that Besu requires JDK 25 to build and run, so check the release notes for the version you plan to deploy before you change any configuration. Record the version in your change ticket so that the telemetry settings you apply can be matched to a known binary. Done means the node starts cleanly on the chosen release with no configuration changes yet applied.
Enable OpenTelemetry metrics on one non-production node
The Besu operator starts one non-production node with --metrics-enabled and --metrics-protocol=opentelemetry, then points it at the Collector endpoint defined in the Besu OpenTelemetry documentation. Keep every other startup flag unchanged so that any new behavior can be traced to this one change. Done means the node restarts, keeps syncing and reports no startup errors in its log. Confirm the exact endpoint and option names against the Besu documentation for your release, since the flags are the ones to verify, not assumed from older guides.
Deploy the OpenTelemetry Collector as agent or daemonset
The platform team deploys the OpenTelemetry Collector beside the test node as an agent, or as a daemonset if the node runs on Kubernetes. Use the contrib distribution from the Collector releases, verify the signed artifact and SHA256 checksum before installing, and keep the version pinned rather than floating to the latest tag. The Collector is vendor-agnostic and does not store data itself, so its only job here is to receive Besu telemetry and forward it. Done means the Collector process starts and its own logs show the receiver listening without configuration errors.
Configure the receiver and Splunk HEC exporter
The engineer writes a Collector configuration with an OTLP receiver that accepts Besu's metrics and traces, a batch processor, and a Splunk HEC exporter that sends data to the HTTP Event Collector endpoint and token supplied by the Splunk administrator. Store the HEC token in a secret store or environment variable, not in the file itself, and restrict who can read the configuration. Check the component stability notes in each component's README, because core components have mixed stability levels. Done means the Collector reports the pipeline as started and the HEC endpoint accepts a test event sent from the Collector, which Splunk confirms with an acknowledgment or a visible test event in the target index.
Enable Splunk log ingestion for Besu log files
The operator configures Splunk to ingest the Besu log files from the test node, either through a forwarder or the Collector's log path, depending on the design chosen for the environment. Logs go to a Splunk index set aside for this test so that search results stay separate from production data. The Besu Splunk documentation notes that the Splunk Enterprise trial Docker image is not suitable for production and limits daily log volume, so use a licensed, production-grade instance for any real evaluation. Done means a search in Splunk returns recent Besu log events with the host and source fields you expected.
Verify metrics and logs arrive end to end
The engineer checks that a specific metric from the test node, such as a block height or peer count, appears in Splunk with a current timestamp, and that the matching log events from the same minute are visible in the same time range. Compare the two sources to confirm they describe the same node by host and instance name. Record the query used, the index, and the time of the check as evidence for the runbook. Done means at least one metric data point and one log event from the node are found in Splunk for the same period, and the values agree with what the node reports locally.
Roll out to RPC nodes with a separate pipeline
The team applies the verified configuration to RPC nodes first, because their load and exposure differ from validators, and watches Collector resource use and Splunk ingestion volume during the change. Give RPC nodes their own dashboards and alerts for request latency and error rates, since those signals matter for them. Keep the same check as before: data points and log events for each RPC node appear in Splunk after the rollout. Done means every RPC node in scope reports to Splunk and the Collector stays within the resource limits set in the design.
Roll out to validators after a monitored change
The operator applies the configuration to validators one at a time, keeping the quorum and block production stable during each restart, and confirms that each validator still signs blocks after its telemetry settings change. Validators need the most attention on consensus signals such as block production, round changes and peer connectivity, so build alerts for those before the rollout begins. Done means each validator in scope sends metrics and logs to Splunk and the network keeps producing blocks throughout the change, with the results recorded in the change ticket.
What to watch on validators compared with RPC nodes
Validator nodes and RPC nodes fail in different ways, so they need different alerts. On a QBFT validator, the questions that matter are whether the node keeps producing and importing blocks and whether it keeps participating in consensus. A validator that stops signing blocks can reduce the network's fault tolerance long before users notice anything wrong, so alerts should focus on block production, block import progress and consensus participation. RPC nodes serve applications and users, so their signals are different: request latency and error rates on the JSON-RPC endpoint, peer count, and sync state relative to the chain head. A healthy validator can still be a poor RPC node if it is slow to answer queries.
Besu exposes these values through its metrics endpoint, which is enabled with --metrics-enabled and can be sent to an OpenTelemetry Collector when --metrics-protocol is set to opentelemetry. Before you wire any of this into Splunk, confirm the exact metric names for your Besu version against the Besu metrics documentation, since names change between releases. The Besu sample Grafana dashboard is a useful reference for which panels operators commonly build, but treat it as a starting layout, not a finished monitoring design.
Thresholds and dashboards must come from your own baseline. Record normal block times, peer counts, RPC latency percentiles and sync lag for each node role over a representative period, then set alerts relative to that baseline and to your validator count and fault tolerance. Ask your team to confirm which alerts page an on-call engineer and which only create tickets, and agree on who owns the validator set and the RPC tier in each organization. Review the baseline after each Besu upgrade, because a release can change what normal looks like.
How Besu metric, trace and log fields map to Splunk and Collector pipelines
This table maps each Besu telemetry category to the place it is configured and the team that usually owns it. Read it as a checklist before go-live: confirm each attribute name and option against the Besu OpenTelemetry and Splunk docs for your Besu release, since names can change between calendar versions. The main takeaway is that Besu sets what it emits, the Collector shapes and routes it, and Splunk decides how it is indexed.
| Telemetry or routing item | Where it is set | Who owns it |
|---|---|---|
| Node metrics and traces | Besu startup options --metrics-enabled and --metrics-protocol=opentelemetry | Besu node operator |
| Metrics and trace export target | Endpoint the Collector receiver listens on, referenced from Besu configuration | Platform or observability team |
| Collector pipelines | Receivers, processors and exporters in the Collector configuration file | Observability team |
| Collector resource attributes such as node role and network name | Resource processors in the Collector, or attributes confirmed in the Besu docs for your release | Observability team with the network operator |
| Splunk index and source type | Splunk HEC exporter settings or Splunk-side ingest configuration | Splunk administrator |
| HEC token | Created in Splunk and stored in a secrets manager, then referenced by the Collector exporter | Splunk administrator and security team |
| Log files and log format | Besu logging options and the file path the log shipper reads | Besu node operator |
| Authentication to the Collector or backend | Authenticator extensions on receivers or the oauth2client extension on exporters | Security team with the platform team |
Securing the telemetry path from node to Splunk
Besu metrics, traces and logs cross several trust boundaries before they reach Splunk: the node, the Collector, the network path and the Splunk HEC endpoint. Each hop needs its own authentication, secret handling and content review, and the node's private key must never be part of what leaves the host.
Authenticate incoming collector connections
Collector authenticator extensions can check the identity of clients sending data to a receiver, for example through OIDC. Decide which components may write telemetry into the Collector, such as a sidecar on each node or a gateway shared by several nodes, and confirm the receiver accepts only those identities. Check the authenticator extension README for the version you deploy, because component stability varies across the Collector's components. Test that an unauthenticated request is rejected before go-live.
Authenticate outgoing exporter requests
The oauth2client extension adds credentials to requests the Collector sends to a backend. Use it where the destination requires a token issued by your identity provider rather than a static secret in configuration. Confirm with your Splunk administrators which authentication method the HEC endpoint accepts in your environment, and confirm that the extension supports it before you plan around it. Record which service account the Collector uses so access can be reviewed and revoked.
Keep HEC tokens out of plain config
A Splunk HTTP Event Collector token written directly into a Collector configuration file is readable by anyone with access to that file, its backups or its deployment manifests. Load the token from an environment variable or a secrets mechanism your platform already runs, and restrict read access to the Collector's configuration. Rotate the token on a schedule your security team sets, and make sure a rotation does not require editing the node's Besu configuration.
Follow Collector configuration best practices
The OpenTelemetry security guidance for Collector configuration covers settings such as limiting which interfaces receivers listen on, avoiding unnecessary exposed endpoints and reviewing the extensions enabled in each pipeline. Apply it as a checklist during review of each Collector file. Keep the configuration in version control so changes to listeners, exporters and credentials are reviewed like any other change to a regulated system.
Encrypt traffic between nodes and the Collector
Telemetry sent over plain HTTP can be read or altered by anyone with access to the network segment. Enable TLS on the path from each Besu node to the Collector and from the Collector to Splunk, and decide who issues and renews the certificates. Confirm the certificate chain your Splunk HEC endpoint presents is trusted by the Collector host. Note that Besu does not terminate HTTPS natively, so the Collector and its network path are where this protection is configured.
Limit log content leaving the network
Besu logs can include operational detail such as peer addresses, network topology and transaction-related messages. Decide which log levels and fields are forwarded to Splunk, and use Collector processors to drop or redact fields that are not needed for monitoring or audit. Agree the filtering rules with your compliance and security teams before the pipeline goes live, and document them alongside the retention settings in Splunk.
Keep the node private key off telemetry paths
The Besu node private key file is not encrypted. It must never be collected as a log file, read by a Collector receiver, mounted into a telemetry sidecar or included in a diagnostic bundle sent to a backend. Restrict the Collector's file access to log directories only, and review any filelog or host receiver for paths that could reach the key directory. Where hardware-backed key storage is in use, confirm the key never appears in the node's data directory that the Collector can read.
Why Besu telemetry stops reaching Splunk and how to fix it
This table lists the failure points that most often stop Besu metrics, traces and logs from reaching Splunk, with the likely cause and the fix for each. Read it top to bottom when data is missing, then check the component named in the cause column before changing configuration.
| Symptom | Likely cause | Fix |
|---|---|---|
| Metrics flag set but no OpenTelemetry data | Besu was started with --metrics-enabled but without --metrics-protocol=opentelemetry, so it exposes Prometheus-style metrics instead | Start Besu with both --metrics-enabled and --metrics-protocol=opentelemetry, restart the node, and confirm the Collector receives data |
| Collector not reachable from the daemonset or agent | The Besu node cannot resolve or open a connection to the Collector endpoint, often because of a wrong host, port or network policy | Confirm the Collector address from the node's network namespace, check Kubernetes network policies or host firewall rules, and test the port before changing Besu settings |
| HEC token rejected | The HTTP Event Collector token is disabled, mistyped, assigned to the wrong index, or sent to a Splunk endpoint with a different TLS certificate | Verify the token in Splunk, confirm the target index is allowed for that token, and check the Collector exporter's endpoint and TLS settings |
| Splunk trial licence limiting daily log volume | The Splunk Enterprise trial Docker image limits daily log volume and is not suitable for production, per the Besu Splunk docs | Use a production Splunk license for real monitoring, and treat the trial only for testing pipelines. Ask your Splunk account team which license type fits your daily volume |
| Log file not ingested | The Splunk input does not point at the file Besu writes, the file path changed after a restart, or the forwarder lacks read permission | Confirm the log path in Besu's logging setup, check file permissions for the Splunk forwarder user, and search the index for recent events to confirm ingestion |
| Unstable or in-development Collector components | A receiver, processor or exporter is at an early stability level, or a log SDK is still in development for the language in use | Check each component's README for its stability level before adopting it, and prefer components documented as stable for production pipelines |
| Version mismatch after upgrades | The Collector, its contrib build, or the Besu release changed component names or options, so an older configuration no longer loads | Pin Collector and Besu versions, review release notes for both before upgrading, and validate the configuration in a test environment first |
Questions buyers ask before connecting Besu to Splunk
What does it cost to connect Besu to Splunk, and what drives the price?
Cost depends on the written scope, which is agreed before work starts. The main drivers are the number of nodes, whether you run one Collector or a gateway tier, how many Splunk indexes, dashboards and alerts you need, and how much log and metric volume you expect. Splunk licensing is a separate cost paid to Splunk, usually sized by daily ingest, so ask your Splunk account team for the ingest estimate before you commit. DeployOpen does not publish a price list. A quote follows from the agreed scope, and it should separate one-time build work from any recurring support or retainer work.
How long does a Besu to Splunk monitoring project take?
Duration depends on how many node types you monitor, how many environments need the pipeline (for example test, staging and production), and how quickly your Splunk and network teams can approve changes. A single validator and RPC pair in one environment is a smaller job than a multi-site consortium with several organizations sending data. The Besu documentation explains the OpenTelemetry and Splunk setup steps, but your change windows and approvals usually set the timeline more than the configuration does. Ask for a schedule in the written scope that names the dependencies on your side.
What must our team provide before the engagement starts?
Plan on supplying four things. First, Splunk administrator access or a named Splunk contact who can create HEC tokens, indexes and roles. Second, network paths from the Besu nodes to the OpenTelemetry Collector and from the Collector to Splunk, including firewall rules and any proxy routing. Third, access to the node hosts or the Kubernetes cluster where Besu runs, with rights to change startup flags. Fourth, a list of which validators and RPC nodes are in scope, with owners for each. Missing any of these is the most common cause of delay, so confirm them during scoping.
Can our in-house team set this up without a partner?
Yes, if your team already runs Besu, operates an OpenTelemetry Collector or Splunk, and has someone who can review the pipeline configuration. The Besu OpenTelemetry and Splunk documentation covers the core flags and data flow. The work gets harder when the team is new to the Collector configuration model, when several environments must match, or when the pipeline must meet internal logging and access rules. A useful test is whether someone on your team can explain where each metric, trace and log leaves the node, who can read it, and how the pipeline is restarted after a failure. If not, a partner can help build and document it, and your team can run it afterward.
Does OpenTelemetry replace Splunk?
No. OpenTelemetry is an observability framework and toolkit for generating, exporting and collecting telemetry such as traces, metrics and logs. It is not a backend, so it does not store, search or visualize data by itself. The Collector receives and forwards the data, and Splunk, or another backend, stores it and provides search, dashboards and alerts. In this design, OpenTelemetry handles collection and routing, and Splunk remains the place where your operators and auditors work with the data. Keeping that split clear helps when you decide which tool owns retention, access control and alert rules.
What does DeployOpen not provide for compliance?
DeployOpen prepares systems, controls and evidence for compliance assessments, including logging and monitoring configuration that your assessors may review. DeployOpen does not issue audit reports, attestations or certifications. Independent auditors and certification bodies perform those assessments. Ask your auditor early which log sources, retention periods, access records and change evidence they expect from the pipeline, and include those requirements in the written scope so the work can be checked against them.
Do the validator and RPC nodes need different monitoring?
Yes. Validators are sensitive to missed blocks and consensus health, so their alerts should focus on block production, peer count and sync state, with tight access to their telemetry. RPC nodes serve applications and users, so their watch list should include request latency, error rates and authentication failures on the JSON-RPC endpoint. Logs from both node types can contain operational detail, so limit who can read them in Splunk. Decide these alert sets during design, because a single generic dashboard usually hides the signals that matter for each role.
Keeping the telemetry pipeline current as Besu and the Collector release
Besu ships calendar releases roughly monthly, so the telemetry pipeline needs a scheduled review, not a one-time setup. The releases page lists 26.10.0 (October 16, 2026) among recent versions, and each release can rename metrics, change command-line flags or alter the OpenTelemetry export path. Before a production upgrade, start the new version on a staging node that uses the same --metrics-protocol=opentelemetry setting and the same Collector configuration as a validator or RPC node. Confirm that every metric name your dashboards and alert rules depend on still appears, that trace export still reaches the Collector, and that the Splunk index receives the expected log lines. Record the results so a later upgrade can be compared against a known baseline.
The Collector moves on its own schedule. OpenTelemetry Collector v0.163.0 was released on October 16, 2026, following v0.162.0 on September 29 and v0.161.0 on September 16, 2026. Component stability levels differ across the contrib distribution, and the stability of a receiver, processor or exporter you rely on can change between releases. Read the README for each component in your configuration, pin the Collector version in your deployment manifests, and move to a newer build only after the staging test passes. Collector releases publish signed artifacts, SHA256 checksums and SBOM files, which your change process should verify before deployment.
Database state is the riskier part of an upgrade. The Besu 26.10.0 release notes state that the Bonsai database upgrades automatically on first start, and older Besu versions cannot open the upgraded database. A telemetry-only change can therefore become a rollback problem if a node is restarted on a newer Besu build. Take and verify a backup of the node data directory before any Besu upgrade, including the nodes that feed the Collector, and confirm that you can restore it onto the previous version. Plan the upgrade window with your validator quorum in mind, so that no more validators than your consensus design tolerates are restarted at once.
How DeployOpen scopes and delivers Besu observability work
DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside the client's own infrastructure, including Hyperledger Besu monitoring with OpenTelemetry and Splunk. Work starts from a written scope, so validator and RPC telemetry, pipeline design and Splunk ingestion are agreed before any configuration begins.
How we scope
- We confirm which Besu node roles (validators, RPC nodes, bootnodes) need telemetry and which teams consume it.
- We agree which telemetry types are in scope: metrics, traces, logs, or a defined subset.
- We record the Splunk target, whether HEC or another backend, and the retention and access rules it must meet.
How we build
- We deploy the OpenTelemetry Collector in your environment, as an agent, gateway or daemonset, matching your cluster layout.
- We configure Besu metrics and OpenTelemetry export, then test that validator and RPC signals arrive as expected.
- We set collector authentication and exporter credentials in line with your security baseline and check each pipeline end to end.
How we hand over
- We document pipeline configuration, component versions, credential locations and the alerts we set up for validators and RPC nodes.
- We walk your operations and security teams through dashboards, log searches and the runbook for pipeline failures.
- We record upgrade steps for Besu and Collector releases, including checks to run before and after each change.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, so telemetry keeps pace with Besu releases.
- We can review Collector component stability notes as part of each planned upgrade before changing production pipelines.
- Any change to the agreed scope is documented in writing before it is built, keeping cost and timing visible to you.
