Besu log management with Elastic Stack or Loki

Besu log management with Elastic Stack or Grafana Loki: what the integration does

Every Besu validator and RPC node writes its logs to local disk by default, so an operations or audit team that needs to answer "what happened on validator three at 02:00" has to log in to each host. Centralizing those logs in one searchable store removes that dependency, and the Besu docs describe two ways to do it. The first is the Elastic path: Filebeat ships node logs, Metricbeat collects node metrics into Redis for Logstash, and a Logstash pipeline parses the output. Besu's JSON log format has a matching pipeline configuration, and the default text format can be parsed with the Grok plugin. The second is the Grafana path: Grafana Alloy collects the logs and sends them to Grafana Loki, which you query through Grafana Explore. Both are documented integration methods described in the Besu docs, so neither requires a custom connector you would have to maintain yourself.

The choice between them is less about Besu and more about what you will operate afterward. Elastic's default distribution is under the Elastic License 2.0, and AGPLv3 is offered as an OSI-approved option, so your legal team should confirm which license terms apply to the version you deploy. Grafana Loki is licensed under AGPLv3, which carries its own obligations if you modify and offer the software to others. Confirm the subscription tier you need for TLS, role-based access and single sign-on on the Elastic side, since those features sit in different tiers. On the Loki side, the stack has no built-in authentication layer, so you will also place an authenticating reverse proxy in front of it. Each option is a separate long-running service with its own storage, upgrade cycle and on-call burden.

Before choosing, confirm three things with your operations and compliance owners: who will run the stack day to day, how long validator and RPC logs must be retained and in which jurisdiction they may be stored, and whether the monitoring platform already in use can absorb Besu logs at your expected volume. The log content also needs review. Node logs can contain peer addresses and transaction-related details, so access to the central store should follow the same access model as the nodes themselves. Settle these questions first, because they determine which path is sensible and what the rest of the build must include.

What you need before shipping Besu logs

Log pipelines fail quietly when versions, storage and network paths are assumed rather than checked. The cards below cover the decisions to settle before any Besu log reaches Elastic or Loki, and each one should be confirmed in your own environment before planning starts.

Besu release line and JDK

Besu ships calendar releases roughly monthly, and the GitHub releases page lists 26.10.0 (October 16, 2026) as the latest. Besu 26.6.0 requires JDK 25 to build and run, so confirm the JDK on every node image before you change log formats. Pick one release line for all validators and RPC nodes so the log lines your parsers expect match what each node writes.

Elastic stack versions in scope

Elastic 9.5.5 and 9.4.8 were released on October 6, 2026, and 8.19.23 is still maintained. Decide which line you will run, because Filebeat, Logstash and Elasticsearch must be upgraded together and the 8.x line reaches end of maintenance on January 15, 2027. Confirm which Elastic release your security team has approved before writing any pipeline configuration.

Loki version and deployment mode

Loki v3.7.8 was published on September 17, 2026. Choose the deployment mode before you size storage, because simple scalable mode is being deprecated and will be removed with Loki 4.0. Monolithic mode fits smaller volumes, and microservices fits very large clusters, so confirm your expected daily log volume with the team that owns the nodes.

Object storage for Loki chunks

Loki keeps compressed chunks and its index in object storage, so the bucket is a platform dependency with its own backup and access rules. Amazon S3, Google Cloud Storage and Azure Blob Storage are the recommended production stores, while MinIO and local filesystems are supported but not typically recommended for production. Ask whether your data residency rules allow the chosen store and who holds the bucket credentials.

Network paths and certificates

Each Besu node needs a route to its log shipper, and the shipper needs a route to Logstash or Loki. Map every hop across firewalls, proxies and cluster boundaries, and decide where TLS terminates on each one. Besu does not terminate HTTPS natively for its JSON-RPC API, so the same principle should guide how you handle any log endpoint you expose.

Reverse proxy in front of Loki

Loki has no built-in authentication layer, so an authenticating reverse proxy such as nginx, HAProxy, OAuth2 Proxy or Pomerium must sit in front of it. Plan how that proxy maps user identity to the X-Scope-OrgID tenant header when multi-tenancy is enabled. Without this layer, anyone who can reach the Loki port can read node logs.

Choosing Elastic Stack or Loki for Besu logs

Choose this path when security analysts already search Elasticsearch and Kibana, and when detection rules, dashboards and case handling live there. Besu logs then become one more data source in a tool the team already uses, so analysts keep their workflow and the new work sits mostly with the platform team.

Team skills: Operators who can run Filebeat, Logstash and Elasticsearch index lifecycle management, plus analysts who write Kibana queries and build saved searches.
Components: Filebeat on each node host, Metricbeat for node metrics, Logstash with a Besu pipeline (the Besu Elastic Stack documentation describes a JSON pipeline and a Grok option for the default log format), Elasticsearch and Kibana.
Cluster shape: A highly available Elasticsearch cluster needs at least three master-eligible nodes. Data tiers (hot, warm, cold and frozen) control how much retention costs over time.
Security features: TLS, file and native realms and role-based access control are in the free Basic tier. Single sign-on, LDAP or Active Directory authentication and field- or document-level security are in paid tiers, and Platinum is no longer offered to new customers.
Kubernetes operations: If you already run Kubernetes, Elastic Cloud on Kubernetes (ECK) manages Elasticsearch, Kibana, Logstash and Beats as Kubernetes resources.

Before you choose this path, confirm which subscription tier covers the single sign-on and audit access your analysts require, and whether your team will operate the stack or buy it as a managed service.

Licensing and support terms that change the log platform choice

The license and support terms of each log platform can settle the choice before any technical comparison begins. Elastic's default distribution of Elasticsearch and Kibana has been under the Elastic License 2.0 since 2021. Elastic added AGPLv3 to the source code options in September 2024, before the 8.16 release, so that an OSI-approved open source option exists. Elastic's licensing FAQ states that the Elastic License 2.0 and the SSPL are not OSI-approved, so a team that requires OSI approval should confirm which license it will take and whether it will use the default distribution or the AGPLv3 code.

Loki takes a different path. Grafana Labs announced on April 20, 2021 that Loki and its other core projects would move from Apache License 2.0 to AGPLv3, and the Grafana Labs licensing page sets out the current terms. Teams that modify Loki, offer it to users over a network, or bundle it into a product should ask counsel how AGPLv3 network-use obligations apply to their setup. Support terms also matter. Elastic's EOL page lists end of maintenance for the 8.x line on January 15, 2027, with 8.x support ending July 15, 2027, and end of maintenance for the 9.x line on October 15, 2027. Elastic states that the Platinum self-managed tier is no longer available to new customers, who are directed to Elastic Cloud, so a team that expects to buy Platinum features should confirm the current tier with Elastic before it designs around them.

Before choosing, ask legal and procurement to confirm license obligations for your specific deployment, including any redistribution, modification and hosting of the logging stack for internal users or auditors. Ask Elastic and Grafana Labs in writing which subscription or support terms apply to the version you plan to run and how long that version will be maintained. Then map the answers to your upgrade calendar. A log platform that must be upgraded within a short maintenance window will need a funded plan for that work, and the Elastic support dates set the first deadline in that plan.

How to configure Besu log shipping step by step

This sequence takes Besu node logs from local files to a searchable platform, with labels that identify each node's role and network, encrypted transport and a validated path from node to query. Each step ends with a check you can run before moving on.

Confirm log format and file locations
Platform or operations engineers on each Besu node check the log format the node writes, where the files live, how they rotate and who can read them. Record the output path, the rotation behavior and the file permissions for the shipper's service account. Done when a sample of recent log lines from every node type (validators and RPC nodes) is saved for parser testing and the file paths are written into your runbook. Rollback: no change is made in this step, so none is needed.
Choose the Elastic Stack or Loki path
Architects and the security lead decide which platform will receive the logs, based on existing skills, the licensing terms each vendor applies to the code you will run, and the retention and access controls your auditors expect. Elastic's default distribution uses the Elastic License 2.0, and its AGPLv3 option is OSI-approved. Loki is AGPLv3 and has no built-in authentication layer. Done when the decision and its reasons are written down and approved. Rollback: a decision record can be revised before any infrastructure is built.
Stand up the receiving platform
The platform team deploys either an Elasticsearch cluster with Kibana, or Loki on object storage with multi-tenancy enabled. An Elasticsearch cluster for high availability needs at least three master-eligible nodes. For Loki, set auth_enabled to true and plan the X-Scope-OrgID tenant values before any shipper connects. Done when the cluster or Loki instance reports healthy and a test document or test stream can be written and read from an admin session. Rollback: remove the empty deployment, since no operational data has been sent yet.
Deploy Filebeat with a Logstash pipeline
The logging engineer installs Filebeat on each Besu node, points it at the log files confirmed in step one, and sends events to a Logstash pipeline. Use a JSON parsing filter for Besu's JSON log format. For the default log format, use the Grok plugin with a pattern tested against sample lines. Done when a unique test message written to a node appears in Kibana with the expected fields populated. Rollback: stop the Filebeat service on the node and remove its configuration, leaving the local log files untouched.
Deploy Grafana Alloy for Loki
If you chose Loki, the logging engineer installs Grafana Alloy on each node, configures it to tail the same files and pushes entries to Loki with the tenant header for that environment. Alloy is Grafana's supported log shipper for Loki. Done when the same unique test message is visible in Grafana Explore under the expected tenant, with no dropped or duplicated entries across a restart of Alloy. Rollback: stop Alloy and remove its configuration; Loki keeps any data already accepted, so review tenant contents before deleting them.
Add labels or fields for role and network
Add consistent metadata to every event: node role (validator, RPC, bootnode), network name, environment and host identifier. In Elastic this is a field set in the pipeline. In Loki these become stream labels, so keep their values low in cardinality and avoid per-transaction or per-account values. Done when a query filtered by network and role returns only the expected nodes and a test search on each label value matches the intended hosts. Rollback: revert the pipeline or Alloy configuration and reindex or relabel any events that were tagged incorrectly.
Enable TLS and authentication on every hop
The security engineer enables TLS between shippers and the receiving platform, and restricts access to the Elastic or Loki endpoints with authenticated users or an authenticating reverse proxy in front of Loki. Elastic made TLS and native user realms free in the Basic tier, while SSO and field-level security are in paid tiers. Done when unencrypted connections are refused, unauthenticated requests fail and a role-limited account can read only its assigned data. Rollback: keep the previous endpoint available only for the cut-over window, then remove it.
Validate end to end and set retention
Run an end-to-end test by generating a known event on each node type and confirming it is searchable in Kibana or visible in Grafana Explore within your agreed latency. Then configure retention. Loki retention is disabled by default, so set it explicitly and confirm the compactor is running. Confirm the retention period with your compliance owner. Done when the test passes on all nodes, retention is verified on a sample stream and the runbook records the restore procedure. Rollback: if events are missing or retention is wrong, disable the shipper, correct the configuration and reprocess from local files still within rotation.

How Besu log fields map to indexes, labels and searches

This table compares how each Besu log element lands in Elasticsearch and in Loki, and why it matters for operations and audit. Read each row as a decision: what the shipper parses, what becomes an indexed field, and what becomes a Loki label. The main takeaway is to keep per-peer and per-transaction values in searchable content so that Loki streams stay bounded.

Source element	Elastic target	Loki target	Purpose in audit or operations
JSON log event fields	Logstash JSON filter maps keys to event fields. Confirm names for your Besu version.	Stored as log line. Parsed at query time with a LogQL json stage.	Reconstructs node events for incident review and audit trails
Default text log line	Grok pattern extracts timestamp, level, logger and message fields.	Parsed at query time with pattern or regex stages. Not promoted to labels.	Keeps default-format output searchable when JSON logging is not used
Log level	Keyword field for filters, dashboards and alert rules.	Candidate label only if the value set stays small.	Separates errors and warnings from routine node output
Node identity added by the shipper	Keyword fields such as host or node name, added by Filebeat or Logstash.	Static label per node, set in the Alloy configuration.	Attributes each line to a validator or RPC node during investigations
Timestamp	Mapped as a date field used as event time.	Log entry timestamp taken from the line or the shipper.	Orders events across nodes. Confirm clock sync on every host.
Message body and stack traces	Analyzed text field for free-text search.	Line content searched with LogQL line filters.	Finds peer, block and transaction messages during triage
Peer IDs, transaction hashes and account addresses	Indexed keyword or text fields for exact-match search.	Kept in line content, never promoted to labels.	Traces specific activity while keeping stream counts bounded
Tenant and retention context	Index lifecycle policy per index pattern.	X-Scope-OrgID tenant header and per-stream retention rules.	Sets how long audit-relevant logs stay queryable

Besu log retention for regulated environments

Retention is a decision the organization makes before the pipeline goes live, not a default the tools supply. In Loki, retention is applied by the compactor and is disabled by default, so logs are kept indefinitely until retention_enabled is set. Retention can be applied globally, per tenant or per stream selector, and each stream rule must be at least 24 hours. That means Besu validator logs, RPC node logs and audit-relevant streams can carry different retention periods, but only if the team separates them with labels and tenants from the start. In Elastic, the equivalent work sits in lifecycle policies that move indices through hot, warm, cold and frozen tiers, which map to data node roles. Each tier needs sizing, hardware and cost decisions of its own.

The periods themselves must come from the organization's record-keeping and audit requirements, and the team and its auditor must define them in writing. Confirm which Besu log streams count as records, who may read them, how long they must remain readable, and how deletion is evidenced. Check whether the retention settings and deletion logs will satisfy those requirements before anything is archived, because a retention rule that removes data too early is as hard to explain to an assessor as one that keeps sensitive data longer than needed.

Object storage choice affects durability as much as retention length. Loki stores compressed chunks and its index in object storage, and its documentation lists Amazon S3, Google Cloud Storage and Azure Blob Storage as recommended production stores. S3-compatible stores such as MinIO and the local filesystem are supported, but the docs do not typically recommend them for production. If the organization runs an on-premises object store, confirm its replication, backup and failure behavior with the vendor, and test restores of Loki data before relying on that store for audit retention.

Securing the log pipeline and the data it carries

A Besu log pipeline carries node operations data and can expose sensitive material if it is set up carelessly. The six controls below cover the platform in front of the logs, the data that enters them, and the people and services that touch them, so each can be checked before the pipeline goes live.

Authenticate access to Loki

Loki has no built-in authentication layer, so an authenticating reverse proxy such as nginx, HAProxy, OAuth2 Proxy or Pomerium must sit in front of it. When auth_enabled is true, Loki isolates tenants by the X-Scope-OrgID header on every request, so the proxy must set or validate that header and never pass client-supplied values through unchecked. Test that one tenant cannot query another tenant's streams.

Use Elastic built-in security features

TLS, the file and native user realms and role-based access control have been free in the Elastic Basic tier since 2019. Single sign-on, Active Directory and LDAP authentication, and field- or document-level security remain in paid tiers. Decide early whether your identity requirements can be met on the free tier, because adding SSO later changes the subscription you need.

Keep key material out of logs

The node private key file is not encrypted, so it must never be read into a log, shipped by a collector or indexed. Likewise, JSON-RPC credentials files and JWT signing material must stay out of the shipper's scope. Confirm what each Besu log level writes, including debug and trace, and set collector filters to drop any field or line that echoes credentials before data leaves the host.

Encrypt transport between components

Logs cross several hops: Filebeat or Alloy to Logstash or Loki, Metricbeat to Redis, and each receiver to its storage. Use TLS on each hop, with certificates issued by your internal authority and verified by the sender, not just accepted. Confirm that Redis, Logstash and Loki listeners are reachable only from the pipeline network, since Loki does not terminate its own authentication.

Scope shipper accounts to least privilege

Filebeat, Metricbeat and Logstash should each use a dedicated Elasticsearch account with only the index write and pipeline privileges it needs, not a superuser or the built-in administrative accounts. For Loki, give each shipper its own tenant credentials through the proxy. Rotate shipper secrets on a schedule you document, and revoke a shipper's access as soon as its host is decommissioned.

Audit access to the log platform

The logs themselves are sensitive, and so is who reads them. Record authentication events, role changes, index and retention edits, and queries made through Kibana, Grafana or the proxy. Decide where these audit records live, who can read them, and how long they are kept, so that auditors can reconstruct who viewed node data and when. Verify that audit records cannot be altered by the accounts they describe.

Common failures when shipping Besu logs and how to fix them

This table lists the symptoms teams most often see when Besu logs move into Elastic Stack or Loki, the likely cause of each and the first fix to try. Read it as a triage starting point, and confirm the exact error text in your own logs before changing a pipeline or retention setting.

Symptom	Likely cause	Fix
Logstash receives lines but no fields are parsed	The log format configured in the pipeline does not match the log output of the Besu version in use	Check the Besu version and its log output, then align the pipeline's format setting with it. Confirm the expected format in the Besu Elastic Stack docs for your release
Grok filter tags events with a parse failure after a Besu upgrade	A release changed the wording, field order or spacing of default log lines, so the existing Grok pattern no longer matches	Pull a sample of new log lines, update the Grok pattern, and test it against both old and new lines before rolling it out to all nodes
Filebeat stops shipping new entries after log rotation	The harvester is still reading the renamed or deleted file, or the rotation settings do not match how Besu writes its logs	Confirm the rotation scheme on the node, check Filebeat's registry and harvester state, and restart Filebeat if the registry points to a removed file
Loki queries return no results for a node that is shipping logs	The X-Scope-OrgID header is missing from the query or does not match the tenant used at write time, or auth_enabled is off or on unexpectedly	Compare the tenant value in the Alloy or Grafana data source configuration with the one used for ingestion, and confirm the auth_enabled setting on both sides
Loki object storage or disk usage grows without limit	Retention was never enabled, so the compactor keeps every chunk indefinitely	Set retention on the compactor for your required period, decide whether it applies globally, per tenant or per stream, and confirm the first deletion run in a test environment
Loki ingestion slows and queries time out	Label values carry high-cardinality data such as transaction hashes, account addresses or request IDs, which creates a very large number of streams	Keep labels to a small fixed set such as node role and network, and move identifiers into the log body where they are searched with line filters
Alloy or Logstash writes fail with authentication or 4xx and 5xx errors	An authenticating reverse proxy in front of Loki or Elasticsearch blocks the write path, strips a header, or enforces a body size limit	Allow the shipper's identity or token on the write route, confirm the tenant header passes through, and check proxy body size limits against batch sizes
Events appear out of order or with timestamps in the future or past	Clock skew between Besu nodes, shippers and the storage backend, or timestamps parsed in the wrong time zone	Run a time synchronization service on every host, confirm the time zone in the parser, and compare event times with the host clock before changing pipeline logic
Log volume is much higher than planned after enabling debug output	Verbose logging levels were left on in production, so every peer and transaction event is written and shipped	Set the production log level explicitly, review what each level emits on your release, and size retention and storage from a measured daily volume

Questions buyers ask before outsourcing Besu log management

What drives the cost and effort of an Elastic or Loki logging build?

The scope drives it: how many Besu nodes produce logs, how much log volume they generate, how long logs must be kept, which platform you already run, and how the pipeline connects to your identity and security tooling. Node count and volume set the size of the storage and ingest tier. Retention sets how much of that storage is kept over time and therefore what recurs. An existing Elastic, Grafana or object storage estate lowers the build effort, while a new platform adds design and operations work. Security integration, such as single sign-on and access roles, adds configuration and testing. DeployOpen quotes come from a written scope that lists these inputs, and each platform's licensing terms should be confirmed separately with its vendor, since some security features sit in paid tiers.

How long does a Besu log pipeline build take?

The timeline depends on the agreed scope, so there is no fixed duration on this page. The main variables are the number of node types to onboard, whether Filebeat and Logstash or Grafana Alloy and Loki is chosen, how many log formats need parsing, the storage and high availability design, and how many review and acceptance cycles your team requires. A single validator environment with an existing platform will move faster than a multi-organization consortium with new infrastructure. Ask for the milestones, the client dependencies on each milestone, and the acceptance criteria in the written scope before committing to dates.

What does our team need to provide for a logging engagement?

Your team typically provides the inputs the pipeline depends on. These are:

Access to the Besu nodes and hosts that will ship logs, with the change windows for configuration updates.
Confirmation of the log format each node writes, including any custom logging configuration, so parsing matches the actual output.
Written retention rules by log type and by environment, including any regulatory minimum and the deletion rules that apply at the end of that period.
Your identity provider details for operator access to dashboards and the query layer.
Storage decisions, such as the object store or Elasticsearch cluster, its sizing approval and backup location.

Confirm each item early, because retention and format decisions change the design.

Can an internal team build and run Besu logging alone?

Yes, a capable platform team can build this with the Besu documentation and the vendor documentation for the stack it chooses. Teams usually need help in three places. The first is the log format and parsing, because the default Besu format needs a Grok pattern and a custom format needs its own pipeline. The second is storage and retention design, including how compaction, index layout and deletion behave under your retention rules. The third is access control and evidence, because operator access, tenant separation and audit trails must match your internal controls. Many teams keep day-to-day operation in house and bring in a partner for the design and the first hardening pass. Your team should decide which of these it can own from the start.

How do Besu and log stack upgrades affect the logging pipeline?

Each upgrade can change the log output, the metrics names or the storage format, so the pipeline needs its own upgrade step. Besu ships roughly monthly under calendar versioning, so plan a regular review of release notes rather than reacting to failures. Log parsers should be tested against a sample from the new Besu version before rollout. Storage upgrades need a tested backup and rollback path, since the Besu 26.10.0 release notes state that an upgraded Bonsai database cannot be opened by older versions, which shows why a backup must come before any change to node data. Elastic and Loki also publish their own release and end-of-life schedules, which should be tracked alongside Besu's release cycle in the operations plan.

Does DeployOpen issue audit reports for our logging setup?

No. DeployOpen does not issue audit reports, attestations or certifications. What DeployOpen prepares is the system, its controls and the evidence that an auditor will ask for, such as log coverage by node, retention settings, access configuration and documented change history. Independent auditors and certification bodies assess that evidence and issue any report. Ask the auditor which log fields, retention periods and access records they expect before the design is fixed, so the pipeline produces what the assessment needs.

Do Elastic and Loki licensing terms matter for a regulated logging platform?

They can. Elasticsearch and Kibana are offered under three licenses: the Elastic License 2.0 (the default distribution), SSPL 1.0 and AGPLv3, and Elastic states that ELv2 and SSPL are not OSI-approved open source licenses. Grafana Loki is licensed under AGPLv3, which Grafana Labs moved to in April 2021. Several security features, including single sign-on and field-level security, sit in paid Elastic tiers, and Loki has no built-in authentication layer, so a reverse proxy is required. Confirm the licence, the feature tier and the support terms with each vendor against your procurement policy before the design is chosen.

How DeployOpen scopes, delivers and supports Besu log pipelines

DeployOpen plans, builds, documents and supports log pipelines for Hyperledger Besu nodes inside the client's own infrastructure. The work starts with a written scope that chooses between the Elastic Stack and Grafana Loki, based on the monitoring stack and retention rules the operator already runs.

How we scope

The written scope names which Besu nodes, roles and log outputs are in the pipeline, such as validators, RPC nodes and boot nodes.
We confirm whether the operator will run Filebeat with Logstash or Grafana Alloy with Loki, and which log format each node emits.
The scope sets retention periods, storage location and access rules for log data, agreed with your compliance and security owners before build starts.

How we build

We configure log shipping on each node, then test parsing against the Besu log format your version actually writes, not a sample.
We validate Grok patterns or Alloy pipelines against real node output, including startup, peer and block import messages.
We set up the retention, tenant and authentication controls in the chosen stack, and check that the reverse proxy or access layer is in place.

How we hand over

You receive configuration files, pipeline definitions, dashboard and index settings, and a record of the versions and tests used.
The handover document explains how to change log formats after a Besu upgrade and how to restore logs from backup.
We walk your operations team through alerting on missing logs, ingestion errors and storage thresholds before sign-off.

How we support

Ongoing support, upgrade checks and log pipeline changes are available as a retainer after handover.
Upgrades to Besu, Elastic or Loki are reviewed against the pipeline before they reach production nodes.
Changes to scope, node count or retention rules are quoted from a new written scope rather than absorbed silently into the original engagement.

Hyperledger Besu log management with Elastic Stack or Grafana Loki