What high availability means for Graylog
High availability in Graylog is a set of separate tiers, and each one fails differently. Graylog server nodes receive, process and index messages. The MongoDB replica set holds configuration and metadata, so losing its primary triggers an election and pauses configuration changes until a new primary is chosen. The Data Node cluster stores and searches logs. The load balancer and input sources, such as syslog, GELF and Beats listeners, decide where senders connect and what happens to traffic while a node restarts. Plan and test each tier on its own.
Teams usually need this when log loss or an ingestion gap is unacceptable, for example when logs serve as security evidence, audit records or the only way to diagnose a production incident. For medium to large deployments, the Graylog multi-node guide recommends three Graylog and MongoDB nodes with MongoDB run as a replica set, three Data Nodes on their own cluster, a load balancer, and is_leader set to false on follower Graylog nodes. Treat that layout as the starting point, then confirm it against your daily volume and the compatibility matrix for your target version.
Adding nodes is the easy part. HA needs deliberate design in four areas. First, sender retry behavior: shippers that buffer and resend over TCP can hold messages through a node restart, while UDP syslog senders may lose messages sent during it. Second, journal sizing: the message journal must hold the backlog that builds while indexing is slowed or paused, and the 3 to 5 times daily ingest guidance on SSDs is a baseline to validate against your own outage scenarios. Third, leader roles: is_leader must match the intended role on every node. Fourth, MongoDB quorum: if a majority of replica set members is lost, configuration writes stop until quorum returns.
HA does not replace backup. A replica set protects against losing a member. It does not protect against a bad configuration change, an accidental index deletion or ransomware that reaches every copy. Keep tested backups of MongoDB and of the Data Node or OpenSearch storage on a schedule your retention requirements support. A single-node Graylog can also be a valid choice for smaller or lower-criticality environments where a short outage is tolerable and the added nodes, licenses and operating effort are hard to justify. Make that decision per log source and document the recovery expectations you accept.
Components and settings involved
Graylog high availability depends on six components working together. Each one has a setting that decides whether ingestion and search keep running when a node fails.
Graylog server nodes
Run three Graylog server nodes so that inputs, processing and the web interface keep working when one node stops. Set is_leader to false on follower nodes so that only one node holds the leader role. After each restart or upgrade, confirm that exactly one node is the leader and that every node can reach MongoDB and the search backend.
MongoDB replica set
MongoDB stores configuration, users, dashboards and other metadata, not log messages. Deploy three members as a replica set so that a new primary can be elected when one member fails. Without a replica set, a single MongoDB failure can block configuration changes and stop Graylog nodes from starting cleanly. Check the member count, test the failover and confirm the MongoDB version against the Graylog compatibility matrix.
Data Node cluster or OpenSearch
Graylog recommends Data Node as the search backend, and the multi-node guidance is three Data Nodes on their own cluster, separate from the Graylog servers. This tier stores indexed log data, so its node count and replica settings decide whether writes and searches survive a node loss. If you use self-managed OpenSearch instead, confirm the version, since OpenSearch 1.x is deprecated from Graylog 7.0 and will be removed in Graylog 8.0.
Message journal
The message journal is a disk-based buffer on each Graylog node that holds accepted messages until they are written to the search backend. Graylog's architecture guidance advises sizing it at three to five times daily ingest on SSDs. If the journal is undersized, a Data Node outage or slow indexing can force inputs to back up or lose messages. Check journal usage and free disk space during each failover test.
Load balancer for inputs and web
A load balancer sits in front of the Graylog inputs and the web interface, so senders and users connect to one address while individual nodes come and go. Configure health checks that remove a failed node from rotation. Confirm that the balancer supports the transport each input uses, such as TCP or UDP for syslog. Without it, each source points at one node, and that node's failure stops its logs.
Index sets and rotation
Index sets decide which streams write to which indices, how the shard and replica settings apply, and when indices rotate and are removed. Rotation can trigger on message count, size or time, and retention in Graylog Open deletes the oldest indices. Size storage from daily ingest times retention days times 120 percent. Use separate index sets per log type so that a noisy source cannot push out retention for the others.
Key design decisions and trade-offs
The choices a team makes before building Graylog high availability, with when each option fits and the operational cost it adds.
| Decision | When it fits | Operational cost it adds |
|---|---|---|
| Search backend: Graylog Data Node or self-managed OpenSearch | Data Node is the option Graylog recommends for new builds. Self-managed OpenSearch fits teams that already run it and need its tooling, and it must be a version the compatibility matrix lists for your Graylog release. | Data Node adds a separate three-node cluster to size, patch and back up. Self-managed OpenSearch adds ownership of cluster security, version pairing and upgrades. OpenSearch 1.x use is deprecated since Graylog 7.0 and will be removed in Graylog 8.0. |
| Load balancer placement and sender targets | Put one virtual address in front of the Graylog server nodes so senders use a single endpoint and a node failure does not change sender configuration. Confirm which input protocols your load balancer can pass through. | Adds a component to size, certificate-manage and monitor. Syslog and UDP inputs need explicit handling, since UDP traffic is not retried by the sender and a balancer may behave differently from TCP. |
| Number of Graylog and MongoDB nodes | Three Graylog nodes and three MongoDB nodes, with MongoDB as a replica set, is the layout the multi-node guide recommends for medium to large deployments. Smaller sites may start with fewer nodes and accept less failure tolerance. | Every extra node adds patching, certificate, monitoring and rolling-upgrade work. A replica set needs an odd voting member count, so two-node layouts cannot survive a node loss without a tie-breaker. |
| MongoDB on the Graylog hosts or on separate hosts | Co-locating MongoDB saves hosts and suits small sites. Separate hosts fit larger sites, where MongoDB resource use or access rules must not compete with Graylog processing. | Co-location means one host failure can hit both a Graylog node and a replica member, and resource contention is harder to see. Separate hosts add more machines, network paths and access rules to manage. |
| Message journal size and disk type | Size the journal from your ingest and the guidance in the system architecture documentation, which advises 3 to 5 times daily ingest on SSDs. Larger journals suit sites where a Data Node outage must not drop input during recovery. | A larger journal uses more disk on every Graylog node and needs monitoring for fill rate. Slower disks can cause backpressure during bursts, so confirm IOPS with your storage team before choosing spinning disks. |
| Virtual machines, containers or bare metal | Virtual machines suit teams that already run hypervisor-based HA and want predictable host behavior. Containers, including Docker Compose, suit teams with a mature container platform. Bare metal fits high sustained ingest with dedicated disks. | VMs add a hypervisor layer to patch and tune. Containers need persistent volume handling for the journal and data paths, and bare metal shifts hardware replacement and imaging work to your team. |
| Failure domain split for the Data Node cluster | Spread the three Data Nodes across separate racks, hosts or availability zones where your site has them, so that one domain failure leaves a quorum of data. Confirm the replica and shard settings with your Graylog version before you rely on them. | Spreading nodes across domains needs low-latency links between them and adds inter-zone traffic. A single-domain layout is simpler to run but loses data availability if that domain fails. |
Build sequence
The ordered path builds the data tier and metadata tier first, then Graylog nodes behind a load balancer, and closes with storage and sender behavior verified end to end.
Confirm the compatibility matrix for your versions
Before ordering hardware, the platform owner checks the Graylog compatibility matrix for the chosen 7.x release against the MongoDB 7.x version and the Data Node or OpenSearch version. Use Data Node as the search backend, since OpenSearch 1.x use is deprecated from Graylog 7.0. Done when the three versions are written down together and each is confirmed as supported in that matrix.
Deploy the three-node MongoDB replica set
The database team installs MongoDB on three separate nodes and initiates them as one replica set, following the Graylog multi-node guide. Graylog keeps configuration and metadata here, so losing the primary must not stop the cluster. Done when the replica set status command shows one primary and two secondaries and a failover test elects a new primary.
Deploy the three-node Data Node cluster
The search team installs Graylog Data Node on three dedicated nodes, separate from the Graylog servers, and joins them into one cluster. Storage for each node should be sized from daily ingest, retention and the 120 percent guidance in the system architecture documentation. Done when the Data Node cluster health check reports a healthy cluster with all three members present.
Install three Graylog nodes with shared configuration
The platform team installs Graylog on three nodes using one shared server configuration that points at the MongoDB replica set and the Data Node cluster. Set is_leader to false on the two follower nodes and leave exactly one node as leader. Done when the system node pages in the Graylog web interface list all three nodes as active and exactly one shows leader status.
Configure load balancer health checks and routing
The network team places a load balancer in front of the three Graylog nodes and routes both the input ports and the web interface through it. Health checks should target a Graylog endpoint that fails when a node is unhealthy, and the team confirms which endpoint their chosen balancer can use. Done when a node taken out of service stops receiving traffic and the web interface still loads through the balancer address.
Size and set the message journal on each node
Each Graylog node gets its own message journal on fast local SSD storage, sized at 3 to 5 times daily ingest according to the system architecture guidance. Journal size protects messages while indexing is slow or a node restarts. Done when the journal settings are applied on all three nodes and the node pages show journal capacity matching the planned value.
Point senders at the load balancer and test failover
Configure log shippers and senders to use the load balancer address for each input, with retry and reconnect settings so brief interruptions do not drop data. Then send a test message through the load balancer, stop one Graylog node, and send another. Done when both test messages are searchable in the expected index set and the failed node's absence does not interrupt ingestion.
Finish index set and retention configuration
Create index sets for each log type with rotation and retention settings chosen from the planned daily ingest and retention period, then confirm the Data Node storage can hold that retention. Use the index model settings for rotation by message count, size or time and confirm the oldest indices are deleted as configured. Done when a test index rotates on schedule and the index set page shows the intended retention.
Verifying that ingestion survives a failure
A high availability design is only proven when each failure has been induced and the message counts have been checked. Run three core tests in a staging copy of the topology before go-live. Stop the MongoDB primary and confirm that the replica set elects a new primary and that Graylog keeps serving the web interface and API. Stop one Graylog server node and confirm that inputs keep accepting messages through the load balancer. Stop one Data Node and confirm that searches return results and indexing continues. Before each test, record how many messages the sender emitted and how many the test window contains, so every test ends with a comparison rather than an impression.
The journal and load balancer tests cover the failures that a healthy-looking cluster hides. For a short outage, stop indexing or a Data Node for a period longer than a routine restart, then watch journal utilization rise on the affected node. After recovery, confirm the journal drains and that the sent and stored counts match. Size the outage test against the journal sizing you planned (3 to 5 times daily ingest on SSDs, per Graylog's architecture guidance), and confirm that the journal does not fill during the test. For the load balancer, stop the Graylog service on one node and confirm its health check marks the node down within the interval you configured, then confirm that no input traffic reaches it until it returns.
Record every run in a test log: date, Graylog and MongoDB versions, Data Node or OpenSearch version, node count, which node was stopped, the sent and stored counts, journal depth over time, and the time each node was marked down or back up. Rerun the full set after every Graylog upgrade, MongoDB or Data Node change, or topology change, because those changes can alter failover and journal behavior. Keep the log with your change records so that an auditor or on-call engineer can see what was proven and when.
When a test fails, decide whether the fault is in the design or the sender before changing the cluster. Start by checking the sender: does it retry on connection errors, does it buffer when the target is unreachable, and does its own counter match what it reports as sent? Then rerun the same test with the sender pointed directly at a healthy Graylog node. If messages arrive there, the load balancer or health check configuration is the likely fault. If loss persists, check the journal, the Data Node write status and whether follower nodes are configured with is_leader set to false. Confirm each sender behavior with its owner, because many shippers drop data during restarts unless they are configured to retry.
Access, secrets and network controls specific to HA
Moving from one Graylog node to a clustered design adds traffic paths, certificates and keys that a single node never exposed. These controls cover the parts of the setup that high availability changes, while general hardening is covered in the Graylog security hardening topic.
Authenticated and encrypted node traffic
Graylog nodes, the MongoDB replica set and the Data Node cluster all exchange data over the network once the design is clustered. Confirm that each connection path uses TLS with certificates the nodes verify, and that no component accepts plaintext connections from peers. Document every path so that none is left open by default.
TLS termination at the load balancer
The load balancer in front of Graylog nodes often terminates TLS for web and API clients, so the certificate and its private key sit on that tier. Decide whether traffic between the balancer and the nodes is re-encrypted, and check that the balancer forwards the client address the nodes expect. Confirm the balancer's health checks reach a path that does not require a login.
Certificate lifecycle across nodes
Each HA component carries its own certificate, and a clustered design multiplies the renewals to track. Record which certificate sits on the balancer, each Graylog node, MongoDB and each Data Node, along with its issuer and expiry date. Agree a renewal procedure that replaces them in a rolling order so that ingestion continues during the change.
Network restriction on database ports
MongoDB and Data Node ports should accept connections only from the Graylog nodes and the administrative hosts that need them, not from user subnets or the balancer. Write these rules as explicit allow lists in the firewall or security groups, and review them whenever a node is added or replaced. Test the rules from a host outside the allow list to confirm they block access.
Password secret and shared credentials
The Graylog password secret and the credentials for MongoDB and Data Node must be supplied at runtime from a secrets store or the host environment, not written into configuration files kept in version control. Confirm that every node reads the same password secret, because nodes with different values cannot share sessions and data. Limit who can read the secret store entries and log access to them.
Replica set key protection
The MongoDB replica set key authenticates members to each other, so its exposure would let an attacker join the set. Store the key file with restrictive ownership and permissions on each member, keep it out of repositories and backups that lack equivalent protection, and rotate it through a planned maintenance window. Confirm the key is identical on all three members before enabling authentication.
Effort and cost drivers
This table lists the inputs that move effort and cost for a Graylog high availability build, what each one changes, and how to keep it manageable, so you can compare scope with your own team's capacity.
| Driver | Why it changes effort or cost | How to keep it down |
|---|---|---|
| Daily ingest and retention | Graylog's system architecture guidance advises sizing log storage as daily ingest times retention days times 120 percent, so storage, Data Node nodes and disks grow directly with both values. | Confirm daily volume per source and the retention each log type needs, and split high-volume sources into separate index sets with shorter retention where policy allows. |
| Message journal sizing | The guidance suggests giving the message journal 3 to 5 times daily ingest on SSDs, which adds fast disk to every Graylog node that buffers inputs. | Size the journal from measured ingest rather than peak estimates, and confirm the disk class with your storage team before ordering hardware or cloud volumes. |
| Inputs and senders that must tolerate reconnects | Each syslog, GELF or Beats source behaves differently when a node or load balancer target fails, so each one needs its own reconnect and failover test. | Inventory every sender, group them by protocol, and test the top sources first. Ask senders whether they support multiple targets or only one address. |
| Number of environments | Lab, staging and production each need the full topology (three Graylog and MongoDB nodes, three Data Nodes and a load balancer), so every environment multiplies build, test and capacity work. | Keep lab and staging smaller only where the failover tests still hold, and ask whether the quote covers all environments or production alone. |
| Network and load balancer changes | Load balancer pools, firewall rules, DNS names and health checks must be created in your data center or cloud, and changes often pass through change control. | Gather network owners and change windows early, and provide a complete list of ports, addresses and health check paths before the build starts. |
| Certificates and identity integration | TLS for inputs, the web interface and Data Node, plus directory or single sign-on integration, add certificate issuance, renewal and group mapping work. | Confirm your certificate authority, renewal process and identity provider before design, and check which SSO options your Graylog edition supports. |
| Migrating existing data when replacing a single node | Moving indices, configuration and MongoDB data from an existing server to the new cluster adds copy, validation and cutover time that depends on the data volume. | Decide early whether historical data must move or can age out, and agree on a retention cutoff so migration covers only what policy requires. |
| Team experience with MongoDB and Data Node | Operating a replica set and a separate search cluster requires skills in backups, failover, upgrades and capacity, and gaps in these areas usually add training or external support time. | Assess the team's current MongoDB and Data Node experience, then decide which runbooks, training and handover sessions to include in the scope. |
| One-time and recurring costs | One-time work covers design, build, failover testing and documentation. Recurring costs include hardware or cloud capacity, patching, upgrade testing, and periodic failover test runs. | Ask any quote to separate one-time work from recurring items, state who pays for cloud or hardware capacity, and confirm how many test runs are included. |
Frequently asked questions
Do you need three Graylog nodes and three MongoDB nodes for high availability?
The Graylog multi-node guidance recommends three Graylog and MongoDB nodes, with MongoDB running as a replica set, for medium to large deployments. Smaller setups can run with fewer nodes, but that is a deliberate risk decision: you accept that a single node failure may interrupt ingestion or search, and the decision should be documented and signed off. The same reference design places three Data Nodes on their own cluster behind a load balancer. Size the node count against daily ingest, retention and the outage your organization can tolerate.
Does Graylog Open support a three-node high availability topology?
The Graylog documentation describes a multi-node architecture with three Graylog and MongoDB nodes, a separate Data Node cluster and a load balancer, with follower Graylog nodes set so they do not act as leader. This topology is documented for the product and does not depend on an edition. Confirm the MongoDB and Data Node or OpenSearch versions against the compatibility matrix for the release you plan to run. Single sign-on options differ between Graylog Open and Enterprise, so check the features you need before you design the authentication layer.
How much does a Graylog high availability build cost and what drives it?
The cost of a build depends on its written scope, so ask for a fixed-scope quote that lists each component. The main cost components are planning and design, the hosts or cloud resources for three Graylog nodes, three MongoDB nodes and three Data Nodes, load balancer and certificate setup, journal and storage sizing, testing and handover. Costs rise with the number of log sources, daily volume, retention length, the number of environments and the change windows you need. Ask which items are one-time and which recur, such as infrastructure and ongoing support, and which items your team provides itself.
How long does a Graylog high availability project take to complete?
The duration depends on scope, the target environment and how ready your team is to supply access, certificates and change windows, so confirm it in a written plan before work starts. Delays usually come from access requests, firewall approvals, certificate issuance and waiting for change windows. Ask the plan to show the phases, the dependencies between them, the test gates and who approves each production change. A plan that names these items gives you a basis for tracking progress against the agreed scope.
What does our internal team need to provide for a Graylog HA build?
Your team provides network access between the nodes, the load balancer and clients, the hosts or cloud accounts where the nodes run, and TLS certificates for inputs, the web interface and the Data Node cluster. Someone with MongoDB replica set experience and someone who can operate Data Node or OpenSearch must be available for design decisions and recovery work. You also need change windows for cutovers, clear ownership of DNS and load balancer settings, and a named approver for production changes. Agree these responsibilities in the scope so that gaps appear early.
Does high availability replace backup and disaster recovery for Graylog?
No. High availability keeps ingestion and search running when one node fails, but it does not protect you from deleted data, corrupted indices, a bad configuration change or a site outage. Back up MongoDB and the Data Node or OpenSearch data, and test restores on a schedule. Disaster recovery is a separate design that decides how current a second site or environment is kept and how quickly service can be restored there. Ask for backup and recovery to be scoped as their own work, with their own acceptance tests.
How DeployOpen scopes, builds and supports Graylog HA
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks, and Graylog high availability is delivered the same way. The work is scoped in writing before it starts, so the node layout, failure behavior and support boundaries are agreed up front.
How we scope
- Confirm the daily ingest volume and retention period, then size the Data Node or OpenSearch storage from that figure.
- Map which log sources must keep flowing through a single node failure and which can tolerate a brief gap.
- Agree the load balancer, DNS and network paths for inputs, the web interface and the REST API in writing.
- Record the Graylog edition, MongoDB version and search backend version against the compatibility matrix before work starts.
How we build
- Deploy three Graylog server nodes and a three-member MongoDB replica set, with exactly one node left as the leader.
- Build the separate three-node Data Node cluster and confirm the message journal is sized on SSDs for the agreed ingest.
- Configure the load-balanced inputs and test sender failover by stopping one node during controlled log replay.
- Document the node roles, journal sizes, index rotation settings and failover test results as part of the build record.
How we hand over
- Deliver a runbook covering node restart order, rolling upgrade steps and how to confirm ingestion after a node loss.
- Provide the configuration inventory, backup locations for MongoDB and Data Node, and restore test evidence.
- Walk your operations team through alerting on journal growth, disk pressure and node health before go-live.
- Hand over access and secrets management details so your team controls credentials after the engagement closes.
How we support
- Offer ongoing support, upgrades and testing as a retainer aligned with the 7.x minor release cadence.
- Plan rolling upgrades across the three nodes and re-check MongoDB and OpenSearch compatibility before each change.
- Review journal behavior, index growth and retention against the original sizing assumptions at agreed intervals.
- Confirm with your own team which support tier, response expectations and maintenance windows apply to your environment.
