Why a Graylog journal troubleshooting plan matters before the disk fills
If journal utilization on your Graylog cluster is climbing, the question is how to bring it back down before the message journal runs out of room. Start by identifying which stage is slow: the search backend, a pipeline or output that cannot keep pace, or the disk under the journal. Each cause calls for a different fix, and treating them as one problem is the usual reason a rising journal keeps rising.
The message journal is the on-disk buffer that sits between Graylog's input processing and its indexing. When a Graylog node accepts a message, it writes the message to the journal first and marks it as committed only after the message has been processed and written to the search backend. The Graylog system architecture guidance advises giving the journal 3 to 5 times daily ingest on SSDs, so a node that ingests a given volume each day needs journal capacity sized as a multiple of that volume. Confirm the journal directory, its size limit and its disk in your own configuration before you plan anything, because the sizing guidance is a starting point, not a substitute for measuring your ingest.
Teams need this runbook before the first incident, not during it. Write down who watches journal utilization, which thresholds trigger action, which disk holds the journal, and who can change pipeline rules or add capacity. Once the journal is full, uncommitted messages can be lost, so the time to decide the response is while the journal is still filling. The plan should also cover how you will confirm the cluster is healthy again, and how you will record what happened for later tuning.
Which bottleneck is filling the journal
A growing journal means messages are arriving faster than Graylog can process and write them out. The four causes below look similar on the dashboard, so the first check on each card separates them before you change any configuration. Work through them in order of what you can observe quickly.
Slow or unreachable search backend
The output side stalls. When the Data Node or OpenSearch cluster rejects or delays bulk writes, Graylog's output buffer fills and messages pile up in the journal. Check the cluster health and the Graylog server log for indexer timeouts or connection errors, then confirm that the backend answers requests from every Graylog node, not only from the one you are logged into. A red or yellow cluster status points here before any pipeline work.
Processing bottleneck in pipelines
Expensive rules slow the process buffer. Pipeline rules that run heavy regular expressions, lookups or many stages can keep the process buffer full even while the backend is healthy. The symptom is high CPU on Graylog nodes with the output buffer mostly empty. Check which pipeline stages changed recently, and compare processing time with and without a suspect rule on a test stream before you tune buffers or add nodes.
Disk space pressure on the journal volume
The journal needs room to grow. The message journal lives on local disk on each Graylog node, and a full volume stops the node from accepting or committing new messages. Check free space on the journal path first, then confirm the journal is sized for the expected ingest. The architecture guidance suggests planning the journal at 3 to 5 times daily ingest on SSDs, so compare that figure with what you provisioned.
High ingest exceeding node capacity
Volume outgrew the cluster. Sometimes every component is healthy and simply more messages arrive than the nodes can process. The symptom is a steady climb in journal utilization that tracks a rise in incoming messages, often after a new source is onboarded. Compare current ingest with the sizing you planned, check whether load is spread evenly across the Graylog nodes behind the load balancer, and confirm that follower nodes have is_leader set to false.
Diagnosis paths by symptom
When the journal climbs and the Graylog server logs show indexing failures or timeouts, the first question is whether the search backend is still taking writes at all. Graylog stores log data in Data Node or self-managed OpenSearch, so a stalled cluster will hold messages in the journal until it recovers. Start at the backend, not at Graylog, because tuning the server will not help if nothing can be written.
- Cluster health: Check the backend cluster status and whether any nodes have left the cluster. A red or repeatedly changing state points to the backend, not to Graylog.
- Disk watermarks: Look for shards that cannot be allocated or writes blocked because a data node is over its disk limit. Free space or add capacity before anything else.
- Write rejections: Search the Graylog server log and the backend log for bulk rejections, circuit breaker errors and timeouts. Their timing shows whether the stall started at a specific event such as a snapshot or a merge.
- Index state: Confirm the write index for each index set exists and is not closed or read-only, and that rotation is still running according to the index model.
- Network and TLS: Confirm the Graylog nodes can still reach the backend endpoints and that no certificate has expired.
Once writes resume, watch the journal drain. If it does not drain, the backend is still limiting throughput, and the next tab's checks apply.
Sometimes the backend accepts writes without trouble, but messages move through Graylog more slowly than they arrive. The journal then grows steadily while the search cluster shows idle capacity. In this case the bottleneck usually sits inside Graylog itself, in the processing buffer, the output buffer or the processing pipelines that parse and enrich each message.
- Buffer utilization: Compare the process buffer and output buffer fill levels on each node. A full process buffer with a quiet output buffer points to pipeline work. A full output buffer points to slow writes to the backend.
- Pipeline rules: Review the rules attached to the busiest streams for regular expressions, lookups against external services and loops over large fields. These are the common causes of per-message cost.
- Lookup and external calls: Check whether any pipeline function waits on a remote service. Slow lookups can hold back every message behind them.
- Node balance: Confirm the load balancer spreads input traffic across all nodes and that one node is not carrying most of the journal.
- Recent changes: Compare the timing of new or edited pipeline rules, content packs and input configuration with the start of the slowdown.
If the pipeline work is clearly the limit, tuning the buffers and rules is the next step. Compare the findings with the sibling guide on tuning processing and output buffers before changing settings.
A journal volume close to full is the most urgent symptom because the journal is the only copy of messages that have been received but not yet written. If the volume fills, Graylog cannot keep accepting input safely, and messages that were not committed can be lost. Treat this as a disk problem first and a performance problem second, and act before the volume reaches its limit.
- Free space on the journal volume: Confirm how much space remains on the file system that holds the message journal directory, and whether other services write to the same volume.
- Journal size limits: Check the
message_journal_max_sizeandmessage_journal_max_agesettings inserver.confon each node. The journal cannot grow past the size limit, and it will not be written to disk beyond that point, so the limit must fit the volume with headroom. - Sizing against ingest: Compare the configured journal with the daily ingest. Graylog's system architecture guidance advises giving the message journal 3 to 5 times daily ingest on SSDs, so a volume sized below that range needs a documented reason.
- Disk type and I/O: Verify the journal sits on SSD storage and that the volume is not shared with a noisy workload.
- Immediate relief: If space is short, reduce incoming volume at the sources or pause non-critical inputs while you add capacity. Do not delete journal segments by hand.
Record the free space trend. A slow decline suggests undersized storage, while a sudden drop points to a burst of input or a runaway source.
Some journals rise during predictable busy periods and recover afterward. This pattern is often normal when the peak stays inside the journal's capacity. It becomes a risk when the recovery time gets longer each day, or when the peak starts to exceed what the nodes can process. The goal is to separate a sizing gap from a processing gap and to confirm the peak is not hiding a backend limit.
- Peak timing: Map the journal level against the clock for several days. Note when it starts rising, when it peaks and how long it takes to return to baseline.
- Source behavior: Identify which inputs or sources drive the peak, such as batch jobs, scheduled scans or end-of-day application logs. Their volume often explains the timing.
- Drain rate after the peak: If the journal drains quickly, the nodes have headroom. If it drains slowly, the nodes or backend are near their limit even when the peak is over.
- Backend behavior during the peak: Check whether the search backend slows, rejects writes or runs maintenance tasks such as merges and snapshots during the same window.
- Capacity trend: Compare ingest across weeks. A steadily rising peak suggests the current node count will not last, and capacity planning should start before the next growth step.
If the peak is predictable and drains fully, document the pattern and set alerts on the drain time. If the recovery time keeps growing, plan additional capacity with the sizing guidance in mind.
Fixes compared: when each one fits
This table compares the main ways to clear a filling Graylog journal, with the trade-off and risk of each. Read it by first confirming the bottleneck (search backend, pipelines, or disk), then choosing the fix that addresses that bottleneck rather than the symptom. Most journal growth is a symptom, so the cheapest change is often a diagnosis step, not a configuration change.
| Fix | Best when | Trade-off | Main risk |
|---|---|---|---|
| Fix or scale the search backend | Indexing is slow or the Data Node or OpenSearch cluster is unreachable or red | Needs cluster capacity or repair work, and may involve shard or disk changes | Stalled writes can back up the journal until the backend recovers |
| Simplify pipeline rules | Processing is slow and a few rules or regular expressions dominate CPU | Requires rule review and testing, with some parsing logic rewritten | Changed parsing can silently drop or misroute fields if not tested |
| Add Graylog server nodes | Processing is CPU bound across all nodes after rules are reviewed | Adds cost and load balancer and configuration work, see the buffer tuning topic | Added nodes can overload a backend that is already the real bottleneck |
| Enlarge the journal volume or size setting | Traffic spikes are short and the backend recovers on its own | Buys time without removing the cause, and needs disk planning | A larger journal that fills anyway can hide a growing problem until disk runs out |
| Reduce ingest at the source | Noisy sources, debug logs or duplicate inputs produce volume nobody needs | Requires coordination with log source owners and may lose data some teams want | Filtering too aggressively can remove events needed for investigations or audits |
Recovering a journal that is near capacity
This sequence takes a Graylog journal from near capacity back to normal operation by finding the stalled stage first, protecting the journal disk, and then correcting the capacity gap so the problem does not return.
Confirm which stage is the bottleneck
Start by reading the journal utilization metric alongside the process and output buffer levels on each Graylog node. If the journal is filling while the input buffers are full, processing is the constraint. If the output buffers are full, the search backend is refusing writes. Whoever runs the cluster should record which stage is stalled and which node it is on before changing anything. This step is done when the team can name one stalled stage and show the metric readings that support it.
Protect the journal volume from filling
Before changing settings, check free space on the disk that holds the message journal, since a full disk stops the journal from accepting new messages and can cause uncommitted messages to be lost. Free space by removing unrelated files or moving them off the volume, and confirm that no other service writes to the same disk. Do not delete journal files by hand. The check is complete when free space is stable and the team has confirmed the journal path and its headroom on every node.
Restore search backend health
If the output buffers are full, the Data Node or OpenSearch cluster is the likely cause. Review cluster health, unassigned shards, disk watermarks, and whether indexing requests are being rejected or timing out. Fix the cause in the backend, such as freeing disk, restarting a failed data node, or resolving a split cluster, and confirm the backend returns to a healthy state. Verify by watching Graylog output throughput recover and checking the server logs for new indexing errors.
Relieve the processing stage
If processing is the bottleneck, look for expensive pipeline rules, heavy regular expressions and extractors that run on every message. Temporarily disable or narrow the most costly rules, or route high-volume sources to a stream that skips them, after the team agrees on the trade-off with the owners of those rules. Then check processing buffer usage on each node. The step is complete when the processing buffers drain and the journal stops growing for a sustained period.
Verify the journal drains
Watch journal utilization over a period long enough to show a trend, not a single reading. A recovering journal should fall steadily while ingest continues. Confirm that no new errors appear in the server logs, that output and processing buffers stay below full, and that all nodes report the same picture. If utilization rises again once the fix is removed, the cause has not been found and the team should return to the first step.
Document the cause and the fix
Record which stage stalled, what the readings were, what was changed, and how long the recovery took on the team's own timeline. Keep the notes with the cluster runbook so the next responder can compare readings. This step produces a short incident record that names the stage, the change, and the metric that confirmed recovery.
Close the capacity gap
Once the journal is stable, compare daily ingest and retention against the storage and journal sizing the team planned. Graylog's system architecture guidance advises sizing log storage as daily ingest times retention days times 120 percent, and giving the message journal 3 to 5 times daily ingest on SSDs. Confirm the current figures with your own measurements, then decide whether to add disk, add nodes, reduce noise at the source, or change retention. Verify the result by recalculating the same measurements after the change.
How uncommitted messages can be dropped and how to prevent it
The message journal is Graylog's on-disk buffer between input and processing. A message sits in the journal until the server has processed it and committed it to the search backend. Messages that have been read but not yet committed are the uncommitted messages, and they are the ones at risk. If the journal fills, or if the volume holding it fails or is lost, those messages can be lost before they ever reach an index. A slow or unreachable Data Node or OpenSearch cluster is the usual reason the journal grows, which is why the journal is a symptom to diagnose rather than a setting to simply enlarge.
Prevention starts with sizing. The Graylog system architecture guidance advises giving the journal 3 to 5 times daily ingest on SSDs, so the journal is sized from your measured ingest rather than from a default. Pair that with monitoring the journal utilization metric and alerting well before it approaches capacity, so the team has time to act while messages are still flowing. Keep the journal on a volume with enough free space and confirm how the message_journal_max_size and message_journal_max_age settings in server.conf interact with your disk, since a journal that is capped too high can exhaust the disk it shares with the operating system.
Backups protect against the volume failure case, but only if they have been restored in practice. Run a restore test that confirms the server can recover its configuration and resume processing after a disk loss, and record how long that takes. Confirm the exact drop behavior when the journal is full, including whether new messages are rejected or old uncommitted ones are discarded, in the official Graylog documentation for your version. Check that version's Graylog docs for index behavior, and treat the answer as a planning input, not an assumption.
Security and operational considerations during an incident
Journal incidents tempt teams to make fast changes on production nodes and to open up backends for diagnosis. The cards below cover the risks that appear while you work the fix, and the checks that keep the recovery from creating a second incident.
Change control on production nodes
Raising message_journal_max_size, changing processor counts or editing server.conf on a live node can stall ingestion or change how messages are committed. Record each change with the node name, the old value, the new value and the reason before you make it. Keep the change tied to the journal alert that triggered it, so a later reviewer can see why the setting differs from the documented baseline. Ask who approves an emergency change before the work starts, not after.
Search backend exposure during diagnosis
When the search backend looks slow or unreachable, it is tempting to open its HTTP port to a workstation or disable TLS verification to run queries. Keep the Data Node or OpenSearch endpoint on the internal network, use the credentials the cluster already defines, and close any temporary firewall rule when the investigation ends. Check that the backend still enforces TLS for node and client traffic after the fix, not only during testing.
MongoDB access while investigating
MongoDB holds Graylog configuration, users and metadata, so an incident is a poor reason to expose its port or run ad hoc queries with a shared administrative account. Use a named account with the least rights needed to read the state you are checking. A journal problem rarely requires writing to MongoDB, so treat any write during an incident as a change that needs a record.
Secrets in support bundles
A support bundle gathered during a journal incident can include configuration files that contain the password_secret, root_password_sha2, database URIs and certificate material. Before you send the bundle to a vendor or community forum, review each file and remove or replace those values. Keep the unredacted copy in the incident store with restricted access, and record who received the redacted version.
Access for the people running the fix
A journal recovery often needs an engineer on the server, a Graylog administrator and someone with search backend rights, which can give one person more access than the task needs. Assign each role separately, remove temporary elevated access once the journal drains, and confirm that the Graylog administrator account is not shared. Review the audit trail for admin actions afterward so the incident record matches what was actually done.
Data loss risk from uncommitted messages
Messages in the journal are not yet written to the index, so deleting journal segments, reinstalling a node or removing its data directory can drop them. Before any disk cleanup or node rebuild, confirm the journal utilization and the state of the search backend, and take a snapshot where the platform supports it. Treat a journal reset as a last resort that the incident owner signs off on, with the loss window written down.
Questions teams ask before fixing a Graylog journal
How much does an outside fix for a full Graylog journal cost?
No price applies to this work in general, because DeployOpen does not publish a price list and a quote follows from the written scope. The cost depends on how many nodes are affected, whether the search backend is Graylog Data Node or self-managed OpenSearch, how much of the cause is already known, and whether the fix is a one-time repair or ongoing support. Ask the provider to split the quote into diagnosis, remediation, verification and any follow-up support, and to state which items are one-time and which recur. Most engagements are fixed-scope, so the written scope should name the systems covered and the deliverables before work starts.
How long does it take to recover a full Graylog journal?
Recovery time depends on the cause and on how much backlog has built up. The main drivers are how quickly the search backend can accept writes again, how many messages sit in the journal, the processing capacity of each Graylog node, and how expensive the pipeline rules are to run. Disk space also matters, because the journal needs room to drain. Ask the provider for a schedule that separates diagnosis from remediation, and confirm which checks must pass before the journal is considered healthy. Your own journal sizing and ingest figures will give the most reliable starting point.
What do we need to provide before someone can fix the journal?
Before work starts, the team should provide read access to the Graylog cluster and its search backend, access to the host metrics and disk usage for each Graylog node, and the current server configuration with secrets redacted. Useful inputs include recent ingest rates, the pipeline rules and streams that changed recently, and the timeline of when the journal began to grow. Access to the Graylog web interface and a maintenance window for any restart or rolling change should also be agreed. Confirm with the provider how they expect credentials to be shared and stored during the engagement.
Can our internal team fix a full Graylog journal alone?
Often yes, if the team has someone who can read Graylog node logs, check the search backend health, and make configuration changes under change control. The work usually follows a fixed order: confirm the backend is reachable and accepting writes, check whether processing or output is the bottleneck, review disk space for the journal, and only then change buffers or pipelines. The harder cases involve a multi-node cluster with a separate Data Node cluster, or repeated journal growth that points to a design problem. Internal teams should weigh the time needed for a repeatable diagnosis against the risk of dropping uncommitted messages during trial changes.
Does Graylog Open or an Enterprise license change the troubleshooting options?
The core journal and buffer checks are available in Graylog Open, which is licensed under SSPL-1.0 and is not an OSI-approved open source license. The license matters more for features that help prevent recurrence. Graylog Enterprise adds a data lake with archive restore, data tiering and a correlation engine, plus 24/7 support, and Graylog Security adds SIEM features. Data tiering can reduce pressure on hot storage, which affects backend capacity. Confirm with Graylog which edition covers each feature you plan to rely on, and check the current pricing terms on the Graylog pricing page before deciding.
Which Graylog and OpenSearch versions matter when fixing the journal?
The Graylog version and the search backend version both shape the fix. Graylog 7.0 to 7.2 need MongoDB 7.x up to 8.2.x and support OpenSearch from 1.1.x up to 3.7.x, while Graylog Security needs OpenSearch 2.0 or later. Use of OpenSearch 1.x is deprecated since Graylog 7.0 and will be removed in Graylog 8.0. Before changing anything, confirm the running versions against the compatibility matrix, because an unsupported pairing can look like a journal problem. Any version upgrade needed to fix the issue should be planned as its own change.
Could a full journal cause us to lose messages, and how do we check?
Yes. Messages held in the journal are not yet committed to the search backend, so if the journal fills or is discarded, uncommitted messages can be lost. Check the journal utilization and the age of the oldest uncommitted message on each node, and confirm the journal has enough disk space to absorb the current ingest. Verify that the search backend is writable before adjusting anything, and keep a record of the settings you change. Ask any provider to confirm how they will protect uncommitted messages during the work and what evidence they will hand over at the end.
What a fixed-scope journal engagement covers
A fixed-scope journal engagement starts with diagnosis on your own Graylog cluster, not with a configuration change. The written scope would name the Graylog server nodes, the search backend (Graylog Data Node or self-managed OpenSearch), the MongoDB replica set, the input and pipeline set involved, and the disks holding the journal directory. Diagnosis would confirm which of the causes in this topic applies: a slow or unreachable search backend, a processing bottleneck, or disk pressure. It would also record the journal utilization trend, the message_journal_enabled and journal size settings in use, and whether any messages were already dropped. Each finding would be tied to evidence you can review later.
Remediation follows the agreed diagnosis and is delivered in your own accounts, data centers or air-gapped network. The deliverables would normally list the specific changes (for example, backend capacity, pipeline rule adjustments, journal sizing or disk expansion), the rollback steps for each change, and the verification checks that show utilization returning to a steady level under normal ingest. Verification would also confirm that no uncommitted messages were lost during the incident. Handover covers a written runbook, the alert thresholds you agreed to adopt, and a walkthrough with the engineers who will operate the cluster. Ongoing monitoring, such as periodic review of journal metrics and upgrade-related checks, can be added as an optional retainer.
Before any work begins, ask for a written statement of scope, deliverables, access requirements and timeline, and confirm which of these depend on your change windows or vendor support status. DeployOpen does not publish a price list, so the quote follows from that written scope and is the document to compare against an internal estimate. Ask what is excluded, such as hardware purchases, licensing for Graylog Enterprise features or Graylog Security, and work on the search backend that sits outside the agreed boundary. Confirm who will hold credentials and where diagnostic data is stored, since support bundles and logs can contain secrets that need redacting before they leave your environment.
What to confirm in your environment before you start
This sequence checks the four facts that decide how a journal problem can be fixed: the software versions in use, the journal sizing set at deployment, the disk underneath it, and whether a recent backup exists. Work through it before changing any setting.
Record the Graylog server version in use
The engineer on call or the platform owner reads the installed Graylog server version from the package manager, the Docker image tag, or the About page in the web interface, and writes it down with the node names. The check is done when every Graylog node reports the same version and the number is recorded in the change ticket. Mixed versions across nodes are a finding to resolve before any journal work.
Confirm the search backend version and type
Identify whether the cluster runs Graylog Data Node or self-managed OpenSearch, then read its version from the cluster health or node information endpoint. Compare both versions against the Graylog compatibility matrix for the Graylog release in use. Note that OpenSearch 1.x use has been deprecated since Graylog 7.0 and is slated for removal in Graylog 8.0. The check is done when the pair is documented as supported or flagged for an upgrade plan.
Check MongoDB version against the matrix
Read the MongoDB version on the replica set members with the shell or the monitoring tooling already in place, and compare it with the MongoDB range the compatibility matrix lists for the Graylog release. A journal problem can coincide with a database that is out of range, so this check keeps it from being misread as a Graylog fault. The check is done when the MongoDB version is recorded beside the Graylog version.
Retrieve the journal settings applied at deployment
Open the Graylog server configuration file on each node and read the message journal directory and size settings, including any maximum journal size. Compare the configured size with the ingest rate and retention in the design document. The Graylog architecture guidance suggests sizing the message journal at 3 to 5 times daily ingest on SSDs, so record how the deployed value was derived. The check is done when each node's journal settings match, or the difference is written down.
Verify the disk type and free space under the journal
On each Graylog node, identify the device that holds the journal directory, confirm with the platform team whether it is an SSD, and check free space with the filesystem usage command. Note any shared volume that also hosts logs, the operating system, or other services, because those compete with the journal for space. The check is done when the device type, mount point and free capacity are recorded for every node.
Review the retention and index rotation settings
In the Graylog web interface, open the index set configuration for each stream and record the rotation strategy (message count, size or time) and the retention strategy that deletes the oldest indices. Estimate, from current ingest, how much data the search backend holds, and confirm that the backend disk has room for the configured retention. The check is done when rotation and retention are written down for each index set and the backend disk margin is known.
Confirm a recent, tested backup exists
Ask the owner of the backup process for the date and type of the latest MongoDB backup and the latest Data Node or OpenSearch snapshot, plus the copy of the server configuration and certificates. Confirm that someone has restored at least one of them in a test environment. The check is done when a dated backup for each component is named and its restore status is known, because a journal fix may require a restart or a rebuild.
Assign owners and agree the change window
Name the person who will change Graylog settings, the person who will watch the backend and disk during the work, and the person who can approve a rollback. Record the agreed maintenance window in the ticket and confirm it with the teams that depend on Graylog alerts. The check is done when each role has a named owner and the team confirms in writing that they accept the window.
How DeployOpen scopes and delivers Graylog journal work
DeployOpen starts by reading the journal symptoms in your own Graylog environment, then fixes the cause in the search backend, pipelines, storage or node layout, not just the journal setting. Work is agreed as a fixed scope with written deliverables, and each change is tested and documented before handover.
How we scope
- Review journal utilization history, node counts, ingest rates and disk layout for your Graylog cluster before quoting.
- Agree in writing which causes are in scope, such as search backend health, pipeline cost, disk capacity or node sizing.
- Confirm which Graylog edition, Data Node or OpenSearch version and access to hosts your team will provide.
How we build
- Reproduce the backlog pattern in a test copy of your setup and measure where messages wait before changing production.
- Apply fixes one at a time, such as search backend capacity, pipeline rule changes or journal disk moves, with a rollback step for each.
- Verify that the journal drains, buffers stay steady and no uncommitted messages are lost during the change window.
How we hand over
- Deliver a written runbook covering alert thresholds for journal utilization, the diagnosis order and who acts on each signal.
- Document every configuration change, its reason and the test result that showed it worked.
- Walk your operations team through the runbook and confirm they can repeat the diagnosis without DeployOpen present.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer for clusters that keep growing.
- Retainer scope covers journal reviews after ingest changes, new pipelines or Graylog version upgrades, agreed in writing.
- Ask for the support terms, response expectations and included upgrade work in the quote before you sign.
