Tuning Graylog processing buffers when ingestion falls behind
Buffer tuning changes how many messages each Graylog node works on at once, and it is the right first step when messages queue on existing nodes while CPU and the search backend still have headroom. Graylog moves each message through two stages. The process buffer holds messages read from the journal while processors run extractors, stream routing and pipeline rules. The output buffer holds processed messages until output modules write them to the search backend. Each buffer has its own processor count, set in server.conf, and a slow stage backs up the stage before it. Since Graylog 6.0, the defaults scale with the CPU cores on each node, so check the value your version actually uses before you change anything. Confirm the current keys and defaults in the Graylog configuration documentation for your release.
Tuning comes before scaling out because adding nodes multiplies the same constraints. Start by checking where the time goes. If process buffer utilization is high while the CPU is mostly idle, the limit is usually concurrency, and raising processor counts may help. If CPU is saturated, the cost sits in the work itself: regular expressions in extractors, pipeline rules that run lookups or broad searches on every message, and rules evaluated for streams that do not need them. Those are cheaper to fix in the rules than by adding threads. If the journal grows while the search backend responds slowly, the buffers are not the cause, and the investigation belongs in the journal troubleshooting guide. Change one setting at a time, and record ingest rate and buffer utilization over a representative period before and after each change.
Before you add nodes, confirm the buffer work has been tried. A new Graylog node brings its own share of the multi-node design, including load balancing, is_leader set to false on follower nodes and enough journal space. Sizing guidance in the Graylog system architecture documentation recommends a journal of three to five times daily ingest on SSDs, and that requirement applies to every node that receives traffic. Buffer changes are easier to reverse, but they still need a rollback plan: note the original values, apply changes in a maintenance window where possible, and watch the gl_ metrics in your monitoring stack through a full ingestion cycle. Ask any partner to state their throughput assumptions in writing.
Buffer and processor settings and when each fits
This table compares the processing and output buffer settings that control how many workers a Graylog node runs and how much memory and CPU they take. Read each row as a decision: when to change the setting, what overcommitting it costs, and what to measure before and after the change. Exact configuration keys and default values must be confirmed against the Graylog 7.x configuration reference before any change goes to production.
| Setting | When to change it | Risk of overcommitting CPU or heap | What to measure before and after |
|---|---|---|---|
| Process buffer processor count | Journal is growing while the input buffer is busy and CPU on the node has headroom, so message processing is the bottleneck. | Each processor holds work in memory and competes with search and output threads, which can raise heap pressure and slow searches. | Journal utilization, process buffer usage, CPU per node and message processing rate over a comparable ingest window. |
| Output buffer processor count | Messages are processed quickly but the output buffer fills, usually because the search backend accepts writes more slowly than Graylog produces them. | More writers push harder on Data Node or OpenSearch, which can create indexing backpressure and make the search backend the new bottleneck. | Output buffer usage, indexing latency and rejections on the search backend, and whether the journal drains after the change. |
| CPU-based defaults (since 6.0) | Use the default as the baseline on a new node, and only override it when measurements show a specific bottleneck. | Setting counts higher than the cores available on a shared host leads to thread contention with the web interface and other services on that machine. | Confirm the default that applies to your node size in the release notes for your version, then record CPU and buffer metrics on the unchanged default first. |
| Batch and timeout settings | Only when the docs for your version describe a batch or timeout option that matches the bottleneck you measured, such as output flush behavior. | Large batches increase memory use per flush and can delay messages reaching the search backend, which shows up as lag in searches. | Message latency from ingest to searchable, memory use per node and whether any messages are dropped or retried. |
| Expensive pipeline rules | When process buffer usage is high and pipeline metrics show specific rules consuming most of the processing time. | Adding processors does not fix a slow rule, and extra threads running the same expensive regular expression multiply the CPU cost. | Per-rule execution time and matches per message before and after rule changes, with a fixed sample of inbound traffic. |
| Regular expression extractors and lookups | When extractors or pipeline functions with regular expressions dominate processing time, or patterns backtrack on long messages. | Unanchored or nested patterns can use large amounts of CPU on a few messages, stalling every processor on that node. | CPU time for the extractor or function, processing rate on a representative message sample and journal growth during peak ingest. |
| Adding a Graylog node instead | When CPU on every node is saturated after rule tuning, or when a single node cannot meet the ingest rate even with a balanced configuration. | A new node adds a leader-eligible or follower role to plan around, and a poorly balanced cluster can shift the load rather than remove it. | Per-node CPU and journal utilization after the node joins, and whether the load balancer spreads inbound traffic across all nodes. |
| Search backend capacity | When journal growth coincides with slow searches or indexing rejections rather than with slow processing on Graylog. | Raising Graylog processor counts increases pressure on a backend that is already the limit, which can make journal utilization worse. | Search backend indexing and query latency, disk and heap use on Data Node or OpenSearch, and the journal trend over a full day of ingest. |
Which pipeline rules and regular expressions slow processing
Most processing slowdowns in Graylog trace back to a small set of rule patterns rather than to raw message volume. The cards below cover the usual suspects, how to confirm each one in your own cluster, and the change that tends to help. Work through them in order before adding nodes, because a fix to one rule often recovers more throughput than new hardware.
Broad or backtracking regular expressions
A pattern with unanchored wildcards, nested quantifiers or alternations that overlap can take far longer on some messages than others. The slowdown appears only on unusual input, which makes it easy to miss in testing. To find it, check which pipeline stages consume the most time in the processing metrics and test suspect patterns against long or malformed sample messages. The usual fix is to anchor the pattern, match a literal prefix first, and replace greedy wildcards with specific character classes.
Lookups that call external services
A lookup table backed by a data adapter or cache that queries an external API, database or DNS server waits on that service for each message it evaluates. When the service is slow or unreachable, processing threads queue up behind it. Confirm this by comparing processing throughput with and without the lookup enabled, and by reviewing the cache settings and timeouts on the adapter. Ask the team that owns the external service what latency and rate limits apply, then size the cache so repeated keys do not hit the service each time.
Many rules evaluated against every message
Pipelines that chain dozens of rules with no early exit force every message through all of them, including messages that no rule will change. Each rule adds condition checks and function calls to the processing path. Review each pipeline's stage order and count how many rules run per message on typical traffic. Grouping rules by source, moving filters to the front of the stage, and using stage-level rule conditions so that only relevant messages enter a stage usually reduces work without changing results.
Parsing that runs on every message
Extractors or pipeline rules that parse a message format are often attached to a broad stream or to the default stream, so they run on traffic they were never written for. Parsing a Windows event from a firewall feed wastes time on every message and can also add unexpected fields. Identify the stream each parser is connected to, then route messages with a stream rule that matches the source or a specific field first. Narrowing the stream filter is usually the cheapest change in this list.
Work repeated on identical messages
Rules that recompute the same derived value, such as a hash, a geolocation or a normalized hostname, for every copy of a repeated event waste processing time. This shows up as high cost for chatty sources such as health checks or heartbeat logs. Look for rules that run on message classes with little business value and test whether dropping, sampling or summarizing them upstream is acceptable under your retention and review requirements. Confirm the decision with the teams that rely on those logs before changing the collector.
How to tune buffers without guessing
This sequence separates a buffer problem from a rule problem, a CPU problem or a sizing problem before any setting changes. It ends with a recorded baseline, one tested change at a time and a clear decision on whether to add nodes.
Confirm the bottleneck in the journal
Start with the message journal trend over a window that includes your daily peak ingest. Check whether utilization climbs during peak and drains off peak, and confirm that the search backend and inputs are healthy, since a slow backend also fills the journal. Journal capacity should match the sizing guidance of 3 to 5 times daily ingest on SSDs. You are done when you can state whether the backlog grows during peak and whether it recovers afterward. If it does not grow, stop here, because buffer tuning will not solve the problem.
Establish a baseline ingest rate
Record incoming, processed and output message rates for each node across a typical day and a peak period. Save the time window, input mix and stream volumes with the numbers so later comparisons use the same conditions. Compare the gap between the input rate and the processed rate, because that gap is the work the buffers must absorb. You are done when the baseline is written down, dated and tied to a specific set of inputs, and the team agrees on it.
Check CPU and heap on each node
Review CPU use, heap pressure and garbage collection on every Graylog node, not only the busiest one. Classify each node as CPU-bound, heap-bound or waiting on output. If one node runs hot while the others idle, check how the load balancer and inputs distribute traffic before changing any buffer setting. Since Graylog 6.0, buffer processor defaults are based on CPU count, so confirm the values actually in use on each node. You are done when every node has a recorded classification.
Review pipelines and regular expressions
Inventory the processing pipeline rules, extractors and lookups, and identify those that run against every message without a stream filter. Pay particular attention to rules that use regular expressions, grok patterns or repeated parsing of the same field, since these consume processing time on each message. Use the processing time figures for each rule where your version exposes them. You are done when you have a ranked list of the most expensive rules with the stream each one applies to. Fix or narrow the top items before adding buffer processors, because expensive rules limit throughput whatever the buffer count.
Change one buffer setting at a time
Adjust the process or output buffer processor settings in the node configuration, changing a single setting on a single change window. Confirm the key names and valid values for your Graylog version, and restart nodes using the documented procedure. Check whether any explicit overrides left from an older release still apply, because they can hold a node below its CPU-based default. You are done when a change log records the setting, node, previous value, new value and time of change.
Check the metrics after each change
After every change, compare journal utilization, output latency, processing time and CPU with the baseline for the same period. Confirm the exact metric names in the Prometheus exporter for your version, since the names that matter for throughput, journal and buffers are the ones to alert on. Keep the change only if the journal stops growing at peak and no node is pushed into heap or CPU saturation. You are done when each change has a recorded result that shows it helped, or it has been reverted.
Retest at peak and decide on scale
Repeat the baseline measurement at peak ingest with the tuned settings and the fixed rules. If the journal still grows and every node is CPU-bound, the remaining gain is in capacity, so plan additional Graylog server nodes behind the load balancer. Multi-node deployments use follower nodes with the leader setting disabled, and the scale decision should follow the Graylog multi-node guidance. You are done when the peak result and either a stable journal or a documented scaling decision are recorded.
When tuning stops helping and more nodes are the answer
Tuning stops helping when the node is saturated even after pipeline rules, regular expressions and buffer processor counts have been reviewed. The clearest sign is CPU pinned across all cores for sustained periods while the message journal keeps growing and does not drain at quiet times. If the journal grows during normal load and shrinks only after traffic drops, the node lacks headroom, and a higher processor count will mostly move the contention around. Before adding hardware, confirm the pattern over a full daily cycle, not a single spike, and check the journal and buffer metrics against the same timestamps so you can see which stage is actually backing up.
Adding a Graylog node does not fix a slow search backend. Graylog server nodes process and route messages, but indexing and search run on the Data Node or OpenSearch cluster, so a slow or full backend will hold back the pipeline no matter how many server nodes sit in front of it. The Graylog multi-node guide describes a reference layout for medium to large deployments: three Graylog and MongoDB nodes with MongoDB as a replica set, three Data Nodes on their own cluster, and a load balancer in front of the Graylog nodes, with is_leader set to false on the follower nodes. Use that layout as the target shape, then confirm each piece's role in your own environment before you scale any tier.
Storage and disk are often the real limit, and they are easy to overlook when the complaint is about throughput. Check disk latency and free space on the Data Nodes and on the journal volume, and compare them with the guidance in the Graylog system architecture page, which advises sizing log storage as daily ingest times retention days times 120 percent and giving the message journal 3 to 5 times daily ingest on SSDs. If those numbers are short, more Graylog nodes will only add messages that wait longer. Ask your team or your partner to confirm the current versions of Graylog and the search backend against the Graylog compatibility matrix before any change to the cluster.
Security and operational risks of tuning changes
Buffer and pipeline tuning changes the way Graylog spends CPU, memory and attention, so each change can hurt throughput or hide events. The cards below cover the risks to plan for and the controls that keep tuning reversible, reviewed and visible to the people responsible for detection.
Buffer counts beyond available CPU
Raising the number of process buffer or output buffer processors looks like a quick throughput gain, but each processor needs a CPU core to run. When the count exceeds what the node has, threads compete for CPU, latency rises and the message journal grows faster than it drains. Before raising a count, compare it with the cores on each node and check the node's CPU and journal metrics under real ingest, not a quiet test window. Keep the change to one node first and watch the result before applying it across the cluster.
Lost visibility from disabled rules
Disabling an expensive pipeline rule or regular expression saves CPU, but it also stops the enrichment, routing or alerting that rule performed. Events may still be ingested and stored while the detections that depend on them quietly stop firing. Before disabling a rule, list the streams, dashboards and event definitions that read its output. Where possible, narrow the rule's scope to the sources that need it, and record which detections were affected so the gap can be reviewed later.
Changes without a rollback plan
A tuning change that cannot be reversed quickly turns a slowdown into an incident. Write the previous values for buffer counts, the rule source and the node settings before editing, and confirm who can restore them and how long that takes. Test the rollback on a non-production cluster where possible. Apply one change at a time, so that if throughput drops, the cause is clear and the revert is a single step.
Unaudited pipeline edits
Pipeline rules and stream routing decide which messages are parsed, enriched, dropped or sent to alerts, so an unreviewed edit can change detection coverage without anyone noticing. Keep pipeline source in version control where your process allows it, require a second reviewer for changes to rules that drop or route messages, and compare the active rule set against the reviewed version after each change. Tell the detection owners before any change that alters message flow.
Change records for tuning decisions
A record of why a buffer count, rule or node setting was changed is what lets the next engineer understand the cluster. Log each change with its date, the owner, the metric that prompted it, the before and after values and the test result. Store these records with your operations runbooks and link them to the related incident or capacity review. Without them, teams repeat past experiments and auditors have no evidence that changes were controlled.
Unreviewed edits to the admin API
Graylog's configuration and admin API can change node settings and pipelines outside the web interface, so access to them is a change-control concern as well as a security one. Limit administrative roles to the people who run tuning, restrict API access to management networks, rotate tokens used by automation and review audit records of configuration changes. Confirm how your Graylog version records these actions before relying on it as evidence.
Questions teams ask before tuning Graylog buffers
Can our own team tune Graylog processing and output buffers?
Yes, if you have an engineer who can read node metrics, change configuration in a controlled way and explain the results. Buffer tuning is mostly a loop: find where messages wait, change one setting or one expensive rule, then measure again. The harder part is usually diagnosis, not the configuration change itself. Teams that already run Graylog in production and have someone comfortable with pipeline rules and regular expressions often manage this in-house. Teams without that experience, or with a journal that is already filling, tend to benefit from an outside review of the bottleneck before changing anything.
What does the work need from our team before it starts?
Tuning needs four things from your side:
- Read access to node metrics and the journal, so changes can be measured before and after.
- A test environment or a representative copy of your pipelines, so rule changes can be checked without affecting live ingest.
- Sample messages from each major source, so expensive parsing can be reproduced.
- Change windows agreed with the people who own the log sources and the search backend.
If metrics are not collected yet, confirm what your monitoring stack already captures before work begins. Graylog's documentation on the multi-node layout is a useful reference for which nodes and backends your team needs to include.
How long does buffer tuning take in practice?
Duration depends mostly on how many pipelines, processing rules and Graylog nodes are involved, and on how quickly changes can be tested and rolled out. A single node with a small rule set is a much shorter exercise than a multi-node cluster with many streams and regular expressions that run on every message. Ask for the duration to be stated in the written scope, with the number of pipelines, nodes and test cycles it assumes. The scope should also say what happens if the bottleneck turns out to be the search backend or disk rather than buffers.
Is Graylog Open enough for tuning throughput?
Graylog Open includes the processing pipelines, streams and the node settings that govern buffers, so tuning does not require a paid license in itself. Graylog Open is licensed under SSPL-1.0, which is not an OSI-approved open source license, and some inputs and features need an Enterprise or Security license. Before tuning, confirm which inputs and features your pipelines rely on, because a throughput problem can sit in a feature that is only available on a paid tier. The licensing terms are described on the Graylog Open product page.
What drives the cost of a buffer tuning engagement?
Cost is driven by the hours needed to measure, diagnose and test, which in turn depend on your environment. The main factors are the number of Graylog nodes, the count and complexity of pipeline rules, how many regular expressions run per message, how many log sources feed the cluster, and whether a test environment must be built. Engagements are normally fixed-scope, so the quote follows from a written scope rather than a standard price. Ask the quote to separate one-time work, such as the diagnosis and rule changes, from any recurring work, such as periodic capacity reviews. DeployOpen does not publish a price list.
How do I know when to add nodes instead of tuning buffers?
Tuning helps when messages are waiting in a specific place, such as a processor buffer filling because one rule is slow. Adding nodes helps when the whole cluster is busy and every node's processing is near its limit after the expensive work has been reduced. Check the message journal, the process and output buffers and the search backend before deciding. Graylog's system architecture guidance also covers sizing the journal and storage, which should be reviewed before a capacity decision is made.
When does a consultant help with Graylog buffer tuning?
A consultant helps most when the journal is growing, the cause is unclear, or the change carries risk to live ingest. Typical triggers include repeated journal alarms, a pipeline rule change that slowed everything down, or a plan to add nodes that needs checking first. A consultant can also set up the measurements and test method so your team can repeat the process later. DeployOpen plans and supports self-hosted open-source platforms inside your own infrastructure, so the work stays in your environment and is documented for your team.
How DeployOpen scopes and delivers Graylog buffer tuning
DeployOpen approaches buffer tuning as a measured change to a running Graylog cluster, starting with the processing and output buffer settings and the pipeline rules that consume them. The work is planned in a written scope, built and tested in your own infrastructure, and handed over with documentation your operators can use.
How we scope
- Review current node sizing, buffer settings, journal use and pipeline rules before agreeing any change.
- Define which throughput target, expensive rules and nodes are in the written scope and which are excluded.
- Confirm your Graylog version, MongoDB and search backend compatibility, and the change window your team allows.
How we build
- Adjust process and output buffer processor settings in a staged environment before any production change.
- Measure ingest, journal utilization and rule execution cost against a baseline after each change.
- Recommend rule rewrites or added nodes only where measurements show buffers are not the limit.
How we hand over
- Deliver a written record of each setting changed, the reason, and the measured effect.
- Document rollback steps and the thresholds at which your team should revisit the configuration.
- Walk your operators through the monitoring signals they need to watch after handover.
How we support
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- Ask for the written scope to state which Graylog version and environment the tuning was validated against.
