How to size Graylog storage before you buy hardware
If you are planning a self-hosted Graylog and need disk and journal numbers before procurement, start with two inputs: your average daily ingest in gigabytes and the number of days of logs you must keep searchable. The working formula in Graylog's architecture guidance is daily ingest multiplied by retention days multiplied by 120 percent, and the message journal should be sized separately at 3 to 5 times daily ingest on SSDs. Those two figures, not the number of servers, drive most of the purchase. Confirm your ingest from a measured period of real traffic rather than vendor estimates, because peaks from a single noisy source can change the answer more than the average does.
Graylog Open stores messages in indices and manages them through rotation and retention. Index rotation can be triggered by message count, index size or elapsed time, and retention then deletes the oldest indices once the configured limit is reached. This means storage is not a one-time disk purchase: the rotation and retention settings you choose decide how many indices exist at any moment, and a volume that fills before retention runs will cause the search backend to stop accepting writes. Sizing errors rarely appear on day one. They show up weeks later as dropped searches, delayed messages or a journal that cannot drain, which is why the headroom in the formula matters more than the first install.
Graylog Open is licensed under the Server Side Public License version 1 (SSPL-1.0), which is not an OSI-approved open source license. The license affects how you may offer Graylog to others, not how much storage you need. The data you store and the way it is indexed are the same across editions, so the sizing formula applies whether you run Graylog Open or a paid edition. Before you order hardware, confirm with your Graylog account team or documentation which edition you will run, because later sections on warm tiers and archives depend on features that some editions include.
Which sizing inputs change the disk and journal numbers
This table compares the inputs that move Graylog disk and journal sizing, from daily ingest and retention to replicas, SSD placement and warm tiers. Read each row to see what the input changes, what pushes it up or down, and what to confirm in your own environment before you buy storage.
| Input | What it changes | What drives it up or down | What to confirm |
|---|---|---|---|
| Daily ingest volume | Base of the storage formula: daily ingest times retention days times 120 percent | Number of sources, verbosity of application logs, debug levels and enabled audit or flow feeds | Measured average and peak bytes per day per source over at least one full business cycle |
| Retention days | Number of days of indexed data kept on hot storage before index deletion | Compliance and investigation needs by log source, plus whether archives exist before deletion | Retention per source and whether any data must stay searchable rather than archived |
| Replica count | Multiplies stored data on the search backend, since each replica is a full copy of shards | Availability target for the search cluster and the number of data nodes available | Replica setting in the index set template and the node count in the Data Node or OpenSearch cluster |
| Index rotation settings | Shard count and size per index, which affects how evenly disk fills and how quickly old data is removed | Rotation by message count, size or time, and the retention rule applied to old indices | Rotation strategy and shard sizing in each index set, and how many indices remain open at once |
| SSD versus spinning disk for hot data | Search latency and indexing throughput, not total capacity | Query concurrency, dashboard load and ingest rate; spinning disk tends to slow under heavy searches | Disk type and IOPS on each data node, and measured search response times under real queries |
| Message journal size | Disk reserved on Graylog nodes to buffer messages while the search backend is unavailable | Daily ingest, the length of backend outages you must survive and the journal's place on SSD | Target outage window, journal location and whether the journal fits within the 3 to 5 times daily ingest guidance |
| Warm tier use | Moves older indices off hot disk, lowering hot capacity but adding storage for snapshots | Enterprise licensing, the repository type (S3, GCS or file system), minimum hot days and slower search on warm data | Licensing edition, repository location and the OpenSearch or Data Node version the warm tier requires |
Why the message journal needs 3 to 5 times daily ingest
The message journal is Graylog's on-disk buffer between the input side and the search backend. When Graylog cannot write messages to Data Node or OpenSearch, for example during a backend restart, a failed node or a network partition, incoming messages are held in the journal until indexing resumes. The Graylog system architecture guidance advises giving the journal 3 to 5 times daily ingest, placed on SSDs. That range is the documented starting point for planning, and it means the journal must be able to hold several days of ingest at your peak rate, not a single day, so that a short outage does not turn into dropped or delayed log data.
Journal size and outage duration interact directly. If the backend is unavailable for longer than the journal can absorb at your ingest rate, the journal fills and Graylog stops accepting new messages from inputs that cannot be buffered, so data loss depends on how each input handles back-pressure. Sizing therefore starts with two questions: how long a backend outage your operations team must survive without loss, and what your sustained ingest looks like during a busy period, not only the daily average. Confirm with your Graylog support or documentation for your exact version how journal full conditions are reported and which inputs can apply back-pressure, because input behavior is the part of the design you cannot size from disk alone.
To check journal state on a running cluster, open the Graylog system overview for each server node and review the journal metrics and utilization, and compare the journal size against the disk allocated to it. Track the uncommitted and unprocessed message counts over time, since a steadily rising backlog after an outage is the signal that indexing is not keeping pace. Also confirm that the journal directory sits on the SSD volume you planned, with the message_journal_dir setting pointing there and enough free space for message_journal_max_size. Build alerts on journal utilization before it reaches its limit, and rehearse an outage in a test environment so the numbers you planned are the numbers you observe.
How replicas, multi-node layout and warm tiers change the numbers
The base formula gives you one copy of your data. Replicas, the node layout that holds them, and any tier that keeps older data elsewhere each change the disk figure in different ways. Work through these cards in order, because each one adjusts the number produced by the card before it.
Replicas multiply stored data
Each index shard can keep one or more replica copies on other nodes, so the disk your cluster consumes is roughly the base figure multiplied by the number of copies, plus overhead. A retention period that looks affordable with no replicas can double or more once replication is enabled. Confirm the replica count in your index set template, and check whether your planned Data Node or OpenSearch cluster reserves headroom for rebalancing and merges. Ask Graylog support to confirm how replica settings interact with your chosen retention.
Three Data Nodes for medium to large
Graylog's multi-node guide recommends three Data Nodes on their own cluster for medium to large deployments. Three nodes give the cluster room to keep replicas on a different node from their primary and to survive a node loss without losing indices. Disk must be sized per node, not only in total, so a single node filling up can stall indexing. Divide your calculated total across the nodes, add headroom for one node failing, and confirm the layout with Graylog before you buy hardware.
Separate Graylog and MongoDB nodes
The multi-node guide places three Graylog server nodes and three MongoDB nodes, run as a MongoDB replica set, on hosts separate from the search cluster. Graylog and MongoDB store configuration and metadata rather than log messages, so their disk needs are small compared with the log store. Do not let them compete with search storage on the same volumes. Plan the MongoDB disk for backups and confirm the MongoDB version range against the compatibility matrix for your Graylog release.
Warm tier with searchable snapshots
A warm tier moves older indices off the hot storage nodes into a cheaper repository, such as object storage or a file system, and keeps them searchable at a slower speed. This reduces the hot disk you need for the most recent days, but it is an Enterprise capability, so the licensing cost must be added to the storage comparison. Confirm the minimum hot days, the repository types supported for your backend version, and the search latency your analysts will accept.
Archives outside the search cluster
Archives keep messages outside the search cluster after their indices age out, so you can hold years of logs without sizing the search backend for them. Archived data is not searchable in place. It must be restored into an index before investigators or auditors can query it, which needs its own storage and load planning. Confirm with Graylog which archive backends, compression options and integrity checks apply to your edition, and decide whether archiving must finish before index deletion.
Steps to produce a sizing estimate you can defend
This sequence turns measured ingest and retention targets into a disk, journal and SSD plan that you can show to a CISO, a finance reviewer or an auditor, and that you can check against a pilot before production load arrives.
Measure daily ingest per source
Collect the daily volume of raw log data from each source you plan to send to Graylog, using the source's own counters, existing SIEM or forwarder reports, or a short capture on the network path. Record the average and the busiest day separately, because the busiest day will set the journal requirement later. Name the owner of each source and the date of the measurement. This step is done when every planned input has a daily figure with a stated method and a named owner, and the total is written down in gigabytes per day.
Set retention for each index set
Decide how many days each group of logs must stay searchable in Graylog, and separate that from any period that will be served from archives. Security, audit and regulatory requirements often differ by source, so group sources by their required retention rather than applying one number to everything. Confirm each period with the team that owns the requirement, such as the security operations lead or the compliance lead. This step is done when each index set has a written retention period and a named approver, and the sources in each set match the ingest figures from the previous step.
Apply the documented storage formula
Multiply daily ingest by retention days, then multiply the result by 120 percent, as the Graylog system architecture guidance describes. Do this for each index set and add the results together to get the hot storage target. Keep the arithmetic in a spreadsheet with the inputs in separate cells so the figures can be reviewed and changed. This step is done when the spreadsheet shows a per-index-set figure and a total, and every input cell links back to a source documented in step one or two.
Add replica and journal capacity
Replicas multiply the disk used by each index, so the replica count in each index set template changes the total directly. Ask the platform owner which replica setting applies to each index set, and add that multiplier to the storage formula. Then size the message journal separately. The Graylog guidance recommends giving the journal 3 to 5 times daily ingest on SSDs, so choose a value in that range based on how long a backend outage you need to absorb. This step is done when the replica multiplier and journal size appear as separate lines in the estimate and the journal figure is tied to a stated outage tolerance.
Choose SSD placement for hot data and the journal
Decide which disks will hold the active index data and the journal, and confirm that both sit on SSDs in the Graylog server or Data Node hosts. Check the storage layout against the multi-node reference design, which uses separate nodes for Graylog, MongoDB and the Data Node cluster. Record the mount points, filesystem and free-space headroom you intend to keep. Ask your storage team to confirm the IOPS and latency the hardware delivers, since the vendor figures do not replace a measurement. This step is done when a diagram or table shows each volume, its media type and its size, and the storage team has signed off the layout.
Run a pilot ingest against the estimate
Send a representative subset of real sources to a test Graylog cluster for a period long enough to see daily cycles and at least one busy day, then compare the observed index size per message and the observed daily growth with your spreadsheet. Confirm that the sources you sampled cover the highest volume inputs, since a small pilot can hide the largest consumers. Document any difference and adjust the estimate before scaling. This step is done when the measured growth rate falls within the tolerance your team agreed in advance, and the difference is explained in writing.
Check journal and disk metrics before go-live
Watch the journal utilization, the input buffer and the free disk on every node during the pilot, including a deliberate pause of the search backend in a test window if your change process allows it. The journal should absorb the pause and drain afterward without losing messages. Watch disk usage trends against the projected figure and confirm that index rotation and retention deletion behave as the index set settings say. Keep the captured metrics and the test record, since they become evidence for the capacity decision. This step is done when the journal drains after the test, disk growth matches the adjusted estimate, and the capacity sheet is approved by the platform owner.
Security and operational considerations for storage capacity
Disk sizing is only useful if the storage it describes stays protected, watched and aligned with policy. The four areas below connect the capacity numbers to the controls that keep stored logs intact and available. Each one has a check you can run before go-live and again during regular operations.
Protect the search backend
The search backend holds every indexed message, so it carries the same sensitivity as the logs themselves. Confirm that the Data Node or OpenSearch cluster is reachable only from Graylog server nodes and administrators, and that transport and admin credentials are stored outside general configuration files. Treat the cluster as a regulated data store when you size it, because a compromise exposes the full retention window rather than recent events. The Graylog security topic covers the detailed controls.
Restrict journal and data directories
The message journal and the Elasticsearch or OpenSearch data directories hold raw messages before and after indexing. Set filesystem ownership so only the service accounts that run Graylog and the search backend can read or write these paths, and keep them on the SSD volumes you sized for. Check that backups and snapshot repositories inherit the same restriction, since a copy of the directory is as sensitive as the original. Record who holds root on each node.
Watch journal and disk growth
Capacity estimates age quickly when ingest rises or a new source is onboarded. Track journal utilization and disk usage on every Graylog and data node, and alert well before either reaches its limit, since a full journal or full disk stops ingest or search. Compare measured daily growth against the figure your formula predicted each month. A sustained gap is the signal to resize the journal or the cluster, not a reason to shorten retention silently.
Align retention with policy
Retention in the sizing formula should come from a written requirement, not from whatever disk allows. Ask legal, compliance and the security team which log sources must be kept, for how long, and whether any must be archived before deletion. Map each answer to an index set or template so the retention period is set once and applied consistently. Re-check the mapping whenever a regulation, contract or internal policy changes, because the disk plan depends on it.
Questions teams ask about Graylog storage sizing
How much effort does Graylog storage sizing take, and what drives it?
Effort depends mainly on how many log sources you have, how their daily volume varies, how long each class of logs must be kept, and how many environments need separate sizing. A single source with stable volume and one retention rule is a short exercise. Multiple sources with different retention, bursty peaks and a tiered design take longer because each index set needs its own inputs and checks. Ask any partner to show which of these factors their estimate covers, and to name the measurements they will take before they propose a disk figure.
Can our own team size Graylog storage without outside help?
Yes, a team with Linux and search cluster experience can do the core calculation itself. Graylog's system architecture guidance gives the basic formula of daily ingest times retention days times 120 percent, and the message journal guidance of 3 to 5 times daily ingest on SSDs. The harder parts are measuring real ingest rather than guessing it, testing behavior during a backend outage, and checking that the chosen backend version matches your Graylog release. Many teams do the arithmetic in house and bring in a partner to validate it under load.
What inputs do we need to provide before sizing starts?
Gather measured daily ingest per log source for at least a representative period, peak rates during known busy windows, and the retention each source must meet under your internal policy or the regulations that apply to you. Also provide the number of Graylog and Data Node hosts you can run, the disk types and capacity available, and your backup and archive plans. Where you expect growth, state the time horizon. Missing measurements are the most common reason estimates fail, so list any source where the volume is unknown.
How do replicas change the storage cost?
Replicas multiply the disk footprint of each index, so the number of copies you keep directly changes how much storage you buy. The Graylog guidance formula does not add replicas on its own, so the replica count and shard layout need to be added to the calculation explicitly. Each extra copy adds disk and some CPU and memory load, and it also affects recovery time after a node failure. Confirm the replica setting in your index set template and check how your Data Node or OpenSearch cluster handles it during rebalancing.
Is the 120 percent factor enough headroom for our environment?
The 120 percent factor is a planning margin from Graylog's architecture guidance, and it may or may not cover your case. It does not account for your compression ratio, index overhead from your field mix, replicas, merge activity, or the space needed for snapshots and restores. Teams with high-cardinality fields, long retention or large restores often need more margin and should test it against real data. Treat 120 percent as a starting point to validate, not a finished number.
Where should the message journal and SSDs go?
The message journal is written to disk on Graylog server nodes before messages reach the search backend, so it needs fast local storage sized for your ingest. Graylog's guidance suggests 3 to 5 times daily ingest on SSDs, which gives the buffer room to ride through a backend outage. Keep the journal on local SSDs rather than shared network storage where possible, and monitor its fill level. Hot index data on the Data Node or OpenSearch side also benefits from fast disks, while older data can move to cheaper media in a warm tier.
What should a quote for storage sizing include?
A quote should state the written scope: which log sources and volumes are covered, which environments are included, and which deliverables you receive, such as a sizing workbook, disk plan, journal settings and a validation test. It should list what you must supply and what is assumed. Ask for the sizing method, how replicas and warm tiers are handled, what the validation test measures, and what is excluded, such as hardware purchasing or ongoing capacity reviews. Ask whether ongoing reviews are a separate retainer.
Does Graylog Open include everything needed to size and run storage?
Graylog Open provides index rotation by message count, size or time, and retention that deletes the oldest indices, which covers basic hot storage control. Features such as data tiering and the data lake with archive restore are part of Graylog Enterprise, and the Enterprise and Security plans start at a minimum of 10 GB per day or 100 Graylog Compute Units. Confirm which features your design needs on the Graylog pricing page before you finish the storage plan, because a warm tier design depends on the license you hold.
How DeployOpen scopes a Graylog storage estimate
DeployOpen scopes every storage estimate in writing before any sizing work starts, so the client knows exactly what the number is based on and what it leaves out. The estimate is not a quote in itself. It is a worked calculation the client can check, built from the inputs below, and it becomes part of a fixed-scope Graylog implementation or a separate sizing engagement. Either way, the written scope names the deliverables before work begins, and any price or timeline follows from that scope rather than from a standard package.
The inputs we ask for are the daily ingest per source group in gigabytes, the retention period each group must meet (hot searchable days and any longer requirement), the number of Graylog and Data Node replicas planned, and whether a warm tier or archive is in the design. We also ask which disks and instance types are available, and whether the message journal will sit on SSDs. Where a figure is an estimate rather than a measurement, we ask the client to confirm it against current log volumes, because the sizing formula is only as good as the ingest figure it multiplies.
The written estimate covers the raw storage formula from the Graylog architecture guidance (daily ingest times retention days times 120 percent), the journal allowance for backend outages, the disk layout per node, and the assumptions behind each figure. It also lists what is not covered, such as compression ratios that have not been measured in the client's environment. Ask whether the estimate will be re-run after a pilot period, and confirm with the Graylog documentation for your target version before accepting any figure as final.
How DeployOpen delivers and supports Graylog sizing
DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure, including Graylog storage sizing and the message journal. We start from your ingest sources, retention requirements and backend layout, then deliver sizing that your team can review, test and maintain.
How we scope
- Written scope lists the log sources, daily ingest estimates, retention periods and index sets covered by the sizing.
- We confirm the Graylog and MongoDB versions, Data Node or OpenSearch backend and cluster layout before any numbers are set.
- Fixed-scope engagements agree deliverables and timelines in writing before work starts, so the quote follows from the scope.
How we build
- We calculate storage from daily ingest, retention and replica settings, then add headroom and journal capacity for backend outages.
- Journal volume and SSD placement are specified per node, and the journal size is checked against your expected ingest peaks.
- We test the sizing against a representative load, watch disk and journal behavior, and adjust index rotation and retention settings to match.
How we hand over
- You receive a sizing workbook with inputs, assumptions, formulas and the results for each index set and cluster tier.
- Runbook notes cover journal monitoring, disk thresholds, index rotation checks and the steps to resize storage as ingest grows.
- Knowledge transfer sessions walk your engineers through the model so they can rerun it when sources or retention change.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, including re-sizing after new log sources are added.
- We review capacity trends with your team and flag when journal use or disk headroom approaches the limits set in the design.
- Ask your quote to name who maintains the sizing model, how changes are requested and how Graylog version upgrades are tested.
