How Graylog index set templates for retention policies keep retention consistent
Index set templates give each class of log source a predefined retention and storage profile, so retention is set once and reused instead of being chosen separately for every index set. A team needs them when many sources feed Graylog, retention differs between sources without a documented reason, and storage cost climbs because some sources keep data longer than policy requires. This page covers the Graylog 7.x documentation, which is the current line at the time of writing, and you should confirm the exact template fields in the release you run.
In Graylog, messages are written to index sets, and each index set controls how its indices rotate and how long they are kept. The Graylog index model describes rotation by message count, size or time, and retention that deletes the oldest indices once the policy limit is reached. A template fixes those rotation and retention choices together with shard and replica settings, so every new index set created from it starts with the same behavior. Graylog ships built-in hot templates for 7, 14 and 30 day retention, and you can add custom templates for other periods, shard counts and replica counts, with one template marked as the default.
Start with an inventory of log sources grouped by how long each group must stay searchable, then map each group to one template and set the default for anything unclassified. Ask your team to confirm how template changes apply to index sets that already exist, because an existing index set may keep its settings after a template is edited. Check shard size and replica choices against your disk plan before you create templates, since each replica stores a full copy of the data. Record the mapping in your runbook so that a new source gets a template by policy, not by whoever creates its index set.
Template options and when each fits
This table compares the built-in 7, 14 and 30 day hot templates, custom templates, shard and replica settings, shard size and the default template. Read each row for the situation it suits and the trade-off it carries, then confirm the exact labels and defaults in your Graylog version before you standardize on one.
| Option | Fits when | Trade-off | Confirm before use |
|---|---|---|---|
| Built-in 7 day hot template | Logs are needed for short-term search and the team has limited hot storage | Older events leave search quickly, so investigations past a week need an archive or restore path | Exact label and retention value in your version |
| Built-in 14 day hot template | Security or operations teams review two weeks of history in routine work | Storage grows about twice as fast as the 7 day option for the same ingest | Exact label and retention value in your version |
| Built-in 30 day hot template | Audits or incident reviews regularly ask for a month of searchable events | Largest hot footprint of the built-in options, so disk and replica planning matter most | Exact label and retention value in your version |
| Custom template | Sources have different volumes or needs, such as high-volume firewall logs versus application logs | Each template is another rule to document, test and review at each change | How templates are created and assigned in your Graylog version |
| Shard and replica settings | Availability requirements justify replicas, or the cluster needs more parallel capacity | More replicas raise disk use and write load on the search backend | Shard and replica defaults in your Data Node or OpenSearch setup |
| Shard size settings | Indices are very small or very large and need tuning for search speed or cluster stability | Poor sizing creates many small shards or oversized indices that are slow to manage | Supported shard size ranges and recommendations in the current docs |
| Default template | A single retention policy suits most sources and new index sets should inherit it | Sources that need different retention can be assigned the wrong policy if the default is not reviewed | Which template new index sets receive by default in your installation |
Setting up index set templates in order
This sequence takes you from an inventory of log sources to a tested default index set template, so each source gets retention and shard settings that match its value and the cluster you actually run.
Inventory log sources and retention needs
Start by listing every log source that will reach Graylog, with its owner, daily volume estimate, and the period it must stay searchable. Record which teams need fast search and which only need records for audits. Ask your compliance lead and the owners of each system to confirm the retention period for each source, because those answers set the template lengths. The step is done when every source has a named owner and a retention requirement written beside it. Check the list with the security and compliance owners before moving on.
Map sources to streams and index sets
Group sources that share a retention need into streams, then route each stream to one index set. Keep groupings small enough that a single retention rule covers all members. Mixing a 7 day audit source with a 30 day security source in one index set forces the longer retention on both. The step is complete when a table shows each stream, its index set, and the reason for the grouping. Confirm the mapping by sending a test message from one source per group and checking that it lands in the expected index set.
Choose a template for each retention tier
Match each index set to a template. Graylog ships built-in hot templates for 7, 14 and 30 day retention, so use those where they fit. Create a custom template only when a group needs a different rotation or shard layout. Document the chosen template name, its retention period, and the rotation setting for each index set. The step is done when no index set is left on an unplanned template. Review the list with the team that owns each source, since a template change later affects how much history is searchable.
Set shard and replica values against cluster size
Set shard and replica counts for each custom template based on the number of data nodes in your cluster. Replicas need enough nodes to hold separate copies, so a replica count higher than the node count leaves copies unassigned. Set shard size so that index sizes stay in a range your backend handles well, and confirm the chosen values against the Graylog index model documentation and your Data Node or OpenSearch sizing notes. Keep a record of the storage sizing result, using the formula in the Graylog system architecture guidance: daily ingest times retention days times 120 percent. The step is done when the storage estimate fits the disk on each data node with headroom for rotation. Confirm the final numbers with the platform engineers who run the cluster.
Make one template the default
Set one template as the default so that new index sets created without a deliberate choice inherit a known retention and shard setup. Pick the default with the team that will create index sets most often, and confirm that its retention is the one your policy calls for when nobody has decided otherwise. The step is complete when the default is set in the Graylog interface and a new index set created without a template choice shows the expected settings. Record the default name in your runbook so future administrators know which setting applies.
Verify rotation and deletion on a test index set
Create a test index set from the chosen template, using a short rotation period and a small test volume so you can watch the lifecycle. Send messages, confirm that rotation creates new indices as configured, and confirm that retention removes the oldest index once the limit is passed. Use a test environment where possible, because deletion is permanent. Check the index list in the Graylog interface and the indices on the search backend to confirm the result matches the template. The step is done when rotation and deletion behave as the template says, and you have written down the observed results. Only then apply the same settings to production index sets.
How retention templates differ by environment
A single team running Graylog for its own systems usually needs few templates. One default template and perhaps one exception for a high-volume source cover most cases, and the team can agree changes in a short conversation without a formal sign-off. The main risk is informal drift: a new source gets a template nobody documented, and later no one remembers why it was set up that way.
- Template count: Keep one default template and at most one exception for a high-volume source, so the storage math stays easy to check.
- Naming: Use names that state the source type and retention period, so the index set list reads clearly without opening each template.
- Approval: The team lead or on-call owner records changes in the team change log, with no separate review board.
- Review cadence: Revisit retention when daily ingest changes, and write the reason for each change in the template description.
Check that the default template matches the storage you sized with the formula in the system architecture guidance, and that at least one person other than the template author knows where the settings live.
When several teams send logs into one cluster, index set templates become a shared service. One team's retention choice uses disk and search capacity that every other team depends on, so template decisions need a named owner and a published catalog of what exists.
- Template count: Map templates to a short list of retention classes that teams agree on, so the cluster does not accumulate near-identical templates.
- Naming: Prefix each name with the retention class and the owning group, so teams can find their own index sets quickly.
- Approval: A platform owner approves new templates and any change that increases retention or shard count, and application owners request changes through a ticket.
- Capacity impact: Each new template adds to the storage budget for the whole cluster, so ask each team for projected daily ingest before approval.
- Default assignment: Set the default template to a conservative class so streams nobody has claimed do not fill the cluster.
Before you publish the catalog, confirm which roles can edit index sets in your Graylog configuration, so that application teams can request changes without being able to change shared templates directly.
Where a regulation or internal policy sets the retention period, the index set template is the technical form of a policy decision. The period is set by the policy owner, and each template should trace back to a written requirement that an auditor can read.
- Template count: Create one template for each distinct retention requirement in the policy, and do not add templates for convenience.
- Naming: Include the policy reference or requirement class in the name, so a reviewer can match each template to the clause it implements.
- Approval: Obtain written approval from the policy or compliance owner before changing any retention, shard or replica setting, and store the change and approver in the change record.
- Deletion control: Retention that deletes the oldest indices removes those messages from search, so confirm that archiving or export is in place before any period is shortened.
- Evidence: Keep exports of template settings and their change history for each review cycle.
Ask the compliance owner which retention periods apply to each log source and confirm them in writing before you build templates. DeployOpen can prepare the configuration and evidence, but the retention periods and any audit outcome come from your policy owner and the independent assessor.
Security and operational risks of retention templates
Index set templates decide how long Graylog keeps data and how the search backend is laid out, so a template edit can affect every log source that uses it. The cards below cover who can change templates, how retention can remove data before it is reviewed or archived, how shard and replica changes affect the cluster, and how to keep a record of changes.
Who can edit templates
Template changes are administrative actions, so the first control is the list of users who hold the permissions to create, edit or apply index set templates. A shared administrator account makes it impossible to tell who changed a retention period. Confirm in your Graylog version which built-in roles and permissions cover index set management, and remove those rights from users who only search or build dashboards. Review the list on a set schedule and after staff changes.
Retention deleting data early
Graylog's retention setting deletes the oldest indices once the policy is reached, so a shorter period set in a template can remove messages that no one has reviewed or copied to an archive. Illustratively, if a template is lowered from 30 days to 7 days, the 23 days of older data may be gone at the next rotation. Before changing a period, confirm that archiving or review is complete for the affected indices and that the change is scheduled with the people who own those logs.
Shard and replica changes
Shard count and replica count are set when an index is created, and changing them in a template affects only new indices. A higher replica count adds disk and indexing load, while an unplanned shard count can leave the cluster with many small shards or uneven load. Test shard and replica settings on a non-production cluster first, then confirm the effect on disk and node capacity with the sizing guidance before applying them to production log sources.
Default template and new streams
The default template applies to index sets that do not name another template, so changing it can alter retention for log sources nobody has reviewed recently. Treat the default as a control point: keep it at a conservative setting, record its value, and check which index sets point to it before any edit. Confirm in your version how existing index sets respond to a template change, since some settings may apply only to indices created after the change.
Auditing template changes
A retention change is only defensible if you can show who made it, when, and what the value was before. Graylog's own audit facilities and your log of administrative actions should both be checked for index set and template edits. Confirm in your deployment whether Graylog records template changes in its audit output and where that output is stored. Export the current template settings on a schedule so you can compare them against the last known state.
Protecting the backend configuration
Templates are stored as configuration in MongoDB, and the search backend holds the indices that retention acts on. A user or script that can reach either backend directly can change retention outside Graylog's controls. Restrict network and account access to MongoDB and the Data Node or OpenSearch cluster, and confirm which accounts hold administrative rights there in your version. Include backend access in the same review as Graylog user roles.
Questions teams ask about index set templates
How much effort does a Graylog retention standard take to set up?
Effort depends mostly on how many log sources you have and how many distinct retention rules your policies require. A standard with one or two templates, such as 7, 14 and 30 day hot retention, is usually a short design exercise followed by configuration and verification. Each additional custom template adds design, naming, shard and replica decisions, and testing against real ingest. Ask any partner for a written breakdown of work by template, source group and environment, since a fixed-scope quote should show what is included and what your team must provide.
What makes the work take more or less time?
The main drivers are the number of log sources and their volume, the number of retention classes your compliance and security teams require, and whether you already run Graylog with a documented index model. Work goes up when sources have no owner, when retention requirements conflict across frameworks, or when a cluster is already holding indices that must be migrated to new templates. It goes down when retention classes are agreed in advance and the Graylog environment is already healthy. Confirm each of these with your team before asking for a quote.
What does our team need to provide for this work?
You need named owners for each log source, a written list of retention requirements from your security, compliance and legal teams, and access to a non-production Graylog environment where templates can be tested. You also need the current Graylog version, the search backend in use, and a sizing estimate for daily ingest. Graylog's sizing guidance uses daily ingest times retention days times 120 percent, so having measured ingest per source makes the template design faster and more accurate.
Can our own team set up index set templates without outside help?
Yes, a team with Graylog administration experience can create index set templates in the Graylog interface and apply them to new index sets. The harder parts are deciding retention classes that match policy, choosing shard and replica settings for your storage layout, and confirming how existing indices behave after a change. Teams without that experience often bring in a partner for the design phase and keep the operating runbook in house. Your decision should weigh staff time against the risk of deleting data too early.
Which Graylog version and backend do we need before changing retention?
Check the Graylog compatibility matrix for your target release, since the supported MongoDB and OpenSearch or Data Node versions differ by Graylog version. Graylog recommends Data Node as the search backend, and use of OpenSearch 1.x has been deprecated since Graylog 7.0 and will be removed in Graylog 8.0. Confirm your versions against the Graylog compatibility matrix before you plan template changes, because a version mismatch can block the work entirely.
Do index set templates need an Enterprise license?
Index set templates for hot retention are part of the self-managed Graylog platform, so you do not need an Enterprise license to define the 7, 14 and 30 day templates or custom shard and replica settings. Features that extend retention beyond hot storage, such as data tiering and archive restore, are licensed separately. Graylog Open is licensed under SSPL-1.0, which is not an OSI-approved license, so review that license with your legal team. Confirm the exact feature mapping for your edition with Graylog before you finalize the design.
How do we check that retention is working after the change?
Verification means confirming that each index set uses the intended template, that rotation and retention settings match your written policy, and that old indices are deleted only when expected. Test with a sample stream in a non-production cluster, then review index set configuration and the number of indices kept per source in production. Keep the evidence of these checks with the change record, since compliance reviewers often ask how retention was set and verified. Auditors assess your controls independently, so ask your partner for the verification steps and records rather than assuming a pass.
How retention settings change storage sizing
Retention days are the input that moves disk need the most. The Graylog system architecture guidance sizes log storage as daily ingest times retention days times 120 percent, so a template that keeps an index set hot for 30 days needs roughly four times the disk of one that keeps it hot for 7 days, for the same ingest rate. Before you choose between the built-in 7, 14 and 30 day hot templates, measure the daily volume each log source actually produces, since a single noisy source can dominate the total. The 120 percent factor is the documented starting point, and you should confirm with your own ingest measurements whether it leaves enough headroom for your message sizes and growth.
The template also sets how long messages stay hot, meaning searchable on the primary Graylog search backend at full speed. Each index set template pairs a rotation and retention policy with shard and replica settings, so the hot period is set by the template, not by the total retention you want for compliance. Map each log source to a template by its investigation needs: security events that analysts query daily may justify a longer hot window, while verbose debug logs may only need a short one. Confirm with your security and compliance owners how long each category must stay searchable before you assign it, because moving data out of the hot period later changes the cost picture.
Once you know the hot period and daily volume, the remaining decisions are how much history to keep and where it lives. The Graylog system architecture documentation covers the sizing formula in detail, and the sizing topic in this group covers the journal and replica math. For data older than the hot window, the warm tier topic explains the Enterprise licensing, snapshot repositories and minimum hot days involved, and the archiving topics cover keeping logs beyond the search cluster. Plan those options together with your templates so that no index is deleted before its archive requirement is met.
How DeployOpen scopes and delivers retention templates
Every retention engagement starts with a written scope agreed before any configuration changes. For index set templates, that scope names the log sources in play, the retention period each source needs, the Graylog edition and search backend in use, and the environment where the work will run, whether that is the client's cloud accounts, data centers or an air-gapped network. Most engagements are fixed-scope, so deliverables and timelines are settled in writing up front. Before the scope is signed, the client should confirm which retention periods come from internal policy, which come from contractual or sector obligations, and who owns the final decision when the two conflict. Those answers drive the number of templates and the effort involved far more than the Graylog configuration itself.
The delivery work maps each log source to a template and then builds the templates in the client's own Graylog deployment. DeployOpen configures the built-in hot templates where they fit, creates custom templates where a source needs different shard or replica settings, and sets one default template so that new inputs do not silently inherit an unintended retention period. Each mapping is tested against a real stream of messages to confirm that the index rotation and retention behavior matches the agreed period, and that shard size and replica counts suit the storage the client has provisioned. The client receives the templates, the source-to-template mapping and the verification results, with all configuration kept in infrastructure they control.
The retention standard is written up as a document the client can hand to internal reviewers or external auditors. It records each source, its template, its retention period, the reason for that period and the owner who approves changes. DeployOpen prepares this evidence and the supporting configuration exports, but it does not issue audit reports or certifications, and the client should confirm with its auditor which records that auditor will expect. After handover, many teams keep an optional retainer for periodic review, so that new log sources are mapped to a template before they go live and changes to retention are checked against the standard. Ask for the review cadence and the change process in writing before the retainer begins.
