Archiving Graylog logs to S3 or object storage

Archiving Graylog logs to S3 or object storage: when a search cluster is the wrong place for years of logs

Archiving moves older log data out of the search cluster and into object storage, so the cluster keeps only the window your analysts search every day. Graylog's default retention works differently. Its index model rotates indices by message count, size or time, and the retention setting deletes the oldest index once the configured limit is reached. Without an archive step, logs that your compliance or legal requirements say you must keep for a longer period are removed from Graylog when their index is deleted. Teams usually need an archive when retention requirements run past what the hot cluster should hold, when storage on the cluster costs more than the value of searching older data directly, or when auditors ask for records they can inspect without an investigation. Confirm the retention periods with your compliance and legal owners before any design work starts, because they set the archive size, the object storage class and the lock settings.

Before you design the archive, confirm licensing. The Graylog pricing page documents archive restore as an Enterprise capability, alongside data tiering and the data lake, so the archive feature may require a license that Graylog Open does not include. Ask Graylog to confirm which edition covers archiving, what version of Graylog and which Data Node or OpenSearch release it depends on, and whether the archive target must be a specific storage type. Also ask how restored data is handled, because restores load the search backend and affect how long you can keep archives usable. The Graylog pricing page is the starting point for that conversation, but the written answer from the vendor should drive your decision.

Once licensing is confirmed, the archive design should answer four questions in writing: which backend holds the objects (Amazon S3 or an S3-compatible store you run), how the data is compressed and checksummed so you can prove integrity later, how long each archived object is retained and whether it is protected from deletion, and how the archive is written before the matching index is deleted. The last point is the one teams miss most often. If retention removes an index before its archive job has finished and been verified, the messages are gone. Plan a check that compares archived counts with the index counts before each deletion, and record the result so an auditor can see it.

Choosing an object storage backend: AWS S3, Ceph, Garage, SeaweedFS or MinIO

Amazon S3 is the reference option when you want a managed service and do not want to run storage hardware. S3 Standard is designed for 99.999999999% durability and 99.99% availability, and it stores data across at least three Availability Zones. Its Object Lock feature adds write-once-read-many storage with governance and compliance retention modes and legal holds, but it works only in versioned buckets. Costs are mostly recurring: storage by volume and storage class, requests, and data retrieval when you restore archives. Your team still owns bucket design, IAM policies, encryption keys, and lifecycle rules.

Object Lock modes: Compliance mode prevents deletion by any user until retention expires, while governance mode allows users with specific permissions to change it. Choose the mode your retention policy requires.
Versioning prerequisite: Enable versioning before you turn on Object Lock, because the lock cannot be used on unversioned buckets.
Regulatory assessments: Amazon reports a third-party assessment of Object Lock for SEC 17a-4, CFTC and FINRA environments. Ask your compliance lead whether that assessment matches your obligations.
Retrieval costs: Restores from colder storage classes carry retrieval charges and delays, so plan restore frequency with your investigation workflow.
Data residency: Confirm the AWS Region and account structure with your data owner before you create the bucket.

Before relying on any feature, confirm the current behavior in the Amazon S3 storage classes documentation and test Object Lock with your Graylog archive job.

Compression, integrity and retention settings that shape the archive

The archive is shaped by four decisions made before the first message is written: how data is compressed, how its integrity is proven, how objects are organized in the bucket, and how long the archive is kept relative to the hot indices. Confirm each one against your Graylog version's archive documentation and your object store's feature list, because the options available differ between Graylog Enterprise archiving, a Data Node or OpenSearch backend, and the S3-compatible store you choose. Compression trades storage cost against the time needed to restore data later, so ask your team to test a restore of a representative time range before settling on a format. Record the chosen settings in your runbook, since changing them later means archiving again or accepting mixed formats in the bucket.

Integrity depends on the store, not only on checksums written by Graylog. Where the backend supports it, enable versioning and S3 Object Lock in a bucket created for the archive. Object Lock works only in versioned buckets and offers governance and compliance retention modes plus legal holds, so decide which mode fits your obligations and who is allowed to override it. Confirm with your auditor whether compliance mode is required, because it cannot be shortened by ordinary administrators once applied. Keep the archive bucket under separate credentials from the Graylog server so that a compromised server cannot delete or rewrite archived objects.

CIS Control 11 asks you to keep recovery data in an isolated copy, such as an offline, cloud or off-site destination with version history, so that an attacker or failed change cannot reach every copy at once. In practice, the archive can serve as one isolated copy if its account, network path and retention settings are separate from the hot indices and the Graylog administrators. Write down who can change the bucket policy, who holds the keys, and how long a deleted object remains recoverable, then test that a deletion from the Graylog side leaves the archived object intact. Confirm that your retention period in the archive is longer than the hot index retention, and that the archive policy, not the index rotation policy, governs how long data survives.

Archive configuration options and when each fits

This table compares the archive configuration decisions you make before the first archive job runs, with the situation each one suits and its trade-off. Read each row as a question to settle with your compliance owner, then confirm the exact setting names and behavior in the Graylog version you run.

Option	When it fits	Trade-off	What to confirm
Object storage backend	Archives must sit outside the search cluster on storage you control or buy from a cloud provider	Amazon S3 offers high stated durability; self-hosted options like Ceph or SeaweedFS shift durability and upkeep to your team	Confirm the S3 API features Graylog archiving uses against your chosen backend
Bucket versioning	You need to recover from accidental overwrites or deletes of archive objects	Versioning keeps prior copies, which adds storage cost and must be covered by your retention plan	Confirm the backend supports versioning; Garage does not implement bucket versioning
Object Lock, governance mode	Archives need write-once protection, with authorized staff able to remove objects under controlled conditions	Privileged users can still override retention, so it proves less to an external assessor than compliance mode	Confirm the backend supports Object Lock and that versioning is enabled first
Object Lock, compliance mode	Regulation requires archives that no account, including administrators, can delete before the retention period ends	Deletion before expiry is blocked for everyone, so mistakes in retention periods are hard to undo	Confirm retention periods with legal and compliance before enabling; confirm the backend's Object Lock modes
Compression	Archive volume is large and search speed on restore matters less than storage cost	Smaller objects save space but add CPU work on archive and restore, and slow restores	Confirm which compression format your Graylog version writes and how it reads it back
Checksum verification	You must show archived data was not altered or truncated, and detect corruption early	Checksums add processing time and need a stored record to compare against	Confirm where checksums are stored, who checks them, and how failures are reported
Archive retention period	Retention is set by a written policy for each log source, such as a regulator or contract	Longer retention raises storage cost for as long as the archive exists, and no deletion is needed until policy allows it	Confirm the period against the written policy and whether it differs by index set
Archive before index deletion	Index retention deletes the oldest indices, so messages must be archived before that step runs	If archiving fails or lags, deleting the index loses those messages unless the archive is verified first	Confirm the archive job completes and is verified before the rotation or retention step removes the index

Setting up archiving in the order that protects your logs

Work through these steps in order so archived logs are protected before Graylog retention can delete the original indices. Each step closes with a check, and the sequence is complete only after a restore has been proven. Confirm the backend, versioning and retention settings with your vendor and compliance owner before you start.

Confirm the Graylog version and edition
The Graylog administrator records which Graylog release runs in production and which edition is licensed. Archiving to a data lake with archive restore is listed as an Enterprise feature, so confirm the license includes it before any bucket is created. Compare the release against the compatibility matrix for MongoDB and the search backend, including whether your search backend is Data Node or OpenSearch. The step is done when the version, edition, license entitlement and backend versions are written in the change record and all of them are documented as supported together.
Choose a backend that meets retention controls
The platform owner selects the object store before anything is built, because the required controls narrow the options. Amazon S3 Standard is designed for 99.999999999% durability and stores data across at least three Availability Zones. Ceph RADOS Gateway documents versioning and lifecycle rules, while Garage does not implement versioning or object lock, so it fits only where neither is required. Ask your Graylog vendor in writing which backends and versions the archive feature accepts. The step is done when a decision record lists each required control, its backend support and the source you checked.
Provision the bucket with versioning enabled
The storage engineer creates a dedicated archive bucket and turns on versioning before any data is written. Versioning is required for Object Lock and keeps earlier object versions when an archive is overwritten or deleted. Use a bucket name and prefix that only the archive process uses, and place the bucket in the region or site your data residency rules require. Check that the bucket versioning status reads enabled, then record the bucket name, region and endpoint in the runbook.
Apply Object Lock where rules require it
If your policy or auditors require write-once storage, the security lead chooses between governance and compliance retention modes and sets a default retention period. Object Lock works only in versioned buckets, and legal holds can protect objects an investigation must keep. Confirm with the backend vendor whether Object Lock must be enabled at bucket creation, because it may not be possible to add later. The step is done when a test object shows the expected mode and retain-until date, and a delete attempt against it is refused under that mode. Object Lock supports your evidence but does not make the deployment compliant on its own.
Create least-privilege credentials for Graylog
The Graylog administrator and storage owner create a dedicated access key for archiving, separate from keys used by other applications. Grant write and list access only to the archive prefix, and withhold permissions to change bucket policies, versioning or lock settings. If Object Lock is active, confirm the key cannot shorten retention. Test the key by writing one object inside the prefix, then attempt a write outside the prefix and a delete of the test object, recording each result. The step is done when the allowed and refused actions match the written permission list.
Run a test archive of one index
The Graylog administrator archives one small, non-production index or sample set using the archiving function your edition provides. Before the run starts, record the index name, time range, message count and expected data size. After the run, compare the number of archived objects and total bytes in the bucket with those recorded values. The step is done when the counts match, the objects appear under the expected prefix, and the Graylog server logs show no archive errors for the run.
Verify checksums and run a restore test
The security engineer compares checksums of the archived objects with values recorded at write time, so altered or corrupted objects are found before they are needed. If the archive process does not record checksums, generate and store them at write time and confirm the backend's integrity options with the vendor. Restore the sample into a separate, non-production Graylog environment and search the restored time range. CIS Controls v8.1 Safeguard 11.4 asks for an isolated copy of recovery data, so keep the restore environment apart from production. The step is done when every checksum matches and the restored search returns the same message count as the original index.
Enable archiving before retention deletes indices
The Graylog administrator turns on the archive schedule for each index set only after the test restore succeeds. Graylog retention deletes the oldest indices under its rotation settings, so an archive that lags behind retention loses data. Review the retention settings and archive status for each index set together, and record which index sets are covered. The step is done when the oldest index shows as archived in both the bucket and Graylog, and the next scheduled retention run is confirmed not to remove any index without a completed archive.

Security and operational risks in archived log storage

Archives hold the logs you may need to prove something years later, so they need the same care as production data. The cards below cover the controls that protect archived logs from misuse, loss and silent failure, and they show where each control has to come from in your chosen backend.

Scope access keys per bucket

Give the archive job its own access key, limited to the archive bucket and the actions it needs to write and list objects. Keep restore and administrative keys separate so a compromised write key cannot read or delete everything. With Garage, permissions are set per access key and per bucket, so confirm the same separation in whichever backend you choose. Review who holds keys each quarter.

Encrypt archives at rest and in transit

Require TLS between Graylog and the object store, and reject plain HTTP endpoints in the archive configuration. For data at rest, confirm which encryption option your backend offers and who manages the keys. Garage does not provide a server-side encryption configuration, so encryption at rest for that backend has to come from the underlying disks or filesystem. Document the choice in the archive design.

Protect against deletion

Archives are only useful if an operator or attacker cannot quietly remove them. On Amazon S3, Object Lock in a versioned bucket enforces write-once-read-many retention in governance or compliance mode. Where the backend lacks object locking, keep an isolated, version-controlled copy in a separate location, in line with the recovery data guidance in CIS Control 11. Test that the copy can be restored.

Rotate credentials on a schedule

Store archive keys in a secrets manager rather than in Graylog configuration files, and set a rotation date for each one. Rotate keys after staff changes and after any suspected exposure, then confirm that the archive job still writes after the change. Record each rotation so the audit trail shows when the archive credentials changed.

Monitor archive job failures

A failed archive job is easy to miss because the source indices still look healthy until retention deletes them. Alert on failed or stalled archive runs and on gaps in the archive by date, and send those alerts to a team that owns them. Confirm before go-live that failures trigger a notification through the Graylog alerting path your team uses.

Check archive integrity regularly

Compare checksums of archived files with the values recorded at write time on a sampling schedule, and restore a sample to a test index to confirm the data is readable. Ask your backend vendor how it protects stored objects against corruption and how it reports repair events. Keep the results with the archive records so you can show the checks happened.

Archiving Graylog to object storage: cost, effort and what you need to decide

Does archiving Graylog logs to object storage require an Enterprise license?

Archiving to object storage is a licensing question, so confirm it against the current Graylog product pages before you plan. Graylog Enterprise adds a data lake with archive restore, data tiering and 24/7 support, according to the Graylog pricing page. Graylog Enterprise starts at 15,000 US dollars per year, with a minimum of 10 GB per day or 100 Graylog Compute Units, and licensing is based on processed data with no per-user or per-device fees. Review the Graylog pricing page to confirm which archive features sit in each tier before you commit to a backend design.

How does archiving cost change with log volume and retention?

Archive cost scales with the volume you keep and the years you keep it, plus the cost of the storage class and any requests or restores you pay for. Your storage bill is one recurring line item, and the pricing model for the object store itself varies by provider and by storage class. Daily ingest multiplied by retention gives the raw data footprint, and your compression choice changes how much of that reaches the bucket. Ask your storage provider for current per-gigabyte and request pricing, and model a few retention scenarios (for example, one year and seven years, illustrative only) against your daily ingest before you choose a tier.

How long does setting up Graylog archiving take, and what drives it?

Setup time depends on how many systems are involved and how much of the backend is new to your team. The main drivers are whether you already run an S3-compatible store, how much access control and retention policy work is needed, how many log sources and index sets must be mapped, and how long validation of restores takes. Teams with an existing bucket, IAM model and retention standard usually finish faster than teams building storage from scratch. Ask any partner for a duration only after they have written scope, because the number of index sets, the compliance requirements and the test plan decide the effort.

Can our internal team set up Graylog archiving without a partner?

Yes, a capable platform team can build this in-house if it has experience with Graylog, object storage and access control. The work includes choosing a backend, configuring credentials and buckets, setting compression and checksum settings, defining archive retention, and testing an end-to-end restore. The risk is less the initial setup than keeping the design correct over time, such as retention changes, backend upgrades and periodic restore tests. Decide early who owns each part, and document the runbook so that the archive keeps working after the original builder moves on.

Which object storage backend should we choose for Graylog archives?

Choose the backend based on your data residency, durability needs, operations capacity and whether you need object locking. Amazon S3 Standard is designed for 99.999999999% durability and 99.99% availability across at least three Availability Zones, according to the Amazon S3 storage classes documentation. S3 Object Lock supports write-once-read-many retention with governance and compliance modes and legal holds, but it works only in versioned buckets. Self-hosted options such as Ceph RADOS Gateway offer S3 API compatibility, while Garage does not implement versioning or object lock, so confirm each feature you depend on with the vendor before you build.

How do we verify archive integrity and prove logs were not altered?

Verify integrity with checksums at write time, periodic checksum verification, and restore tests that compare results with the original messages. Enable versioning and, where your evidence needs it, object lock so that deleted or overwritten archives are detectable or blocked. CIS Control 11, which covers data recovery, asks for an isolated copy of recovery data, so keep at least one archive copy in a separate account or location. Ask your team to document how checksums are generated, where they are stored and who may change retention settings.

What should we ask for in a quote for Graylog archiving?

Ask for a written scope that names every deliverable, so you can compare quotes on the same basis. Request the list of Graylog and backend versions covered, the index sets and retention rules in scope, the backend and storage design, the integrity and access controls, the restore test plan and the documentation handover. Also ask which items are one-time, such as design and setup, and which recur, such as support or periodic restore testing. DeployOpen does not publish a price list and provides a quote based on the written scope, so confirm which of these items your quote covers and which remain with your team.

Limits to confirm in your environment before you commit

Archiving Graylog logs to object storage depends on several limits that the documentation does not settle for your version, backend and regulator. Confirm each one with the vendor or with your own tests before you commit to a design, because a wrong assumption here affects both restore plans and audit evidence.

Graylog version and archive support

Archive features depend on your Graylog edition and release line, and the current documentation covers version 7.2. Confirm that archiving and restore are available in the exact version and license you will run, and check the upgrade notes for any change to archive formats or index handling. Ask the vendor whether archives written by one release can be restored by a later one, and test that path on a copy of real data before relying on it.

S3 API compatibility of the backend

Object storage products implement the S3 API to different depths. Confirm which operations your archive job needs, such as multipart uploads, lifecycle expiration, checksums and presigned access, and test them against the chosen backend. Some S3-compatible stores omit versioning, object locking or bucket policies, which can block the controls described in the rest of this section.

Object Lock only in versioned buckets

On Amazon S3, Object Lock works only in versioned buckets and offers governance and compliance retention modes plus legal holds. Confirm whether your backend offers equivalent write-once controls at all. If it does not, plan compensating controls such as a separate off-site copy with restricted delete rights, and document that choice for your auditor rather than assuming immutability.

Restore speed and search backend load

Restoring archived messages into a searchable index adds load to the search cluster and takes time that depends on volume, compression and network throughput. Ask how long a restore of a representative month takes on your hardware, and whether restores can run on the leader only or in batches. Schedule large restores away from peak investigation work so analysts are not competing with the restore job.

Durability claims for non-AWS backends

The 11-nines durability figure published for Amazon S3 Standard describes that service, not a self-hosted store. Self-hosted backends protect data through replication, erasure coding and the disks under them, so ask the vendor for its replication model and failure assumptions and test a node loss before go-live. Keep a copy of archives in a separate failure domain, consistent with the recovery-data isolation idea in CIS Control 11.

Retention rules set by your regulator

The retention period, the immutability requirement and the acceptable storage location come from your regulator or your legal and compliance team, not from Graylog or the storage vendor. Get the required period and any evidence obligations in writing before you choose a lifecycle rule, since deleting an archive early can breach the obligation and deleting late raises storage cost. Confirm how your auditor will verify archive integrity before you finalize checksums and retention settings.

Where archive design fits with sizing and warm tiers

Archive design is one of three tiers that hold Graylog data, and the right choice depends on how long each log type must stay searchable. Hot storage in Graylog is the set of indices that search reads directly, and the index model manages it through rotation by message count, size or time and through retention that deletes the oldest indices. Index set templates decide those settings for each log source. Before choosing a tier, sort your sources into retention bands (for example, the most recent days that analysts query daily, the months that support investigations, and the years that auditors may request) and confirm with your compliance owner which bands carry a legal or contractual minimum.

Size the hot tier first, because it sets the cost of the search cluster. The Graylog system architecture guidance estimates log storage as daily ingest times retention days times 120 percent, and the message journal needs its own SSD capacity of 3 to 5 times daily ingest. Only data inside the hot retention window counts toward that formula. Warm tier snapshots move older indices to a cheaper repository, but the Enterprise licensing, minimum hot days and OpenSearch or Data Node version required for that tier must be checked before you commit to it. Read the sibling pages on sizing and on the warm tier for those figures, and confirm them against your own licence.

Archives sit beyond both tiers. Set archiving to run before index deletion, so that messages leave the search cluster in a verifiable copy rather than disappearing when the oldest index is removed. Write the final decision as a table for each source: hot days, warm days if any, archive years, the bucket or repository that holds the archive, and the owner who approves a restore. Then test one restore path end to end. The restore process is covered on its own page, but the archive design must be proven to allow it, or the retention period you promise to auditors is only a plan.

Archiving before index deletion: how to avoid losing logs

Does Graylog delete old indices by default?

Yes, Graylog's index model rotates indices by message count, size or time and applies retention that deletes the oldest indices once the configured limit is reached. Deletion happens on the search cluster, and nothing is copied out first unless you configure it. Check the retention strategy on each index set before you assume logs are kept, because a short retention setting removes data on schedule without any warning in the search interface.

What order should archive jobs run in before retention deletes an index?

Archive first, verify second, delete last. Confirm that the archive job for each index has completed and that the objects exist in the bucket before retention is allowed to remove the source index. In practice this means the archive window must finish before the retention trigger fires, with enough margin to cover a failed run and a retry. Schedule retention so it never runs against an index that has not been archived and checked.

How do I make sure an archive job finished before an index is deleted?

Use a check that reads the archive result, not just the job status. Compare the message count and index name recorded at archive time with the objects written to object storage, and keep the checksum manifest with the archive. Treat a job that reports success but writes fewer objects than expected as a failure. Record the result in a log that the retention process can read, so deletion is blocked when verification is missing.

What should I test before turning on automatic deletion?

Test the full chain in a non-production environment with a copy of real retention settings: ingest, index rotation, archive, checksum verification, deletion and restore. Confirm that a restored index returns the same message counts as the original and that an interrupted archive job leaves the source index in place. Also test what happens when the bucket is unreachable, because deletion should stop rather than continue. Keep the test results as evidence for your retention review.

Can my own team set up archiving before index deletion, or should we hire a partner?

Your team can do it if it has someone who owns Graylog retention, storage administration and scripting, and can write a runbook for failures. A partner is useful when the scope includes several index sets, a verification design you must defend to auditors, and tests that need to be repeated after upgrades. DeployOpen works on a fixed scope agreed in writing, so the deliverables, test cases and handover are defined before work starts. Ask any partner to show how it blocks deletion when archiving fails.

What does this work cost, and what drives the price up?

The cost has three parts: the one-time design and build work, the storage you keep paying for every month, and any ongoing support you choose. Storage cost depends on how much data you retain, how long, and the storage class you pick. Build effort rises with the number of index sets, the number of backends, the verification you need to explain to auditors, and how many restore tests you run. Ask for a quote that separates one-time work from recurring storage and support, and states the assumptions about volume and retention.

Which object storage backend should hold the archives?

Choose the backend based on durability, immutability needs and who operates it. Amazon S3 Standard is designed for 99.999999999% durability and stores data across at least three Availability Zones, and S3 Object Lock supports write-once-read-many retention in versioned buckets. Self-hosted options such as Ceph RADOS Gateway offer S3-compatible APIs, while Garage lacks object locking and versioning, so it is a poor fit where immutability is required. Confirm feature support against the backend's own documentation before you commit.

How DeployOpen scopes and delivers Graylog archiving

DeployOpen plans and builds Graylog archiving to S3 or compatible object storage inside your own cloud accounts, data centers or restricted networks. Each engagement starts with a written scope that fixes the backend, retention rules, integrity checks and handover deliverables before any configuration changes are made.

How we scope

Confirm which Graylog edition, version and search backend your archiving design depends on before work starts.
Record the retention periods, legal hold needs and restore expectations that set the archive design.
Agree in writing which object storage backend, compression and checksum method, and archive-before-deletion steps are in scope.

How we build

Configure the archive target, access credentials and bucket settings in your environment, with retention controls set as agreed.
Set up archiving so messages are written and verified before the related indices are removed by retention.
Test restores from the archive into a non-production search cluster and record checksum verification results.

How we hand over

Deliver the written configuration, retention schedule, access model and restore runbook for your operations team.
Document the archive integrity checks, what they cover, and how failures are detected and escalated.
Walk your engineers through routine archive checks and restore steps so your team can run them without us.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer, scoped separately from the build.
Review archive storage growth, backend changes and Graylog upgrades against the agreed retention plan on a schedule you choose.
Agree the support boundary in writing so responsibilities for backend storage, Graylog and audit evidence stay clear.

Archiving Graylog logs to S3 or object storage