How Thanos uses Ceph RGW for Prometheus long-term storage
Prometheus keeps its own time series on local disk, and its default retention is 15 days, so metrics that your teams need for capacity planning, incident review or audits have to leave the Prometheus server. Thanos can keep that history in object storage you operate, and Ceph RGW is a supported destination because Thanos treats S3-compatible stores as a production client. The Thanos storage documentation describes its Minio client as configurable against other S3-compatible stores, and Ceph is named as one of them. You do not add Thanos to a Prometheus setup for a single function. Its components are optional and each writes or reads a different part of the data path.
The components that write blocks to the bucket are the sidecar, Receive and Compact. The Thanos sidecar runs beside each Prometheus server and uploads TSDB blocks as Prometheus produces them, every 2 hours. If Prometheus pushes metrics through remote write instead, Thanos Receive uploads blocks on the same 2-hour default and identifies tenants through the THANOS-TENANT header. The Thanos compactor then downsamples older data, and it must run as a singleton against each bucket, so plan one compactor per bucket and confirm the deployment prevents a second instance from starting. The supported method for pointing these components at Ceph RGW is the object storage configuration file that Thanos reads, with the endpoint, bucket, credentials and TLS settings you set for your gateway. Confirm each key against the Thanos object storage documentation for the Thanos version you deploy.
The Ceph-specific detail to confirm is list_objects_version: v1. Thanos advises this setting for S3-compatible APIs that lack ListObjectsV2, and some Ceph versions fall into that group. Check whether your Ceph release supports ListObjectsV2 on RGW by testing a list call against your bucket before you go live, and set the value only if the test requires it. Record the Ceph release, the Thanos version and the result of that test in your change record, because both sides change over time. Thanos publishes frequent releases, so re-run the test after each upgrade of either component.
What you need before connecting Thanos to RGW
Thanos needs a working Prometheus, an object store it can reach over a verified TLS connection, and credentials scoped to one bucket. Each card below is a prerequisite to confirm before you write a single Thanos configuration file, because most failed first attempts trace back to one of these items.
Prometheus with persistent disk
Thanos requires one or more Prometheus installations at v2.2.1 or later, each with persistent disk. The Thanos sidecar reads blocks from that local disk and uploads them, so a Prometheus on ephemeral storage loses data before it reaches the bucket. Confirm the version on every Prometheus you plan to attach, and confirm that the sidecar will run beside it with access to the data directory. If you plan to upgrade Prometheus to the current LTS line before the rollout, do that first so the sidecar is tested against the version you will keep.
Ceph RGW service and Ceph release
You need a running Ceph cluster with an RGW service reachable from the Thanos hosts. Check the Ceph release you run against the releases page, where Squid and Tentacle are the actively maintained lines, and confirm that your release lists ListObjectsV2 support. If it does not, plan to set the list_objects_version setting described in the storage configuration. Ask your storage team who owns RGW sizing and whether the gateway is already serving other workloads, since Thanos compaction adds sustained read and write load.
Dedicated bucket and access key
Create a bucket used only for Thanos blocks, with a dedicated RGW user and access key that cannot read other buckets. Thanos blocks are written by the sidecar or Receive, read by the store gateway, and rewritten by the compactor, so a shared bucket makes permission mistakes and retention collisions likely. Record who owns the key, where it is stored, and how it is rotated. The compactor must run as a singleton per bucket, so decide now which single host or workload is allowed to run it.
Network path and TLS certificates
Confirm that every Thanos component that touches object storage can reach the RGW endpoint on its HTTP or HTTPS port, including through any firewall, load balancer or proxy in between. If the endpoint uses HTTPS with a certificate issued by an internal certificate authority, make that CA trust bundle available to every Thanos host and verify it with a test connection before deploying. Ask the network team whether egress rules restrict traffic from the Prometheus hosts to the object store, because blocked uploads often appear only as delayed retention rather than an obvious error.
Thanos release to deploy
Deploy a currently published Thanos release. The latest published releases at the time of writing are in the v0.42.x line, with v0.42.4 published July 30, 2026. Pin the exact version in your deployment manifests, read the release notes for any storage or configuration changes, and rehearse the upgrade on a non-production bucket. Thanos is licensed under Apache-2.0 and is a CNCF Incubating project, so confirm license and support terms with your legal and procurement teams before rollout.
Prometheus local retention settings
Before the sidecar uploads blocks, set Prometheus minimum and maximum block duration to equal values so local compaction is disabled. Otherwise the sidecar can upload data that Prometheus later rewrites, which the Thanos sidecar documentation warns can corrupt uploaded data. Review local retention too, since Prometheus keeps its own copy until its retention limit is reached. Agree with the platform owners which team is responsible for these Prometheus flags, because they change how every Prometheus in the fleet behaves.
Configuring the Thanos object storage client for Ceph RGW
This sequence takes an empty Ceph RGW bucket and a Prometheus server to the point where Thanos uploads blocks to Ceph, can read them back through a store query, and compacts them without a second writer.
Create the RGW bucket and service user
The Ceph administrator creates a dedicated RGW user for Thanos and a bucket that holds only Thanos blocks, so retention and access can be managed separately from other data. Use a user with the minimum bucket permissions needed, and record the access key and secret key in your secrets store rather than in shared configuration files. Confirm the endpoint is reachable from the Prometheus host over HTTPS and that a test object can be written and listed with the same credentials. The step is complete when that test succeeds from the host that will run the sidecar and the compactor.
Write the object storage configuration file
The platform engineer writes the Thanos object storage configuration with the S3 type, the RGW endpoint, the bucket name, the access key, the secret key and the signature settings your RGW deployment requires. Keep the file readable only by the service account that runs Thanos, and point each component at the same file so every writer and reader uses one configuration. Confirm the exact keys and their supported values against the Thanos object storage documentation for the release you deploy. The step closes when Thanos starts with the file and reports no authentication or endpoint errors in its logs.
Set list_objects_version for older RGW versions
Check whether your Ceph RGW version supports ListObjectsV2. Where it does not, add the list_objects_version setting with the value v1 to the object storage configuration, as the Thanos storage documentation advises for S3-compatible APIs that lack that call. Confirm the behavior in your own environment by testing a listing with and without the setting, because the Thanos documentation names some Ceph versions without listing which ones. The step is complete when the sidecar or compactor lists the bucket without errors on the chosen setting.
Align Prometheus block duration for the sidecar
If the Thanos sidecar will upload blocks, the Prometheus engineer sets the minimum and maximum block duration to the same value, which disables local compaction. The Thanos sidecar documentation warns that leaving them unequal can corrupt uploaded data, so the change belongs in the Prometheus configuration before the sidecar starts. Restart Prometheus under your change process and confirm the flags are in effect. The step closes when Prometheus runs with equal values and its local data directory shows blocks being cut at the configured interval.
Start the sidecar and confirm the first upload
Start the Thanos sidecar next to each Prometheus server with the shared object storage configuration. Prometheus produces two-hour blocks, and the sidecar uploads them to the bucket as they appear, so the first upload normally follows the first completed block. Verify that a block directory with a meta file appears in the bucket and that the sidecar log shows a successful upload. The step is complete when the bucket holds a block from the Prometheus server and the object count matches the local blocks.
Verify a store query reads the blocks back
Start a Thanos store gateway pointed at the same bucket and connect a query component to it. Run a query over a time range older than the local retention to confirm Thanos serves data from Ceph RGW rather than from Prometheus. Check the query latency and error rate for the store gateway, and note any listing or range-read errors that point to RGW configuration problems. The step closes when a historical query returns the expected series from the object store.
Start the compactor as a single instance
Run the Thanos compactor against the bucket as a singleton, because the compactor documentation requires one instance per object storage bucket. The compactor merges blocks and downsamples older data, which generates reads, writes and deletes on RGW, so watch the RGW request rate, OSD load and bucket object count after it starts. Plan capacity for that background load with your Ceph operations team before the first large compaction run. The step is complete when the compactor logs its first completed pass and the bucket shows compacted or downsampled blocks.
How compaction load lands on the object store
The Thanos compactor is the component that generates most of the background load on the bucket after ingest, and it has to be planned for separately from the sidecar or Receive upload path. It downsamples raw data older than 40 hours to 5-minute resolution, then downsamples 5-minute data older than 10 days to 1-hour resolution. It also compacts smaller blocks into larger ones. Each pass reads existing blocks from RGW, writes new downsampled and compacted blocks, and deletes the originals once the replacements are in place. Because the compactor must run as a singleton against each bucket, all of this work runs through one process, so its throughput is set by how fast that one instance can read and write objects over the network to RGW. Run a single compactor per bucket and do not scale it horizontally against the same bucket.
For RGW, the result is a mix of sequential reads, large object writes and a steady stream of deletes, arriving in bursts when a downsampling window closes. The compactor's requests reach the bucket index as well as the data pool, so the number of objects in the bucket and the index sharding of that bucket both affect latency. Ask your Ceph team to confirm how the bucket index is sharded for the expected block count, and to check RGW request rate and latency during a first full compaction cycle, not just during steady ingest. Also confirm the list_objects_version setting in your Thanos configuration, since listing behavior affects how the compactor discovers blocks on older Ceph releases.
Sizing questions follow from this profile: peak request rate during a compaction pass, bucket index load as block counts grow, the network path between the compactor and RGW, and how long a full compaction pass takes against your retention. The Ceph hardware recommendations say to benchmark before buying, because no two clusters are alike, so run the compactor against a representative copy of your metrics before committing to RGW hardware. Size memory for peak use rather than calm periods, since compaction and query load can coincide. Treat these figures as things to measure in your own environment, not numbers to copy from another deployment.
Mapping Thanos components to Ceph RGW objects
This table maps each Thanos component to what it reads or writes in the Ceph RGW bucket, the Prometheus setting it depends on, and the Ceph-side resource it consumes. Read each row as a checklist item for the bucket, the RGW user and the access key that component needs.
| Component | Reads or writes in the bucket | Prometheus or Thanos dependency | Ceph-side resource consumed |
|---|---|---|---|
| Sidecar | Uploads TSDB blocks as Prometheus produces them, roughly every 2 hours | Runs beside each Prometheus; min and max block duration must be equal to disable local compaction | Bucket write access and an RGW user with a dedicated access key |
| Receive | Uploads TSDB blocks to the bucket every 2 hours by default, one set of blocks per tenant | Prometheus pushes through remote write; tenants are identified by the THANOS-TENANT header by default | Bucket write access and an RGW user; tenant layout should be planned before the first upload |
| Compact | Reads blocks and writes downsampled blocks back to the bucket; downsamples data older than 40 hours to 5-minute resolution, then older 5-minute data to 1-hour resolution | Must run as a singleton against each bucket, so only one compactor should hold the bucket | Bucket read and write access, plus local scratch disk for compaction work |
| Store | Reads blocks from the bucket to serve historical queries | Queried by Thanos Query; does not write to the bucket | Bucket read-only access; RGW load grows with query volume over old data |
| Query | Reads nothing directly from the bucket; fans out requests to Sidecar, Receive and Store | Connects to Prometheus through Sidecars and to stored blocks through Store | No bucket credentials; confirm which components it reaches over the network |
Securing the metrics bucket and access path
A Thanos deployment writes historical Prometheus data to one bucket, and that bucket is reachable by every component that reads or compacts it. The controls below cover who can reach the bucket, how traffic is protected, where credentials live, and how long the data is kept.
Dedicated RGW user with bucket scope
Create a separate RGW user for Thanos rather than reusing an administrator or a shared application account. Grant access only to the metrics bucket, and check that the sidecar, store gateway, compactor and any receiver all use the same narrowly scoped identity. Confirm with your Ceph version which bucket policy and capability settings are available, since the exact options depend on the release you run.
TLS on the RGW endpoint
Terminate TLS on the RGW endpoint so that metric samples, object names and credentials are not sent in clear text between Thanos and the gateway. Check that the Thanos object storage configuration points at the HTTPS endpoint and that the certificate chain is trusted by every Thanos component. Keep the certificate renewal process on the same schedule as other services that use the gateway.
Keys out of plain configuration
The S3 access key and secret key sit in the Thanos object storage configuration file, which is easy to copy into version control or a ticket. Load the file from a secret store or mount it from a Kubernetes Secret with restricted read permissions, and use --objstore.config-file only with a file that is not world-readable. Rotate the key pair on a planned schedule and after any staff change.
Messenger v2 secure mode for cluster traffic
Treat cluster traffic as a separate hardening item from the client path. Ceph's messenger v2 protocol, which uses port 3300, offers a secure mode that encrypts post-authentication traffic with AES-GCM and adds integrity checks. Confirm that secure mode is enabled for the monitors and OSDs in your cluster, and that the change is planned with your Ceph upgrade and maintenance process.
Sensitive labels in metric data
Prometheus series carry label values such as hostnames, tenant names, pod names and sometimes user identifiers, and those labels persist in every block Thanos uploads. Review label sets with the data owner before enabling long-term storage, and drop or relabel fields that the data policy does not allow you to keep. Treat the bucket as a copy of production telemetry, not as a cache.
Retention and access under data policy
Set retention on the bucket and the compactor to match the retention period your data policy defines, and confirm who may read the bucket, list its objects and restore data. Document the owner, the access review cadence and the deletion procedure, since a Thanos bucket grows without an automatic expiry unless you configure one. Ask your compliance lead which retention and access rules apply before any production write.
Why Thanos fails to list or upload blocks on Ceph
This table pairs each common Thanos failure against Ceph RGW with its likely cause, the fix and a check to confirm it. Read it when list calls error, uploads look wrong or the bucket contents do not match what Prometheus produced. Most failures trace back to the list API version, local compaction left on, a second compactor or endpoint and TLS settings, so check those first.
| Failure | Likely cause and fix | Check to confirm |
|---|---|---|
| List calls fail on the bucket | The S3 API version lacks ListObjectsV2, which some Ceph versions do not provide. Set list_objects_version: v1 in the Thanos object storage configuration. | Run Thanos with the setting and confirm bucket listing completes without errors in the sidecar, store gateway and compactor logs. |
| Uploaded blocks are corrupted | Prometheus still runs local compaction because min and max block duration differ. Set both to the same value so the sidecar uploads intact blocks. | Confirm the Prometheus flags show equal min and max block durations, then check that uploaded block directories open cleanly in a test query. |
| Duplicate or missing data in the bucket | More than one compactor runs against the same bucket. Keep a single compactor per bucket and stop any leftover or replicated instances. | List running compactor processes across all hosts and confirm only one targets this bucket before reviewing query results for gaps or duplicates. |
| Endpoint or TLS connection errors | The endpoint URL, port or certificate chain does not match the RGW gateway. Confirm the endpoint, use the correct scheme and trust the issuing certificate authority on every client host. | Test a basic S3 request from a Thanos host with the same credentials and CA bundle, and confirm a successful response over HTTPS. |
| Signature or request rejected | Access keys are wrong or expired, or the client path style does not match the RGW bucket addressing. Correct the keys and the addressing setting in the configuration. | Use a standard S3 client with the same keys and endpoint to list the bucket, and confirm the request succeeds with the same addressing mode. |
| Clock skew breaks requests | Host time on Thanos components differs from the Ceph gateway enough to invalidate signed requests. Sync all hosts to the same time source. | Compare system time across Prometheus, Thanos and RGW hosts and confirm the time synchronization service reports them in sync. |
| Blocks missing after a Ceph change | The bucket name, prefix or object store configuration changed between components, so readers and writers point at different data. Use one shared configuration file everywhere. | Compare the object storage configuration on each component and confirm the bucket and prefix values are identical. |
Questions buyers ask before connecting Thanos to Ceph RGW
How much does it cost to connect Thanos to Ceph RGW?
Neither Thanos nor Prometheus has a paid edition, so the cost comes from infrastructure, storage, implementation and the people who run the system. Thanos is licensed under Apache-2.0 and Prometheus is licensed under Apache-2.0, so there are no license fees for either project. The costs you will pay for are the Ceph cluster capacity that holds the blocks, the Prometheus servers with persistent disk, the compute for Thanos components, and the implementation and operating effort. Ask for a quote from a written scope that lists each of these separately, and state whether it covers one-time setup, recurring support or both.
How long does a working Thanos setup on Ceph RGW take?
The duration depends on how many Prometheus servers you connect, whether you use the sidecar or Receive, and how much of the Ceph side is already in place. Thanos requires one or more Prometheus v2.2.1 or later installations with persistent disk, plus object storage, so an existing Prometheus estate and a working RGW endpoint shorten the work. A written scope should list the Prometheus version, the sidecar or Receive design, the bucket layout, the compactor placement and the verification steps, because each of these changes the timeline. Ask for the schedule as a list of milestones with the work each one completes.
What does our team need to provide for the project?
Your team needs to provide access and decisions that only your organization can make. This usually includes:
- Prometheus servers, their current retention settings and the hosts or namespaces where they run
- A Ceph RGW endpoint, a bucket and a dedicated user with credentials that can be stored securely
- Network paths between Prometheus, Thanos components and the RGW endpoint, with TLS certificates your trust chain accepts
- Named owners for Prometheus, Ceph and the on-call process after handover
Also confirm the Ceph release in use, because the list_objects_version setting depends on which S3 APIs the endpoint supports.
Can our own team set up Thanos on Ceph RGW without a partner?
Yes, a capable platform team can do this without an external partner, because both projects are open source and their documentation covers the configuration. The work that needs careful attention is the sidecar configuration, where Prometheus minimum and maximum block duration must be equal to disable local compaction, the list_objects_version: v1 setting for S3-compatible APIs without ListObjectsV2, and the singleton placement of the compactor for each bucket. Teams without recent Prometheus or Ceph operations experience often choose to scope a fixed piece of work with a partner and then run the system themselves. Decide this by comparing your staff's time and experience against the checklist in a written scope.
What drives the cost of compaction on the object store?
Compaction load depends on how much data you keep and how it is written. The Thanos compactor downsamples raw data older than 40 hours to 5-minute resolution, and 5-minute data older than 10 days to 1-hour resolution. The compactor must run as a singleton against each bucket, so its work is serialized and it reads and writes many objects. Load rises with the number of series, the retention period, the number of blocks and the request rate on RGW. Ask your Ceph team to size RGW request capacity and OSD throughput against a test that reflects your block count and retention, and benchmark before buying hardware.
How does retention change the storage capacity we need?
Storage grows with the number of active series, the sample rate, the retention period you set and the downsampling you keep. Prometheus local storage defaults to 15 days, and Thanos keeps historical blocks in object storage, so long retention moves the bulk of the data to Ceph. Downsampled blocks take less space than raw data, but you still need to decide how long to keep each resolution. Ask for a capacity estimate that uses your measured ingest rate, the replication or erasure-coding policy chosen for the RGW data pool and headroom for growth. Plan for spare capacity so that a full cluster does not block block uploads.
Which Prometheus and Thanos versions should we plan around?
Plan around the versions your team can keep patched, not the newest release alone. The Prometheus release cycle begins a new minor release every 6 weeks, and a minor release generally stops receiving bug fixes after those 6 weeks. Prometheus long-term support releases receive bug fixes for one year, and the current LTS line is 3.13, supported until July 31, 2027. Thanos published v0.42.0 on July 8, 2026 and v0.42.4 on July 30, 2026. Confirm the version policy with the vendor pages before you select a target, and include upgrade work in the written scope.
Does Thanos replace the need for Prometheus local storage?
No. Thanos reuses the Prometheus storage format and works on top of existing Prometheus deployments, so Prometheus still collects, scrapes and stores recent data locally. Prometheus local storage is not clustered or replicated and should be managed like any other single-node database, so you still need disk sizing and backup decisions for those servers. The sidecar uploads blocks every 2 hours, and Receive uploads them every 2 hours by default when Prometheus pushes metrics. Plan both layers together so that outages on one do not leave gaps in your long-term view.
What changes for your team after go-live
After go-live, the work moves from building the Thanos path to keeping it correct. Four routines need named owners: Thanos upgrades, compactor runs, RGW capacity and request load, and restore tests of blocks. Confirm with your Thanos owner which release line you will follow and how upgrades are tested against a staging bucket first. Thanos releases are published on its GitHub releases page, and the release cadence of your Prometheus version should be checked at the same time, since Prometheus minor releases stop receiving bug fixes after six weeks.
The compactor is the component most likely to change load on the object store. It must run as a singleton against each bucket, and its downsampling passes read and rewrite large volumes of blocks. Watch each run's duration and error count, and watch the RGW request rate, bucket object count and used capacity over time, not only at peak. Confirm with your Ceph owner which alerts cover RGW pool fullness and gateway latency, and whether the compactor's I/O is scheduled away from client traffic. Thanos's list_objects_version setting also belongs in the upgrade checklist, because a Ceph upgrade can change how the S3 API responds.
Restore testing is the check that proves the long-term data is usable. On a schedule your team sets, pick a block from the bucket, pull it into a separate Thanos Store or query path, and confirm that queries over historical ranges return expected results. Record the outcome and the time taken. Ask your Ceph owner whether RGW data is protected by replication or erasure coding, and whether a second copy exists outside the cluster, since Thanos has no independent copy of blocks once they are uploaded and local compaction is disabled.
