Why teams need Prometheus long-term storage with remote write
If you run Prometheus, you have probably hit the limit of its default local retention, which is 15 days. That window is fine for debugging last week's incident but too short for capacity planning, audit evidence or a year-over-year comparison of traffic and error rates. Local storage also has no clustering or replication. The Prometheus storage documentation says it should be managed like any other single-node database, so keeping history for longer means either growing one server's disk indefinitely or sending the data somewhere built for the job. Remote write is the mechanism for the second option: Prometheus sends its samples over the remote write protocol to an external long-term store while it keeps scraping and evaluating rules locally.
The idea is simple, but the operating model is not. You now run two storage systems instead of one. The external store needs its own sizing for ingest rate, active series, retention and query load, and it needs its own backup, upgrade and access controls. Prometheus buffers outgoing samples in a remote write queue. If the receiving store slows down or becomes unreachable, that queue grows, and the queue settings decide how much data is held in memory and how far behind the remote copy can fall. Teams often discover this only during an outage, so the queue depth, retry behavior and the point at which samples are dropped should be understood before go-live rather than after the first incident.
Query behavior also changes. Recent data is served from the local TSDB, while older data may come back from the remote store through remote read or from a query layer placed in front of both. Those two sources do not always behave identically. Functions, step alignment, staleness handling and label semantics can produce slightly different results depending on where a sample was read from, and dashboards or alert rules built against one source may not match the other. Confirm with your chosen store which PromQL features it supports and how it handles deduplication if you run replica pairs, since both are common sources of confusing numbers.
Prometheus documents that it values reliability over precision, and it is not a good fit where every single sample must be exact, such as per-request billing. That position carries over to the long-term design. Do not assume that remote write will keep every sample under load, a network partition or a restart. Build a test that stops the receiving store for a defined period, confirms what the queue does, restores the store and compares local and remote results for the same time range. Record the queue metrics, the retention you actually need for each use case, and the query paths that each team relies on. The design is sound when those tests pass, not when the configuration looks correct on paper.
Which Prometheus and external store components this design touches
Remote write to long-term storage involves several Prometheus components that each hold a different part of the data path. Read the cards in order: the local database and retention decide what is kept on the server, the sender and remote read decide what leaves it and how it comes back, and the external store and recording rules decide what arrives and what it costs to keep.
Local time series database
Prometheus writes samples to a local time series database organized into two-hour blocks, with a write-ahead log that protects recent data before a block is compacted. This database is not clustered or replicated, so it should be managed like any other single-node database. If the disk fills or the volume is lost, samples not yet sent to remote storage are lost with it. Check disk capacity, filesystem behavior and backup of the data directory before relying on it.
Local retention settings
The default local retention is 15 days, and it can be set by time, by size, or both. Retention controls how far back local queries can go when the remote store is unavailable, and it determines how much disk the server needs. Setting size limits too low can delete blocks that the remote write sender has not yet shipped. Confirm the retention values against your query needs and your disk sizing before go-live.
Remote write sender
The remote write sender reads samples from the write-ahead log and pushes them to the external store over the remote write protocol. Its queue settings control how many samples are buffered, how many parallel shards send them, and how long retries continue. Misconfigured queues cause growing lag, dropped samples or memory pressure on the Prometheus server. Measure lag and failed sends after each change to these settings.
Remote read path
The remote read path lets Prometheus query historical data held in the external store alongside local data. It is useful for longer lookbacks, but queries that span both sources depend on the store's latency and throughput, and heavy queries can slow the Prometheus server. Decide which dashboards and alerts may use remote data, and test their response times against realistic time ranges.
External long-term store
The external store accepts the remote write protocol and keeps samples beyond local retention. Its supported protocol version, ingestion limits, retention behavior, multi-tenant settings and compatibility with your Prometheus version are the decisions that matter most. Confirm all of these with the store's vendor or project before you commit, and ask how it handles out-of-order or duplicate samples and what happens during its own outages.
Recording rules that reduce volume
Recording rules precompute expressions and store the results as new series, and you can choose which series to forward. Sending only aggregated or high-value series lowers storage cost and ingestion load on the external store. Dropping raw series that dashboards still need, or writing rules that produce high-cardinality labels, breaks queries and inflates storage. Review rule outputs and the series filters on the sender before sending everything.
Local retention only, remote write or remote read: which design fits
This table compares four ways to keep Prometheus data beyond its local window, from extending disk on one server to pre-aggregating before samples leave. Read each row against your retention target, query needs and the team that will run a second system. The main takeaway is that remote write solves retention, while remote read adds a query path whose consistency and latency you must test before relying on it.
| Option | When it fits | Operational cost | Confirm with the store vendor or team |
|---|---|---|---|
| Extend local retention on one node | Retention needs are modest, one team queries the data, and losing the host means losing history you can accept | Disk on the Prometheus host grows with retention and series count; the server stays a single point of failure and needs its own backup plan | Not applicable, but confirm that your volume sizing and snapshot or backup approach cover the full retention window |
| Remote write to an external store, local queries for recent data | Long retention is required, dashboards mostly read recent data, and a separate store is acceptable for history | A second system to run, monitor and secure; a write queue that can back up during store outages; per-series cost on the remote side | Write throughput limits, how the store handles out-of-order and duplicate samples, retention and downsampling options, authentication for the write endpoint |
| Add remote read so one query path spans both | Engineers need one PromQL endpoint across local and historical data and can accept slower queries on old ranges | Queries that reach the remote store add latency; results can differ at the seam between stores; more load on the store from read traffic | Query latency at your typical range, how the store fills gaps during replay, whether PromQL results match local data, and read authentication |
| Pre-aggregate with recording rules before samples leave | High-cardinality raw series would be costly to store, and aggregated history meets your reporting and capacity needs | Recording rules must be written, versioned and tested; raw detail is lost for series you do not record; rules add evaluation load on the server | Whether the store bills or limits by series count, how it treats recording rule names, and whether it supports downsampling so you avoid duplicating the same data |
How to configure remote write step by step
This sequence moves from sizing the metrics to a verified canary, so the external store receives only what it can keep and query. Each step closes with a check you can run on the Prometheus instance or the store. Remote read is optional and is added only when queries need to span both stores.
Inventory metrics and their cardinality
The platform team lists which jobs and exporters feed Prometheus, how many active series each produces, and which labels create the most series. Use the TSDB status page and PromQL queries that count series per metric name to find the heavy contributors. Record the results, because they set the write volume the external store must absorb and strongly affect its cost. Done when you have a ranked list of metrics by series count and a note of any label with unbounded values, such as user or request IDs, that should be dropped before samples are sent.
Select and verify the external store
The platform owner shortlists stores that accept the Prometheus remote write protocol and confirms, using the store's own documentation, which protocol version, authentication methods and TLS settings it expects. Confirm that the store's endpoint is reachable from the network where Prometheus runs and that client certificates or credentials can be issued to it. Ask the vendor for ingest limits and for how the store handles out-of-order or duplicate samples. Done when a test write of a single sample is accepted by the store's endpoint and returns a success response.
Set retention on both sides
Set local retention on each Prometheus instance to cover the window needed for alerting and recent dashboards. The default local retention is 15 days, and it can be set by time or by size. Set the retention in the external store separately, based on compliance requirements and the queries your teams run. Local storage is single-node and not replicated, so manage it like any other database, and keep both retention values documented together. Done when the local flags and the store's retention policy are recorded in the same change and reviewed by the team that owns queries.
Add the remote write block securely
Edit the Prometheus configuration to add a remote_write section that points at the store's endpoint. Include the TLS settings for the certificate authority, client certificate and client key, and load credentials from a mounted file or secret store instead of placing them inline in the main file. The security model sets TLS 1.2 as the minimum, so confirm the store supports it. Validate the file with promtool before reloading, and use a reload method your team controls, since the lifecycle endpoints are disabled by default. Done when the configuration check passes and the instance starts without errors in its logs.
Tune queue settings after reading defaults
Read the default queue settings in the Prometheus remote write documentation for the version you run before you change any value. Change one parameter at a time, and only if the remote write metrics show samples failing or queuing under your expected write volume. Keep the store's ingest limits in view, because raising concurrency beyond what the store accepts moves the bottleneck to the store. Done when the queue and failed-sample metrics stay flat under normal load and each change is recorded with the Prometheus version and the reason for it.
Configure remote read only when needed
Add a remote_read block only if dashboards or alert rules must query data older than local retention through Prometheus. Queries that span both stores depend on the store's latency, so they can slow dashboards and should be tested with real query ranges. Remote read uses the same kinds of TLS and credential settings as remote write. If your teams only need historical data, they can query the store through its own interface and you can skip this step. Done when a query for a time range older than local retention returns data through Prometheus within a response time the dashboard owners accept.
Run a canary on one instance
Enable remote write on one Prometheus instance first and leave the others unchanged. Let it run through a full cycle of your normal workload before drawing conclusions. Verify three things: samples arrive at the store, the remote write metrics show no sustained backlog, and a known series returns the same value from both Prometheus and the store at the same timestamp. Done when all three checks pass, and only then roll the same configuration to the remaining instances, one at a time.
What drives the cost and effort of remote write storage
This table lists the factors that change what remote write storage costs and how much work it takes to run, and how each one can be controlled. Read the one-time and recurring columns together to separate design and migration effort from storage, compute and upkeep that continue after go-live. Costs for a specific environment depend on sizing that a design phase should establish, not on any figure in this table.
| Driver | Effect on cost and effort | How to keep it down | One-time or recurring |
|---|---|---|---|
| Active series and label cardinality | Every unique label combination is a separate series, so more series means more ingestion, storage and query load on the remote store. | Review high-churn labels before rollout, drop unused series with relabeling, and avoid unbounded values such as request IDs in labels. | Recurring: storage and compute grow with series count |
| Scrape interval | Shorter intervals produce more samples per series, increasing write volume and storage in direct proportion to the sample rate. | Set intervals per job based on how fast the signal changes, and keep short intervals for the few targets that need them. | Recurring: scales with sample volume |
| Number of Prometheus instances sending | Each sending instance adds its own write stream. Redundant pairs that both write can double ingestion unless deduplication is designed in. | Confirm which instances write, use external labels to identify replicas, and check how the store handles duplicate samples. | One-time design, recurring ingestion cost |
| Retention length required by the business | Longer retention multiplies stored data and backup volume, and it raises the cost of every query that scans old data. | Set retention by use case, downsample older data where the store supports it, and document the retention period each team needs. | Recurring: storage grows with retention |
| Store ownership model (self-hosted or hosted service) | A self-hosted store adds cluster compute, disk, patching and on-call work. A hosted service shifts those to a provider but adds per-usage charges and data location questions. | Compare total ownership over the expected life, confirm where data resides, and get written pricing units from each provider. | One-time selection; recurring operations or service fees |
| Authentication and network path | TLS, client certificates and basic authentication add configuration and certificate rotation. Routing through firewalls or proxies adds components to operate and troubleshoot. | Use one certificate process for all senders, keep the network path short, and store credentials in managed secrets rather than in configuration files. | One-time setup; recurring certificate and secret rotation |
| Remote read and cross-store queries | Serving long-range queries from the remote store adds query load and latency, and dashboards may need changes to choose the right source. | Limit remote read to the queries that need it, keep recent queries on local storage, and test dashboard and rule performance against the remote store. | One-time dashboard migration; recurring query load |
| Compliance evidence such as retention proof | Assessors may ask for proof that data was kept for the stated period and was not altered, which requires logs, access records and tested restores. | Agree the evidence list with the compliance owner early, automate retention checks, and schedule periodic restore tests. | One-time control design; recurring evidence collection |
| Upkeep and upgrades of the store and senders | Remote write components and the store need version updates, compatibility testing and monitoring of dropped or failed writes. | Track supported versions for Prometheus and the store, test upgrades in a non-production copy, and alert on write failures. | Recurring: upkeep, on-call and upgrade testing |
How remote write changes for single node, HA, Kubernetes and air-gapped setups
A single Prometheus server is the simplest case. One process scrapes targets, writes samples to its local TSDB and, when remote write is enabled, pushes the same samples through an in-memory queue to the external store. Failure analysis is short: if the server stops, collection stops, and if the remote endpoint is slow or unreachable, the queue grows until it is drained or dropped. The main risk is that the write-ahead log and queue live on the same local disk, so losing that disk loses the samples that have not yet reached the store.
What to plan for on a single node:
- One sender, one ordering path: only one Prometheus writes each series, so duplicate handling in the store is usually not needed.
- Queue capacity: size the remote write queue so a store outage of your expected length does not drop data. Confirm the queue settings and their defaults for your Prometheus version in the official documentation.
- Local disk as a single point of failure: the TSDB and write-ahead log are not replicated, so back up and monitor the volume.
- Store acknowledgment: confirm whether the store returns errors that cause retries, and how long it accepts out-of-order samples.
- Restart behavior: check how much unsent data survives a restart and how long replay takes.
Before go-live, simulate a store outage in a test environment and confirm that the queue backlog recovers without gaps in the store.
When two Prometheus instances scrape the same targets for availability, both can send remote write for the same series. The store then receives duplicate samples with identical labels and timestamps, and it must handle them without double counting or rejecting the second writer. The duplicate-handling behavior is a property of each store, so confirm it with the vendor or project before you choose a design. The detailed pair design, external labels and alerting behavior are covered in the high availability topic, so this tab only covers the storage consequences.
Decisions the storage side forces:
- Who deduplicates: decide whether the store merges replica series, whether one replica is dropped at ingestion, or whether queries deduplicate at read time.
- Replica identity: confirm that each instance sets a replica label so the store can tell writers apart when it needs to.
- Out-of-order rejection: a replica that falls slightly behind may send samples the store rejects as out of order. Confirm how the store reports these and whether it counts them as errors.
- Query behavior: check whether dashboards and alerts see duplicate or merged results, and whether the reader knows which view it is using.
- Cost impact: duplicate writes roughly double ingestion volume unless the store or the sender filters them.
Request the store's duplicate-handling documentation and test a pair against it before relying on it for retention.
In Kubernetes, the Prometheus configuration, remote write endpoint and credentials are usually delivered as objects rather than files on a host. The remote write section lives in the configuration, while the endpoint URL, basic credentials and TLS material are better kept in Secrets and mounted into the pod. How these are delivered depends on whether you use the Prometheus Operator, a Helm chart or plain manifests, so confirm which mechanism your platform team has standardized on.
Points to settle for a cluster deployment:
- Configuration delivery: choose between a ConfigMap, a Secret or an operator resource, and confirm how changes trigger a reload in your Prometheus version.
- Secret mounting: mount client certificates and tokens as files with restricted permissions, and avoid putting credentials in plain values in the manifest.
- Certificate rotation: plan how renewed certificates reach the pods and whether a restart is required.
- Pod restarts and storage: the TSDB and queue need persistent volumes, so confirm the storage class and what happens when a pod is rescheduled.
- Network policy: allow egress only to the store endpoint, and confirm the store can reach the cluster if it pulls data.
Before rollout, check that a Secret rotation reaches the running pod and that the remote write queue resumes after a restart.
In an air-gapped network, the store must sit inside the isolated environment with the Prometheus servers. There is no external endpoint to reach, so every component, including the store, its dependencies and its certificate authority, must be installed from inside the network. Remote write itself needs no internet access, but the store must be deployed and kept patched through approved internal channels. Certificates must be issued by an internal authority that both sides trust, and the trust chain must be tested before go-live.
What changes in an isolated network:
- Store placement: deploy the store in the same isolated zone or a tightly controlled internal zone, and confirm the path from Prometheus to it.
- Internal certificates: issue server and client certificates from the internal CA, and confirm the minimum TLS version the store and Prometheus both support.
- No external endpoints: remove any default endpoint, telemetry or update URL from the configuration and confirm the store does not call out.
- Offline updates: plan how Prometheus and store images, packages and signatures reach the network and how they are verified.
- Duplicate handling and retention: confirm the store's behavior with the vendor, since offline support and documentation may differ.
At acceptance, test certificate expiry handling and confirm from the network side that no unexpected outbound connections occur.
How to secure the remote write path and its credentials
Remote write sends every sample off the Prometheus host, so the transport, the credentials that authorize writes, and the people who can read the stored data all become part of the security design. The cards below cover the controls to decide on before the first sample leaves the network.
Encrypt the remote write endpoint with TLS
Every remote write request carries metric samples and labels across the network to the external store, so the endpoint should only be reached over TLS. Confirm that the store's endpoint presents a certificate that Prometheus can validate against the CA you trust, and check the minimum TLS version the store accepts. Prometheus supports TLS 1.2 as the minimum, so the store should be configured to match or exceed that level.
Use client certificates where the store supports them
Client certificate authentication lets the store verify that a write comes from a specific Prometheus instance, which is stronger than a shared secret that can be copied from a configuration file. Prometheus supports client certificates, but the receiving store must also be able to validate them. Confirm this with the store's vendor, and plan how certificates are issued, rotated and revoked for each Prometheus server.
Allow basic authentication only over TLS
HTTP basic authentication is a practical option when the store does not support client certificates, but without TLS it sends credentials in cleartext on the wire. Prometheus stores basic authentication passwords as bcrypt hashes on its side, which protects the password at rest on the Prometheus host, not in transit. Treat basic authentication as acceptable only when the remote write URL is HTTPS and certificate validation is enabled.
Protect credentials in the remote write configuration
The remote write section of the Prometheus configuration file can hold passwords, bearer tokens, and certificate key paths, so that file becomes a credential store. Restrict read access on the file to the Prometheus service account, keep it out of shared repositories, and inject secrets at deploy time from a secret manager where your platform allows it. Plan how credentials are rotated, and confirm that rotation does not require a configuration change that exposes the old value in logs or version history.
Keep admin and lifecycle APIs disabled
Prometheus disables its admin HTTP APIs by default. The enable-admin-api flag turns on destructive operations such as deleting series, and the enable-lifecycle flag turns on reload and shutdown endpoints. Leave both off in production unless a documented runbook needs them, and if they are enabled, limit who can reach the web listener. Check the running process arguments after each change to confirm the flags are not set.
Restrict who can read stored metrics
Retained metrics often include sensitive labels such as hostnames, tenant identifiers, customer names or internal service paths, and the external store may keep them for much longer than local retention. Define which roles can query the store, separate read access from write access, and decide how long data and access logs are kept. Put these retention and access policies into the design before the first sample is sent, because changing them later means reviewing data that has already been written.
How to verify that remote write works and fails safely
A remote write setup is only finished when you have shown, with tests you can repeat, that samples reach the external store intact, that Prometheus keeps working when the store does not, and that the data it holds matches the retention you promised. Treat these checks as acceptance criteria for the project. Agree the pass conditions in writing before the cutover, record the results, and rerun the same tests after every Prometheus upgrade or change to the remote write configuration. The Prometheus storage documentation is clear that local storage is single-node and not replicated, so the local TSDB is your fallback during a store outage, and the tests below are designed to prove that fallback works. Start with delivery. Pick a set of known series, such as a synthetic counter emitted by a test exporter and a handful of production series with stable labels. Over a window of several hours, query the same series from the local Prometheus and from the long-term store, aligned on timestamps, and compare the values. Expect small gaps only where you expected them, for example during a planned restart. Check the remote write metrics exposed by Prometheus itself, including the counters for samples sent and failed, and the queue lag, so that you can see delivery state without relying on a dashboard alone. Confirm the agreed tolerance with the people who own the store and write the number down before you run the test. Next, simulate a store outage. Block network access to the remote endpoint or stop the receiving service, then confirm that Prometheus keeps scraping, keeps evaluating rules, and keeps serving local queries. Watch the write queue grow and the failure counters rise. Then restore the store and confirm that the backlog drains and that the samples written during the outage appear in the store with their original timestamps. Ask the store operator how long they can absorb a backlog and what happens when the remote write endpoint rejects samples as too old or out of order. A failed test here means Prometheus stalled, dropped data without a counter increment, or failed to drain the queue once the store returned. Each of these should stop the cutover and trigger a review of queue settings and the store's ingestion limits. Then test restarts. Restart Prometheus while a backlog is queued, and confirm from the write-ahead log replay messages in the logs that the restart completes and that the queued samples are delivered afterward. Compare the local series count and the store series count for the same window. The test passes when no gap appears that is not explained by the scrape interval and the documented restart behavior. Also verify retention on the store against the written policy: query for data older than the stated limit, confirm it is removed on schedule, and confirm that the local retention setting, which defaults to 15 days unless changed, is not confused with the longer store retention. Record which team owns each retention setting. Finally, cover the rules and queries that depend on the data. Alerts that run on local data must still fire when the store is down, so trigger a known test condition and confirm the alert reaches Alertmanager and the on-call receiver. If remote read is part of the design, run queries that span the local and remote periods and confirm the results are consistent at the boundary, and check the response time against the limits your users need. Confirm with your vendor or the store documentation which PromQL functions are safe to run across remote data, because behavior at the edges varies by store. Keep the results in a signed test record, since that record is what you will hand to auditors or an internal change board.
What to monitor, back up, upgrade and document after go-live
Remote write moves long-term storage out of the Prometheus server, but the server still holds the write-ahead log and the queue that feeds the store. The cards below cover what to watch, what to back up, and how to keep the pipeline current on the release cycle without losing samples.
Backlog and failed-send alerts
Prometheus exposes remote write metrics for samples waiting in the queue, samples that failed, and retries. Alert on a backlog that keeps growing over a sustained window and on a rising failed-send rate, rather than on brief spikes during a store restart. Route these alerts to the team that owns the store as well as the team that owns Prometheus, since either side can cause the failure. Confirm exact metric names against your Prometheus version before writing alert rules, and test the alerts by stopping the store in a non-production environment.
External store disk and ingestion
Monitor the external store's disk growth, ingestion rate and query latency as separate signals. Disk usage shows when retention will force deletion, and ingestion rate shows whether the store keeps up with the series volume Prometheus sends. Ask your store vendor or project which metrics expose disk headroom and write throughput, because naming and availability differ between systems. Set alerts with enough lead time to plan a capacity change, and review them whenever series volume changes.
Backups for store and configuration
Local Prometheus data is not replicated, so treat the TSDB on each server as a single-node database and decide whether its loss is acceptable. The long-term store needs its own backup and restore test, separate from any Prometheus snapshot. Keep the Prometheus configuration, alerting rules, recording rules and references to secrets in version control and in a separate location. Restore the store and configuration into a test environment on a regular schedule so the procedure is proven before an incident.
Upgrade timing on release and LTS lines
A new Prometheus minor release cycle begins every 6 weeks, and a minor release generally stops receiving bug fixes after those 6 weeks. Long-term support releases receive bug fixes for one year. The current LTS line is 3.13, supported until July 31, 2027, while LTS 3.5 reached end of life on July 31, 2026. Choose the line that matches your patching capacity, and schedule upgrades before the line you run leaves support.
Retesting remote write after upgrades
A Prometheus upgrade can change remote write behavior, metric names or defaults, so repeat the write path test after every upgrade, including LTS patch releases. Confirm that samples arrive in the store with the expected labels, that backlog and failure metrics still read correctly, and that alerting and recording rules evaluate. Keep the test scripted so it can run against staging with production-like series volume. Record the results with the change ticket so the next upgrade starts from a known baseline.
Retention, cardinality and outage runbook
Document the retention policy for local storage and for the external store, including who approves changes. Local retention defaults to 15 days and can be set by time or size, so state which limit applies. Set cardinality budgets per team or job, because remote write volume and store load grow with the number of series. The outage runbook should describe what Prometheus does while the store is down, how to read the queue, and when to pause, drop or replay samples. Review these documents after each significant change.
What remote write storage costs and what the work needs from our team
What drives the cost of Prometheus remote write storage?
The main cost drivers are the number of active series, how long you keep samples, who owns the external store, and how much cross-store querying you need. Prometheus itself is licensed under Apache-2.0 with no paid edition, so the spend comes from infrastructure, storage, implementation and the people who run it. Series count and scrape interval set the sample volume that every remote write payload carries. Retention multiplies the stored volume. Store ownership decides whether you pay a vendor, run a self-hosted store in your own accounts, or both. Cross-store queries add work for the query layer and for anyone who builds dashboards against two sources. Ask any partner to show how each driver was measured in your environment before accepting a number.
How long does a remote write storage project take?
The duration depends on scope, not on a fixed template. A single Prometheus server sending to one store with a short metrics inventory moves faster than a fleet of federated Prometheus servers that need replicated sources, relabeling and new dashboards. The longest steps are usually the inventory, the sizing test against real sample volumes, and the failure-mode testing of the store and the queue behavior during outages. Each engagement is fixed-scope, so the written scope sets the timeline. Ask for the phases, the dependencies on your teams and the test evidence each phase produces before signing.
What does our team need to provide before work starts?
Your team needs to provide four things. First, a metrics inventory listing jobs, targets, approximate series counts and which metrics matter for alerting and capacity. Second, access to the Prometheus hosts and the long-term store, including accounts, network paths and the ability to change remote write settings in a test environment. Third, retention decisions covering how long raw samples, downsampled data and each team's data must be kept, which often depends on policy or audit needs. Fourth, named owners for on-call, who will receive alerts about the pipeline, the store and the query layer after go-live. Missing any of these usually extends the schedule.
Can our internal team set up remote write storage alone?
Yes, if the team has operations skills for both Prometheus and the chosen store, and time to test failure modes before relying on the pipeline. Prometheus local storage is not clustered or replicated and is managed like any single-node database, so the team must understand WAL behavior, retention limits and what happens when the remote endpoint is slow or unavailable. Most of the risk sits in testing: outage replay, backpressure, restarts and query results across local and remote data. If your team cannot allocate those test cycles, a partner can supply them under a fixed scope.
What does a fixed-scope quote for this work include?
A useful fixed-scope quote lists the systems in scope (Prometheus servers, store, query path, Alertmanager if affected), the metrics inventory and sizing method, the retention policy to be implemented, and the security controls for remote write and read endpoints, including TLS and authentication. It should name the deliverables such as configuration, runbooks, dashboards and test reports, the test scenarios covering outages and restarts, the acceptance criteria, and the client responsibilities from the previous question. It should also separate one-time implementation fees from any recurring retainer for upkeep, and state what is excluded, such as storage capacity purchases or new application instrumentation.
Does remote read make queries slower or less accurate?
It can, and the trade-off should be decided before rollout. Remote read lets Prometheus query the external store, but queries that span long ranges depend on the store's performance, network latency and the amount of data returned. Results can also differ at the boundary between local retention and remote data if the two sources are not aligned. The Prometheus documentation says the system values reliability over precision, so teams with strict accuracy needs should test the query paths they depend on. Ask for query benchmarks against representative dashboards and alert expressions as part of the scope.
What should we confirm about the store before choosing a design?
Confirm the store's supported remote write and read protocol versions, its write throughput at your sample volume, its retention and downsampling options, its authentication and encryption support, and where the data physically resides. Confirm how the store behaves during outages and whether it can accept the backlog Prometheus queues. Ask who operates it, who patches it and how it is backed up. These answers come from the store's own documentation and your environment, so request written confirmation from the vendor or project before the design is fixed.
How DeployOpen scopes, builds and hands over remote write storage
DeployOpen plans and implements Prometheus long-term storage with remote write inside your own infrastructure, so samples leave local storage for an external store you choose. Each engagement starts from a written scope covering retention, sizing, query behavior and the operating model, and ends with documented, tested handover to your team.
How we scope
- Inventory the Prometheus servers, targets and metrics that need retention beyond the local default of 15 days
- Agree in writing which external long-term store receives remote write and who operates it
- Define retention periods, storage sizing inputs and the queries that must work through remote read
- Record the security requirements for remote write endpoints, including TLS and authentication settings
- Confirm the fixed-scope deliverables and timeline before any build work starts
How we build
- Configure remote write on each Prometheus server with queue settings sized to the agreed sample rate
- Enable TLS and client certificate or basic authentication on the remote write path, avoiding cleartext credentials
- Test sample delivery, dropped-sample alerts and recovery after the external store is unavailable
- Validate that dashboards and recording rules return the same results from the external store as expected
- Keep admin and lifecycle APIs disabled unless the scope explicitly requires them
How we hand over
- Deliver a written runbook covering remote write configuration, retention policy and storage capacity checks
- Document the query trade-offs of remote read, including latency and the data available from each source
- Walk your operators through alerts that signal failed or lagging remote write delivery
- Provide the configuration, test results and change records needed for your internal review
- Agree which owner maintains the local TSDB, the remote store and the Prometheus versions in production
How we support
- Offer ongoing support, upgrades and testing as a retainer after go-live, if your team wants it
- Re-test remote write and retention behavior when Prometheus releases are upgraded
- Review storage growth and retention settings with your team on an agreed schedule
- Ask your quote to separate one-time implementation fees from recurring retainer and infrastructure costs
- Confirm with your storage vendor the supported retention, compaction and access options before committing

