When Prometheus agent mode fits edge and multi-cluster collection
Prometheus agent mode is a scrape-and-forward configuration. It discovers targets, scrapes them, and sends the samples to central storage through remote write, without keeping a queryable local database. It fits when metrics from many sites, clusters or edge locations must land in one place, but running a full monitoring stack at every location is too heavy to operate well. Typical cases include small hardware at plants or retail sites, fleets of Kubernetes clusters run by separate teams, and locations with intermittent links to the central network.
Local storage is the constraint that shapes the design. The Prometheus storage documentation states that local storage is not clustered or replicated and should be managed like any other single-node database. A full server at each site therefore means every site carries its own data, retention, backups and upgrade cycle, and each site's history depends on one node. Agent mode moves long-term retention and cross-site querying to the central system, which makes that store the component you must size, secure and make highly available. Confirm with your storage vendor how it handles many concurrent remote write senders.
Decide before you build whether your teams can accept what you give up. Agent instances do not serve local PromQL queries, evaluate rules against their own data for dashboards, or raise alerts on their own, so a site that loses its link to central storage loses visibility until the link returns unless you plan a buffer. Ask who receives alerts when a site is isolated, whether on-site staff need dashboards during an outage, and which rules must keep working without the uplink. If a condition must alert even when the uplink is down, it cannot depend on an agent instance alone. Write those answers down before you choose the architecture.
Check the fit against the project's own guidance. The Prometheus overview says the project values reliability over precision and is not a good choice where 100% accuracy is required, such as per-request billing. Agent mode does not change that, so any collection path that needs exact counts should be designed separately. Also confirm the release line you will run. A minor release receives bug fixes for 6 weeks, while long-term support releases receive bug fixes for one year. Agree which line the agent fleet and the central server will follow, and who upgrades each tier and when.
Which deployment pattern fits each site
This table compares three ways to place Prometheus across sites: a full server at each site, an agent at each site that forwards samples to central storage, and one central server that scrapes across the network. Read each column against your outage tolerance, edge query needs and link capacity. The main takeaway is that agent mode removes local rules and queries, so the central storage choice and the link design become the foundation of the deployment.
| Decision factor | Full server per site | Agent per site with central storage | Central server scraping across network |
|---|---|---|---|
| Local outage tolerance | Best when sites must keep collecting and alerting during a WAN outage | Collection can continue through short link loss if buffering holds; confirm limits in testing | Weakest, since a network cut stops scraping and the site has no local copy |
| Query needs at the edge | Local PromQL and dashboards keep working without the central link | No local queries; on-site dashboards depend on central reads, so confirm latency | All queries run centrally, so edge users depend on the network path |
| Bandwidth between site and center | Low; mainly dashboards, federation or any long-term copy you configure | Moderate to high; the full sample stream crosses the link, so size it for that | High; every scrape crosses the network, so target count and interval drive traffic |
| Number of sites | Suits a few sites that each need full, independent operation | Suits many sites with smaller or uniform footprints feeding one store | Suits a small number of networks with reliable reach to every target |
| What changes for alerting | Rules and Alertmanager run locally, so each site can alert on its own | No local rule evaluation; alerting moves to the central side, so plan for that | Rules run centrally; a network cut can silence targets without a clear signal |
| What the team operates | A server, local storage, retention and upgrades at every site | Agents at each site, plus one central store, its write endpoint and retention | One central server, but scrape reach, firewall rules and credentials across all networks |
| Connection and central storage | Remote write is optional, for long-term copies only | Remote write is required; remote read serves queries back. Choose and confirm the storage system | Central server uses local storage, which is single-node and not replicated, so plan backups |
How to size agent mode against a full server per site
Size agent mode from measurements of your own estate, because the right figures depend on what each site scrapes and how often. The main driver is active series per target, since every exposed series adds samples on each scrape. Multiply that by the number of targets each agent will carry and by the scrape interval, and you get the sample rate the agent must handle. Remote write throughput comes next. The agent batches samples, sends them through its write shards, and queues what it cannot deliver yet. If the central endpoint slows, that queue grows. Relabeling that drops unneeded series at the agent reduces every downstream figure, so review it before you size hardware. Treat these values as inputs you revisit as the estate changes.
The edge host carries the write-ahead log and the in-memory head that holds recent series, so its memory and disk needs follow the same series and sample figures. Confirm in the agent mode documentation how the WAL is truncated once remote write accepts samples, because that behavior decides how much disk a long outage consumes. Bandwidth and latency to central storage matter in two ways. They set how quickly the queue drains after an outage, and they set how much spare capacity a site needs for catch-up traffic. A link that handles steady state can still fall behind after an extended disconnection. Test that recovery path directly by cutting the link in a lab that mirrors one site and watching queue and disk behavior.
A full server per site keeps its own local time series database, and the storage documentation sets default local retention at 15 days, configurable by time or size. That local history helps during a central outage, but it also means each site needs disk sized for retention, not only for the write path. The storage docs state that local storage is not clustered or replicated and should be managed like any single-node database, so each server needs its own backups, upgrades, and disk alerting. Multiply that by the number of sites and the operational load grows with them. Agent mode moves long-lived data to one central store, which concentrates sizing and failure handling in one place, although that store then carries the combined write rate of every site.
Collect the inputs before you compare options. For each job, record the active series count, scrape interval, and sample rate from the Prometheus server at each site, along with the number of targets an agent would carry. Measure write rates on the central side during peak periods, and document the outage window the business expects the edge to tolerate. Ask any vendor or implementation partner to size from those measurements and to state its assumptions in writing, including queue settings, disk, memory, and bandwidth. A proposal that cannot show how it used your numbers is a reason to ask more questions before you sign.
Setting up agent mode and remote write in order
This sequence takes each edge node or cluster from release selection to a verified central feed. Each step names the owner and the check that shows it is finished, so site collectors are trusted before alerting responsibility moves to the central side.
Choose the release line and confirm agent mode
The platform or SRE team picks the Prometheus line that will run on every collector. The current long-term support line is 3.13, which receives bug fixes until July 31, 2027. Minor releases generally stop receiving bug fixes six weeks after they ship, so fleets that cannot be upgraded often should sit on a long-term support line. Read the release notes for the chosen line to confirm agent mode is supported, then start a lab collector with the documented settings. Done when the test collector runs in agent mode and exposes its own health metrics to a scratch endpoint.
Define central storage target and authentication
The platform team and the owner of the central storage system agree on the receiving endpoint, tenancy model and the labels every sample must carry. Create a dedicated identity for each site so access can be revoked without affecting the others. Prefer client certificates. If HTTP basic authentication is used, the password should be bcrypt-hashed on the receiving side and sent only over TLS. Confirm the authentication method against the receiving system's documentation. Done when a test write with a site credential succeeds and an unauthenticated write is rejected.
Write scrape configuration and discovery per site
Each site's team writes its scrape configuration using static targets or the service discovery that fits its platform, such as Kubernetes or file-based discovery. Set external labels in the global configuration so every series carries a site identifier. Keep relabeling rules identical across sites so label names and values match in the central store, and review label cardinality before the first run. Done when the targets page on each collector shows the expected target count and a sample series carries the correct site label.
Configure remote write with TLS and credentials
The security owner and the site team configure the remote write block so the endpoint uses HTTPS with server certificate verification enabled. Set the minimum TLS version to 1.2 or higher, and load client certificates or passwords from mounted secret files instead of inline configuration. Leave queue settings at their defaults until the baseline test is complete. Done when the endpoint configuration shows TLS enforced, a plain HTTP URL is rejected by the review checklist or validation step, and a packet capture confirms the credentials are not sent in cleartext.
Confirm samples arrive centrally
The team responsible for the central store queries it for series carrying each site label and compares the results with the targets each collector reports. Check that timestamps are close to current time and that the number of active series per site matches the expected scope. Investigate any site that sends nothing or sends a partial set before moving on. Done when every site shows samples in the central store, and the team records the query used for each site so the check can be repeated after upgrades.
Test outage handling and recovery
The operations team cuts the connection between one test collector and the central store for a planned window. Confirm how the collector buffers samples and what it discards according to the documentation for the chosen release, and record the agreed loss tolerance before the test starts. Restore the connection and check whether buffered samples arrive and whether any gap falls outside that tolerance. Done when the test result is written down, including the observed behavior, and the team agrees it matches the design.
Place alerting and rules deliberately
Because local rule evaluation and local alerting are not part of the collector role, the rules and the alerting path move to the central side. The alerting owner runs Alertmanager centrally and, if it runs as a cluster, configures Prometheus to reach every Alertmanager instance directly, which the Alertmanager documentation recommends over placing a load balancer in front. Define a site-down alert based on missing data from each site. Done when a test alert from a central rule reaches the receiver and the site-down alert fires during the outage test.
Security controls for agents and the central endpoint
Agent mode moves collection to many sites, and each agent and the central endpoint it writes to become places where access must be controlled. The cards below cover the authentication options Prometheus provides, the admin and exporter surfaces that should stay closed, and the gaps that a reverse proxy has to fill. Review each one for every agent and for the endpoint that receives remote write.
TLS and client certificates
Prometheus supports TLS for its HTTP endpoints and client certificate authentication, with TLS 1.2 as the minimum version. For agent mode, decide which side presents a certificate, how certificates are issued and rotated for each site, and how a revoked site is cut off. Confirm the minimum TLS version and certificate handling in the web configuration file for your exact release before you rely on it.
Basic authentication with bcrypt
HTTP basic authentication can protect the agent and central endpoints using passwords stored as bcrypt hashes, so plain passwords never appear in the configuration. Basic authentication without TLS sends credentials in cleartext, so it must never run on a plain HTTP listener. Check that every scrape and remote write job that uses it also uses TLS, including jobs that pass through a network you do not fully control.
Admin APIs kept disabled
Prometheus disables its admin HTTP APIs by default. The admin API flag enables destructive operations such as deleting series, and the lifecycle flag enables reload and shutdown endpoints. Keep both off on agents and on the central endpoint unless a specific runbook needs them, and restrict them to a management network if they must be on. Verify the effective flags on each running process, since a copied startup script can re-enable them.
Exporter URL parameter exposure
Exporters such as SNMP and Blackbox take the target to probe from a URL parameter. Anyone who can reach these exporters over HTTP can make them send requests to arbitrary hosts, which can expose internal networks. Bind exporters to localhost or a private interface, allow access only from the Prometheus agent that scrapes them, and restrict which targets each exporter may reach where the exporter supports it.
No built-in single sign-on
Prometheus has no built-in single sign-on for its web UI or API. Operators normally place an authenticating reverse proxy in front of the UI and API and let it handle user identity, while the agent and remote write paths keep their own certificate or basic authentication. Decide which identity provider the proxy uses, how sessions expire, and which paths the proxy must not expose.
Reverse proxy gaps to verify
The Prometheus security model notes there is no built-in CSRF protection on endpoints that change state. A reverse proxy in front of the UI should therefore block or tightly control mutating requests, and should pass only the paths users need. Test the proxy against the admin and lifecycle endpoints directly, because a path rule that misses one of them undoes the other controls.
Operating many agents after launch
Running agents across many sites is recurring work, and most of it lands on the central team. Each agent's health has to be visible centrally: whether it is up, whether its scrapes succeed, and whether remote write is failing or falling behind. An agent keeps no query history for operators to fall back on, so a silent remote write failure appears only as a gap in central data. Before go-live, confirm how long each agent buffers samples during a central outage, how the team is alerted when buffering starts, and who checks the remote write failure signals each day.
Keeping agents on supported release lines is the second recurring task. The Prometheus project starts a new minor release cycle every six weeks, and after those six weeks a minor release generally no longer receives bug fixes. Long-term support lines receive fixes for one year. The current LTS line is 3.13, supported until July 31, 2027. Ask your implementation partner or internal team which release line each site runs, which line the fleet will standardize on, and who decides when a site moves. A fleet that spans several minor versions means the team supports several combinations at once.
Rolling upgrades across many sites should move in small waves, starting with one canary site. Before each later wave starts, check the central health and remote write signals for the canary and for the sites already upgraded. Site-level failure alerting must be defined centrally, because an agent does not evaluate alerting rules and cannot report its own absence. Central rules should fire when a site's expected series stop arriving, and each site should be mapped to an owner. Test those rules by deliberately stopping one agent in a non-production site and confirming that the alert reaches the right people.
If the central team keeps alerting, run Alertmanager as a cluster using its cluster flags, and list every Alertmanager instance in the Prometheus alerting configuration. Do not place a load balancer in front of the instances. Prometheus needs to send alerts to each instance so that the cluster can deduplicate and gossip state between them. In the operations plan, ask for the following: the agent health and remote write alerts with their owners, the release line and upgrade schedule for each site, the canary and rollback steps for upgrades, the Alertmanager cluster layout and its test procedure, and the on-call hours and escalation path for each site.
Limits to confirm before you commit to agent mode
Agent mode trades local capability for a lighter footprint at each site, and every trade-off below needs checking in your own environment before rollout. The cards cover accuracy, storage, lost features, release support, central capacity and outages. Confirm each one against the release you plan to run.
Reliability over precision
The Prometheus documentation says the project values reliability over precision and is not a good choice where 100% accuracy is required, such as per-request billing. Agent mode does not change that trade-off, because samples still pass through scraping, a local write-ahead log and a remote write queue. List which dashboards, alerts and reports feed billing, audit or contractual decisions, and confirm with your architects whether those metrics need a different system.
Local storage is not clustered
Local storage on each agent is a single-node database. The Prometheus storage documentation states that it is not clustered or replicated and should be managed like any other single-node database. If a collector's disk fails before its samples reach central storage, the unsent data can be lost. Confirm how each site will be backed up or rebuilt, who watches disk usage on the collectors, and how the write-ahead log behaves after a host restart.
Local query and rule loss
Agent mode removes local query and rule capability. Each site can scrape and forward samples, but PromQL queries, recording rules and alerting rules must run on the central side. Plan where operators will investigate a site during a wide-area outage, and confirm which alerts for that site will fire and from where. Check whether any current alerting depends on rules evaluated at the edge.
Release line and support window
The current LTS line is 3.13, released July 1, 2026 and supported until July 31, 2027. The latest release on GitHub at the time of writing is v3.15.0, published September 25, 2026. Minor releases generally stop receiving bug fixes six weeks after release, so a non-LTS choice requires a steady upgrade cadence. Confirm agent mode behavior, flags and known issues in the release notes for the exact version you deploy.
Central store write throughput
Every site sends samples through remote write, so the central store must absorb the combined rate from all collectors, plus bursts after a reconnect. Ask the operator of your long-term store for its ingestion limits, any per-tenant or per-series quotas, and how it responds when writes slow down. Measure the sample rate of each collector before sizing the central side, because the required capacity depends on your series count and scrape interval.
Gaps during network partitions
When the link between a site and the central store breaks, the collector keeps scraping and buffers samples until the connection returns. That buffer is bounded by local disk space and retention settings, so a long partition can leave gaps in central data. Test a partition in a staging environment, compare what the central store shows afterward, and decide which alerts must not depend on data that may arrive late.
Questions buyers ask before engaging a partner on agent mode
Does Prometheus agent mode cost money to license?
No. Prometheus is licensed under Apache-2.0 according to its GitHub repository, and there is no paid enterprise edition of the project itself. Your costs come from the infrastructure that runs the agents and the central server, the storage that holds samples, the network that carries remote write traffic, the implementation work and the people who operate the estate. Those are the components to budget. Ask any partner to separate one-time design and build effort from recurring infrastructure and operations so you can compare options on the same basis.
What drives the cost of an agent mode rollout?
The main drivers are the number of sites or clusters, the volume of active series each one scrapes, the size and retention of the central long-term storage, and the bandwidth between each edge and the center. Engineering time is often the largest single item. Designing and testing failure behavior, such as what happens to buffered samples when a WAN link drops or the central endpoint is unavailable, takes deliberate effort. Ask for a quote that states the site count, expected series range per site, storage retention target and the failure scenarios that will be tested.
How long does an agent mode engagement take?
Duration depends on how many sites or clusters are in scope, how standardized their configuration already is, whether central storage exists, and how many failure scenarios must be verified before handover. Your own environment and the number of approvals in your change process will also set the pace. No general timeline applies, so ask for a schedule derived from the written scope, with milestones for the pilot site, the rollout waves and the failure testing. Engagements at DeployOpen are fixed-scope, so the agreed scope and timeline are set in writing before work starts.
What does DeployOpen need from our team to start?
You will need to provide a short inventory of the sites or clusters to be covered, the target discovery methods they use, the central storage or remote write endpoint you plan to use, and the network paths between edges and the center. Access to the environments where agents will run, a named contact for change approvals and a decision on retention and alerting ownership are also needed. Because agent mode does not evaluate rules or send alerts, decide early where alerting will run, and make sure the people who will receive those alerts are part of the design.
Can our own team run agent mode without a partner?
Yes, if you have people who are comfortable with Prometheus configuration, remote write tuning, storage operations and failure testing across unreliable links. Many teams start in-house for a single pilot site and bring in help for the central tier or the failure design. The honest test is whether someone on your team can explain what happens to samples during an outage, how the central store is backed up and how the configuration is rolled out to every site. If those answers are not yet clear, a scoped partner engagement can close the gap while your team keeps ownership.
Is agent mode a good fit when we need exact per-request counts?
Probably not on its own. The Prometheus documentation says the project values reliability over precision and is not a good choice where 100% accuracy is required, such as per-request billing. Agent mode also gives up local querying and alerting at the edge, so anything that depends on immediate local evaluation needs a different design. Review the accuracy requirements of each use case before committing, and confirm with the Prometheus documentation whether your intended metrics fit the model.
What does a fixed-scope engagement include for agent mode?
A fixed-scope engagement defines the sites, configuration deliverables, integration points, testing and handover in writing before work begins. DeployOpen plans, implements, documents and hardens open-source platforms inside your own cloud accounts, data centers or air-gapped networks, so the collectors and central components run where you control them. Scope, deliverables and timelines are agreed in writing first, and a quote follows from that written scope. Ongoing support and upgrades can be arranged as a separate retainer.
How DeployOpen scopes and delivers agent mode deployments
DeployOpen plans, implements, hardens and documents Prometheus agent mode in infrastructure you control, including edge sites and clusters you run. Because Prometheus is Apache-2.0 licensed, the cost sits in infrastructure, storage, implementation and the people who run it. Each engagement starts with a written scope agreed before work begins.
How we scope
- Written scope that names each site type, cluster and remote write destination in the agent topology
- Written list of what agents give up, including local queries, rule evaluation and alerting, and where each job moves
- Fixed-scope engagement with deliverables and timeline agreed in writing before work starts; ongoing support available as a retainer
How we build
- Agent configuration per site with scrape targets from service discovery or static configuration, and relabeling to keep labels consistent
- Remote write pipeline to central storage, with tested retry behavior, label policy and a sizing record for each agent
- Access controls: TLS with client certificates, and an authenticating reverse proxy in front of any exposed web UI or API
How we hand over
- Runbook covering agent restarts, remote write backlog checks, configuration reload and upgrade steps
- Documented sizing, retention and central storage settings, with the reasoning behind each choice
- Test evidence for scrape coverage, central-side alert routing, and confirmation that admin APIs stay disabled unless approved
How we support
- Ongoing support available as a retainer for agent upgrades, configuration changes and fleet changes
- Upgrade planning across release lines, because minor releases generally stop receiving bug fixes six weeks after release
- Review of exporter exposure, since exporters such as SNMP and Blackbox take targets from URL parameters
