Prometheus high availability guide

How Prometheus high availability works when local storage is single-node

Start from the situation this design has to survive: a host fails, a patch window takes a server offline, or a disk is lost, and the monitoring stack must keep evaluating rules and sending alerts anyway. Prometheus does not solve this with a clustered or replicated database. The storage documentation states that local storage is not clustered or replicated and should be managed like any other single-node database. Availability therefore comes from three things working together: two or more Prometheus servers that scrape the same targets, a clustered Alertmanager, and remote write to an external long-term store for queries that need to span both replicas. Each piece has its own failure modes, and the design is only as strong as the weakest of them.

A pair of identical Prometheus servers sounds simple, but it creates problems that a single server never has. Both servers scrape the same targets, so every series exists twice, and queries that do not account for the duplicate will double-count or show jitter when one replica restarts. Each replica also evaluates the same alerting rules independently, which means that without coordination the same alert can fire from both servers. Deciding how to label each replica, which replica is authoritative for dashboards, and how duplicate series are removed at query time are design decisions that must be made before the first server is built, not after an incident exposes them.

Alerts have a stricter requirement than dashboards. An alert that is never sent is a missed incident, and an alert that is sent twice erodes trust in the paging system. Alertmanager handles grouping, inhibition and silences, and it can run as a cluster through its --cluster flags so that instances share notification state. Prometheus should be configured to send alerts to every Alertmanager instance directly rather than through a load balancer. Confirm the exact cluster settings and deduplication behavior against the Alertmanager documentation for the version you deploy, then test a failover on purpose: stop one Alertmanager and one Prometheus replica and verify that notifications still arrive exactly once to each receiver.

Prometheus values reliability over precision, and that choice shapes what high availability can promise. The project documentation says it is not a good choice where 100% accuracy is required, such as per-request billing. A replica that was down for part of a window will have gaps, and a replacement server does not backfill missing samples from its peer. This is acceptable for operational alerting and capacity dashboards, but it means that auditable or billing-grade measurements need a different design. Write this boundary into the requirements document so that stakeholders understand which views are authoritative and which are best effort.

Before building, ask the team to confirm five points: the retention each replica keeps locally, the remote write target and how long it retains data, how queries will span replicas and deduplicate series, which Alertmanager version and cluster settings will be used, and who runs the failover test and how often it is repeated. Each answer changes the storage sizing, the operating procedure and the cost of the arrangement, so they belong in the written scope before any server is provisioned.

Which Prometheus components need redundancy and why

A single Prometheus server keeps its own local data and sends alerts from one process, so each component in the monitoring path can silence visibility or alerting on its own. The cards below take each component in turn, state what fails when it runs on one node, and name what the design needs to add.

Prometheus server and local TSDB

Each Prometheus server writes samples to its own local time series database in two-hour blocks, protected by a write-ahead log. Local storage is not clustered or replicated, so a single server that crashes or loses its disk leaves a gap in history and no metrics during the outage. A high availability design runs a redundant pair of servers scraping the same targets, and you should confirm how each server's disk, WAL recovery and restart behavior fit your recovery objectives.

Alertmanager cluster

Alertmanager receives alerts from Prometheus and handles grouping, inhibition, silences and notification to receivers. A single instance means that when it is down, alerts are not delivered at all, even though the rules still evaluate. Run multiple Alertmanager instances joined with its cluster flags, and point Prometheus at every instance rather than at a load balancer, so each instance receives the alerts and deduplication happens inside the cluster.

Remote write receiver or long-term store

The default local retention is 15 days, so anything older is only available if it was sent elsewhere. Remote write sends samples to an external store that keeps history beyond that window and gives queries one place to read from when a replica is down. A design that writes from two replicas must also define how duplicate series are deduplicated, and you should confirm the store's durability, write throughput and retention settings against your own requirements.

Query layer across replicas

Two replicas that scrape the same targets produce overlapping series, and users need one consistent view. A query layer, such as a federation or deduplicating front end, decides which replica answers a query and merges results when one replica has a gap. Without it, dashboards and rule evaluations can show different values depending on which server responds, so confirm how your chosen tool handles replica labels and staleness.

Scrape targets and exporters

Targets and exporters are the sources of truth for metrics, and a single exporter instance is itself a point of failure for the series it exposes. If an exporter goes down, its metrics disappear whatever the health of Prometheus. Inventory which exporters carry critical signals, run redundant instances where the platform allows, and treat exporters that take targets from URL parameters as an access control concern, because anyone who can reach them can direct requests to arbitrary endpoints.

Choosing between redundant pairs, remote write and a single server with backups

This table compares four ways to protect Prometheus availability, from identical server pairs to remote write and a single server with fast restore. Read each row for when the option fits, what it costs to operate and what it leaves unprotected. The main takeaway is that no option covers every failure, so the choice depends on which outages your alerting must survive.

Option	When it fits	Operational cost	What it does not protect against
Two identically configured servers scraping the same targets	Alerting must keep working when one server, host or zone fails, and teams accept duplicate data	Two scrape loads on every target, double storage and retention, and a query layer or dashboards that must handle duplicate series	Shared configuration errors, bad rules deployed to both servers, or a failure in the shared network path to targets
Single server with fast restore from backup	Small estates where short gaps in metrics are tolerable and the team can rebuild quickly	One server to run, plus regular backup of the TSDB and configuration and a tested restore runbook	Data gaps during the outage and any metrics not yet captured by the last backup, so alerts are silent until restore completes
Remote write to an external long-term store	Long retention, centralized queries or a durable copy of samples is needed beyond local disk	A second system to size, secure and operate, network and credential management, and queue tuning on the sending side	Loss of the local server still stops scraping and alerting, and a remote store outage can back up or drop samples depending on configuration
Query layer that reads across redundant replicas	Teams need one query endpoint over two or more Prometheus instances with the same targets	An extra component to deploy and upgrade, plus careful handling of replica labels and duplicate series in queries	Problems in the query layer itself, and it does not make alerts fire from a surviving server unless alerting is also set up for that path

Building a redundant Prometheus pair step by step

This sequence turns one Prometheus server into a redundant pair in which both replicas scrape the same targets, evaluate the same rules and send samples to a shared long-term store. Each step closes with a check, so the team confirms the pair works before alerting depends on it.

Inventory targets and confirm scrape coverage
Your platform team lists every job, target and exporter the pair must scrape, along with the service discovery source or static configuration behind each one. Mark the short-lived jobs that need the push gateway, and confirm that every target is reachable from both planned server hosts, not only from one network segment. The step is done when the inventory matches what current monitoring covers and every gap has a named owner. This list becomes the source for the configuration in the next step.
Deploy two identical Prometheus servers
Install the same Prometheus release on two separate hosts, ideally in failure domains your infrastructure team has agreed, and copy the same scrape and rule configuration to both. Give each server its own external label, such as a replica identifier, so series and alerts can be traced to the replica that produced them later. Leave the admin API disabled, which is its default, and do not enable the lifecycle endpoints unless you need them. Check by diffing the two configuration files and confirming both processes start and serve their web endpoints.
Point both servers at every Alertmanager
Configure each Prometheus server with the addresses of every Alertmanager instance in the cluster, and send alerts to each instance directly, skipping any load balancer, as the Alertmanager documentation describes. Alertmanager instances find each other through its cluster flags, so set those from the Alertmanager documentation for the version you run and confirm the exact flag names there. Check that both Prometheus servers list every Alertmanager as active, then send a test alert and confirm it reaches the receiver. Also confirm in the Alertmanager documentation how it handles the same alert arriving from both replicas.
Configure remote write to long-term storage
Add a remote write section to both Prometheus configurations that points to the chosen long-term store, using identical settings on each replica. Local storage is single-node and is not replicated, so this step is what preserves history beyond the local retention window, which defaults to 15 days. Confirm in the store's documentation the endpoint, authentication method and any tenant or label requirements. Check that samples from both replicas arrive in the store with their replica labels, and that the remote write queue on each server does not keep growing.
Stand up the query path across replicas
Place a query layer in front of both servers and the long-term store so dashboards and API consumers send PromQL to one endpoint. The layer has to combine series from the two replicas without double counting, which depends on the replica label set in the previous steps, so confirm that capability with the product you choose before you rely on it. Protect the query endpoint with the TLS and authentication controls described in the Prometheus security model. Check by running the same query through the layer and directly against each server, then comparing the results.
Load alert rules and verify evaluation
Deploy the same alerting and recording rule files to both servers and validate their syntax before loading, using the promtool checks included with the Prometheus distribution. Reload each server through the mechanism your team has enabled, and keep the lifecycle endpoints off if you do not need them. Check that both servers show the same rule groups, evaluate the same expressions and fire the same test alert. Investigate any difference before go-live, because replicas that disagree about rules will disagree about pages.
Test failure and record operating steps
Stop one Prometheus replica during a window agreed with the service owner, and confirm that the other keeps scraping, evaluating rules and sending remote write. Restart the stopped replica and check that it resumes scraping, and confirm from the long-term store's documentation how any gap in its history is handled. Record how the replicas are monitored, which team responds to a replica failure, and how upgrades will be rolled out one replica at a time. The step is done when the test results and runbook are signed off by the service owner.

How alerts from two Prometheus servers reach one Alertmanager cluster

Each Prometheus server evaluates its alerting rules and sends every firing alert to each Alertmanager instance listed in its alerting configuration. With two Prometheus servers and two Alertmanager nodes, that means four alert streams feed one logical service. The Alertmanager cluster is what turns those streams into a single view. The nodes share state for grouping, inhibition and silences, so a notification decision made on one node is respected by the others. Without that shared state, each node would group, inhibit and silence on its own, and the team would see different behavior depending on which node handled a given alert.

A load balancer in front of Alertmanager is a common instinct, but it is not the documented pattern. Alertmanager's guidance is for Prometheus to point at every instance directly, and for the Alertmanager nodes to form a cluster through their peering settings. Before you build this, confirm the cluster peering setup and the --cluster flags against the Alertmanager documentation for the version you run. Check the listen address, the peer list, and how each node advertises itself, since a node that cannot reach its peers will still accept alerts and quietly operate without shared state. Record the exact configuration in your runbook so the next person can see why each node lists the others.

The first test is a silence created on one node. Create a silence for a test alert through the web interface or API of one Alertmanager node, then query the other node and confirm the same silence appears and that the matching alert is marked as silenced there. Repeat the check from the other direction. If the silence appears on only one node, the cluster is not forming correctly, and the fault should be fixed before any production rule depends on silences. Also test expiry, because a silence that does not clear on schedule on every node causes the same confusion as one that never propagates.

The second test is duplicate alerts from the pair. Both Prometheus servers should fire the same alert with the same labels, and the expected result is one grouped notification per receiver, not two pages. Confirm this in the receiver logs or the test channel. Duplicates only group when their labels match, so check the external labels on each Prometheus server. If those labels differ, Alertmanager treats the alerts as separate, and the on-call engineer receives two pages for one fault. Finally, stop one Alertmanager node and confirm that notifications still arrive, because a test that ignores failover does not show high availability.

What drives high availability effort and cost on this build

This table lists the factors that change how much work a Prometheus high availability build takes and what it costs to run afterward. Read each row as a lever: the driver explains why it matters, and the second column shows how to keep it under control. The main takeaway is that most of the effort comes from scope decisions made before any configuration is written, so fixing the target inventory and alert ownership early matters more than any single setting.

Driver	Why it changes effort or cost	How to keep it down
Number of targets and scrape interval	Every target adds discovery, relabeling, authentication and monitoring of scrape health on each replica. Shorter scrape intervals raise load on every Prometheus instance in the pair and on remote storage.	Fix the target inventory before build, group targets by job, and set scrape intervals per job rather than globally. Confirm the interval each team actually needs.
Series cardinality	High label cardinality multiplies memory and disk use on both replicas and inflates remote write volume. It also makes queries and deduplication harder to verify.	Review label sets in the design phase, drop unbounded labels with relabeling, and agree a series budget per job with service owners.
Volume and retention of remote write data	Remote write sends every sample to an external long-term store, so ingest volume, retention period and the store's own sizing rules drive recurring storage and query costs.	Decide retention by use case (alerting history, capacity planning, audit), use recording rules to reduce what is sent, and confirm the store's pricing and sizing model with its vendor.
Number of alert receivers and routing rules	Each receiver needs credentials, templates, test paths and failover checks. Deep routing trees make it harder to prove that each alert reaches one accountable team.	Limit receivers to the channels teams actually use, document the routing tree with owners, and test each route during a clustered Alertmanager failover drill.
Number of environments (production, staging, air-gapped)	Each environment repeats the pair, the Alertmanager cluster, certificates and configuration. Air-gapped networks add offline packaging, internal mirrors and manual upgrade steps.	Build one reference configuration as code and parameterize it per environment. Agree early which environments need full high availability and which can run a simpler layout.
Compliance evidence needs	Assessors often ask for proof of configuration, access control, retention settings, backup and failover tests. Collecting this evidence after go-live takes more time than designing for it.	Map the controls your framework expects to specific configuration and logs, and capture failover test results and change records as the build proceeds.
Integration with existing on-call tooling	Connecting Alertmanager to paging, ticketing or chat tools involves webhook endpoints, authentication, escalation rules and duplicate suppression across two or more alert sources.	Inventory the on-call tools and escalation policies before design, confirm which alerts page and which create tickets, and test duplicate handling with both Prometheus instances firing.

How the HA design changes on a single host, Kubernetes and air-gapped networks

On a single host, the pair and clustered Alertmanager pattern cannot protect against the most likely failure, because both instances share one machine and one failure domain. Local storage is single-node and not replicated, so restore-based recovery protects your data but not your alerting. If the host fails, alerts stop until the host is rebuilt and the data restored.

Restore point. Samples written after the last usable backup are lost unless remote write or snapshots capture them. Ask how often backups run and how they are verified.
Restore time. Measure how long it takes to rebuild the host, reinstall the configuration and restore storage in your environment. Do not assume the figure from a previous platform.
Alerting gap. No alerts fire during an outage. Set up an independent check from outside the host that tells you monitoring itself is down.
When a pair is still needed. If a silent gap in alerting is unacceptable, run two Prometheus instances on separate hosts, each scraping the same targets, with Alertmanager instances spread across hosts as well.

Confirm the restore time, backup frequency and whether a second host is available before accepting a single-host design.

Securing replicas, the Alertmanager cluster and remote write credentials

A redundant Prometheus pair is only as trustworthy as its weakest endpoint. Each replica, Alertmanager peer, remote write sender and exporter exposes an interface that needs transport encryption, authentication and restricted control paths. The cards below cover the security settings to review for each of these, and where a separate authentication layer fits.

TLS on every listener

Prometheus supports TLS and client certificate authentication with TLS 1.2 as the minimum version. Apply it to each replica's web listener, to scrape connections and to remote write endpoints, so that both members of a pair and any client that queries them use encrypted connections. Confirm the certificate chain and rotation process with your PKI team before go-live, because a certificate expiry on one replica can silently break its scrapes or the comparison queries that depend on it.

HTTP basic authentication with bcrypt

Basic authentication works with passwords stored as bcrypt hashes in the web configuration file, so plaintext passwords do not need to sit on disk. Use it only over TLS. Without TLS, basic authentication sends credentials in cleartext, which exposes them to anyone on the network path between a client and a replica. Set a process for rotating these passwords on both replicas at the same time so that the pair and its dashboards stay in step.

Admin and lifecycle endpoints restricted

Admin HTTP APIs are disabled by default. The admin flag enables destructive operations such as deleting series, and the lifecycle flag enables reload and shutdown. Keep both off on production replicas unless a specific runbook needs them, and if they are needed, restrict them to a management network or to a named automation identity. Verify this on each replica separately, since a configuration drift on one member is easy to miss in a pair.

Authenticating proxy for SSO

Prometheus has no built-in single sign-on, and its security model notes there is no built-in CSRF protection on mutating endpoints. An authenticating reverse proxy in front of the web UI and API is the usual answer, with OIDC or SAML handled by the proxy. Decide early which identity provider and proxy you will use, and confirm that the proxy protects both replicas and Alertmanager. The separate SSO topic covers the detail of this setup.

Remote write and cluster peer traffic

Remote write sends samples to external storage, often with a bearer token or basic credentials, and Alertmanager cluster peers exchange state over their cluster port. Both carry sensitive data. Place remote write credentials in restricted files, not inline in the main configuration, and use TLS for both the remote write endpoint and the Alertmanager cluster channel. Confirm that peer ports are reachable only between Alertmanager instances, not from application networks.

Exporters that accept URL targets

Some exporters, including SNMP and Blackbox, take their targets from URL parameters. The Prometheus security model warns that anyone with HTTP access to such an exporter can make it send requests to arbitrary endpoints, which can turn the monitoring host into a probe for internal networks. Limit who can reach these exporters, run them with the narrowest network egress that their checks need, and review their scrape configuration for targets that users or tickets can change.

How to test Prometheus high availability without losing alerts

The core test for a Prometheus high availability design is simple: break one component on purpose and confirm that alerts still reach a person. Each redundant pair, whether two Prometheus servers scraping the same targets or a clustered Alertmanager, should be tested by stopping one member while the other keeps working. Run these tests in a non-production environment that mirrors your scrape configuration, rule files and receiver setup, and record what happened at each step so the results can be repeated after upgrades.

Start with the Prometheus pair. Stop one server and confirm that a test alert defined in your rules still fires and that the expected notification arrives through Alertmanager from the surviving server. Then stop one Alertmanager node and confirm that notifications still go out. Prometheus should be configured to send alerts to every Alertmanager instance, not through a load balancer, so check that the Prometheus configuration lists each Alertmanager address and that the surviving instance delivers the grouped notification. Confirm the behavior of duplicate notifications too, since the cluster is designed to deduplicate them and your receivers should be checked for that.

Next, restart one Prometheus server and watch the remote write queue. If you send samples to long-term storage, the restarted server should resume sending and the backlog should drain without manual intervention. Track the queue and send metrics that Prometheus exposes about remote write, and confirm that the backlog reaches a steady state within a window you agree on in advance. Ask your storage vendor or the Prometheus remote write documentation for the exact metric names for your version, and confirm them in your environment before writing the pass criteria.

Then test queries across both replicas. Run a PromQL query that spans the data from both servers and inspect the results for duplicate or missing series. Because the two servers scrape the same targets, some duplication is expected, and it should be handled by the query layer or by external labels, depending on your design. Compare the alert outcome rather than expecting identical sample values, because Prometheus documents that it values reliability over precision. Small differences in timestamps or values between replicas are normal, and a test that demands exact equality will fail for reasons that do not affect alerting.

Finally, check that failure is visible. A design that fails silently is worse than one that fails loudly. For every test, confirm that the loss of a server or Alertmanager node produces a signal, such as an alert on a scrape target being down, a dashboard panel showing a replica missing, or a watchdog alert that is expected and continuously firing, so that a missing signal is itself detected. Write these checks down as acceptance criteria. Decide who receives the failure alert, how long a silence may last during maintenance, and what the on-call engineer should do when one replica is gone, because a redundant design that no one notices is still a single point of failure.

What to monitor, back up, upgrade and document after go-live

Redundant Prometheus pairs and a clustered Alertmanager only stay useful if someone can see when one replica drifts, a remote write queue backs up or a receiver goes silent. The cards below cover what to watch, what to back up, how to plan upgrades and what to write down so on-call engineers can act quickly.

Replica health and scrape success

Each Prometheus replica exposes its own up series for every target, so a failed scrape shows which replica saw it. Track scrape duration and sample counts per replica and compare them across the pair. A replica that scrapes fewer targets than its peer is a monitoring gap, even when every alert looks quiet. Confirm that dashboards and alerts keep the replica label so the pair can be read side by side.

Remote write queue and failures

If replicas send samples to long-term storage through remote write, the queue and failure metrics show whether data is reaching it. Watch for queues that fill up, samples that are retried or dropped, and sends that fail for longer than usual. Alert on sustained growth so short spikes do not page anyone. Confirm the exact metric names for your Prometheus version in the storage documentation before building dashboards.

Alertmanager cluster membership and notifications

Each Alertmanager instance should report its cluster peers, and a member that drops out should raise an alert of its own. Alert on failed notifications per receiver, since a receiver that rejects messages is often the failure teams notice last. Prometheus should list every Alertmanager instance directly, so confirm that no load balancer sits in front of them. Send a test alert through each receiver after every routing change.

Backup of configuration, rules and state

Back up the Prometheus configuration, rule files, service discovery inputs and any secrets they reference, kept under version control with restricted access. Alertmanager configuration and silences also need a copy, so include its state directory in the backup plan. Local TSDB data lives on a single node, so decide whether losing a replica's history is acceptable or whether remote write covers it. Test a restore into a scratch environment.

Upgrade planning on the release cycle

Prometheus starts a new minor release cycle every 6 weeks, and a minor release generally stops receiving bug fixes after those 6 weeks. Teams that want a longer window can run the LTS line, where the current LTS is 3.13, released July 1, 2026 and supported until July 31, 2027. Test each upgrade in staging against your rules and storage, then upgrade replicas one at a time so the pair stays available.

Replica topology and failover runbook

Document which replica serves which targets, how the pair is labeled, where remote write points and how each Alertmanager is reached. The failover runbook should state what on-call checks first, how to confirm the surviving replica holds the rules and when to rebuild the failed one. Record the owner of every alert receiver, including the escalation contact and the hours they cover, and review that list at each rotation change.

What buyers ask about Prometheus high availability cost, time and effort

What actually drives the cost of Prometheus high availability?

The cost is driven by scope, not by a license fee, because Prometheus is Apache-2.0 licensed and has no paid enterprise edition. The main drivers are the number of scrape targets, how much data you keep locally and for how long, whether you add a long-term store behind remote write, how you expose the query layer to users, and how much testing and handover the work includes. Ask each partner to price these separately so you can see which items are one-time work and which recur as infrastructure or upkeep. The Prometheus cost of ownership topic covers the recurring side in more detail.

Can our own team build a redundant Prometheus pair alone?

Yes, many teams can, provided they have engineers comfortable with Prometheus configuration, service discovery, PromQL and Alertmanager clustering. The storage docs are clear that local storage is not clustered or replicated, so the design has to be built deliberately: two Prometheus instances scraping the same targets, Alertmanager instances that Prometheus points at individually, and a clear plan for querying across replicas. The harder part is usually the operating model, including who gets paged, how upgrades are tested against alert rules, and how the setup is documented for the next engineer.

What do we need to provide before a Prometheus high availability engagement starts?

Expect to supply a target inventory, network access from the Prometheus hosts to every scrape target and to the Alertmanager receivers, and a named on-call owner who can accept the new alerting path. You will also need contacts for each notification receiver, such as your paging and chat tools, and a decision on storage: where local data lives, how large the volumes are, and whether a long-term store is in scope. Missing inventory and unclear receiver ownership are the most common reasons scope changes after kickoff, so resolve them in the written scope.

How do we ask a partner for a fixed-scope quote?

Ask for a written scope that lists the targets or services in scope, the exporters and service discovery methods, the storage and retention design, the Alertmanager and receiver work, the query layer, testing, documentation and handover, and what is excluded. Ask which items are one-time and which are recurring, and what assumptions the price depends on, such as the number of targets or the choice of long-term store. DeployOpen works from a written scope and does not publish a price list, so a quote follows from the scope you agree. Compare quotes on the same scope, not on headline totals.

Does a high availability design need compliance evidence?

It depends on the framework you are assessed against and on what the auditor asks you to show. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications, and independent auditors make that determination. For monitoring, the evidence often includes documented configuration, access controls on the Prometheus and Alertmanager interfaces, retention settings and records of alert handling. Confirm early which controls your assessor expects you to evidence, and ask the partner how the design will produce that evidence. Authentication and TLS choices should be settled before the evidence is collected.

How long does Prometheus high availability take to put in place?

The duration depends on the number of targets, the number of teams that rely on alerts, the storage and long-term design, and how much testing and handover are included. No general duration is supplied here, so ask each partner for a timeline tied to the written scope and to your change windows. The Prometheus implementation plan and cost topic describes how these factors are scoped, and it is the right place to compare sequencing and dependencies before you commit.

What breaks most often in a redundant Prometheus setup?

The most common problems are duplicate or missing alerts when replicas disagree, Alertmanager instances that are not all listed in the Prometheus configuration, and queries that return gaps because one replica was down. Teams also run into storage that fills faster than planned, scrape targets that were never inventoried, and upgrades that change behavior in rules. Check that every Alertmanager instance is listed, test a replica failure on purpose, and confirm that dashboards and rules work when one instance is unavailable. Pair these checks with your own runbook so the on-call owner knows what a degraded state looks like.

How DeployOpen works on Prometheus high availability

DeployOpen plans and builds Prometheus high availability inside your own infrastructure, starting from a written scope that names the Prometheus pairs, Alertmanager cluster and remote write target involved. Each stage ends with documented configuration, tested failover behavior and an operating handover your team can run without us.

How we scope

Inventory the targets, rule groups and alert receivers that must survive a single Prometheus instance failing.
Confirm where the long-term store will run, who owns it and which query path dashboards will use.
Agree in writing the deliverables, exclusions and timeline, since most engagements are fixed-scope.

How we build

Deploy redundant Prometheus instances with identical scrape and rule configuration, then verify both scrape the same targets.
Configure a clustered Alertmanager and point every Prometheus at each Alertmanager instance rather than a load balancer.
Set up remote write to the agreed durable store and test queries that span replicas, including known trade-offs in reliability versus precision.

How we hand over

Test failover by stopping each instance and Alertmanager node, recording what alerts fire, duplicate or go silent.
Document configuration, storage retention, TLS and authentication settings, and the runbook for restoring a failed node.
Walk your operators through the runbooks and leave ownership of configuration, credentials and rule changes with your team.

How we support

Offer ongoing support, upgrades and testing as a retainer, covering Prometheus release changes that affect the HA setup.
Review remote write backlog, storage growth and Alertmanager cluster health against the agreed runbooks.
Re-test failover after upgrades or topology changes, and report what changed and what your team should confirm.