How Kubernetes monitoring with Prometheus fits your cluster
Prometheus is Apache-2.0 open-source software that pulls metrics over HTTP from targets it discovers through the Kubernetes API, and it runs inside your own clusters and data centers, so metric samples and alert history stay under your control. For a platform team running self-managed clusters, the first decision is scope: which signals Prometheus owns, which it does not, and who maintains each piece. Logs and traces fall outside Prometheus, and dashboards are usually built in a separate tool that queries its data. Confirm early which visualization tool your organization will use and who owns its access model.
The integration covers scrape configuration, including the Kubernetes service discovery roles and the relabeling rules that decide which pods, services and endpoints are scraped and how their labels are shaped. It also covers control plane metrics from the API server, scheduler and controller manager, node metrics from the kubelet and a node exporter, and state metrics for objects such as deployments and pods. Your team then writes alert rules, runs Alertmanager for grouping, silences and notification routing, and decides how long data stays on local disk and when it is sent to external long-term storage through remote write.
The integration does not cover the instrumentation inside your applications, which determines what metrics those workloads expose and must be designed by each application owner. Prometheus also does not provide exact event counts. Its documentation says it values reliability over precision, so it is a poor fit where every event must be counted exactly, such as per-request billing. Local storage is not clustered or replicated, so each Prometheus server behaves like a single-node database that needs disk planning, backups and a restart plan. Confirm each of these boundaries against your ownership model before the build starts.
There is no paid edition of Prometheus itself, so its cost comes from the infrastructure that runs it, the storage that holds its data, the implementation work and the people who operate it. Infrastructure includes compute for each Prometheus server and Alertmanager instance, plus disks sized for your retention period. Storage grows with the number of active series, the scrape interval and the retention you choose, and long-term storage is a separate system with its own license and operating cost. Implementation is usually a one-time effort, while upgrades, rule maintenance and on-call response recur. Prometheus minor releases follow a six-week cycle, so plan on the long-term support line, which the project's release cycle page lists as supported until July 31, 2027 for the current line. Ask for a quote that separates these components, states assumptions about cluster count and retention, and names who maintains each one after handover.
What you need before configuring Prometheus for Kubernetes
Six inputs decide whether the Prometheus configuration in the following sections works on the first pass. Confirm each one with the platform owner, the security team and the people who will run the stack after go-live. Together they set which Kubernetes endpoints Prometheus can reach, the release line you will upgrade on, and the disk, permissions and alerting components it depends on.
Supported Kubernetes version and distribution
Confirm the cluster runs a minor version the Kubernetes project still maintains. As of September 2026 the maintained lines were 1.37, 1.36 and 1.35, and the release page lists an end-of-life date for each. Record the distribution as well, since kubeadm, managed services and vendor builds differ in how control plane metrics endpoints are exposed and how kubelets are configured. Ask the platform owner to confirm the version in writing before you choose scrape targets, because service discovery depends on the API objects that version provides.
Prometheus release line
Choose between the long-term support line and the latest minor release. The current LTS line is 3.13, which receives bug fixes until July 31, 2027, and the latest release is v3.15.0. Minor releases generally stop receiving bug fixes six weeks after their cycle begins, so a standard minor release needs a dated upgrade plan. Pin the exact image tag in your manifests, confirm where the image is sourced from, and record the upgrade window in your change calendar before the first deployment.
Storage for local retention
Prometheus writes samples to a local time series database organized in two-hour blocks with a write-ahead log, and it keeps 15 days of data by default. Size the volume from your scrape interval, series count and the retention limit you plan to set, whether by time or by size. Local storage has no built-in clustering or replication, so the volume needs its own snapshot or backup approach and a storage class that supports expansion.
RBAC rights for scrape identities
Prometheus discovers targets through the Kubernetes API, so it needs a service account with read access to the objects its scrape jobs use, such as nodes, pods, services, endpoints and ingresses. Confirm who can create the cluster roles and bindings, and who approves them. Keep the rights read-only and scoped to what each job requires, and list the namespaces Prometheus may watch. Broad cluster-wide read access increases the impact if the Prometheus pod is compromised.
Network paths to scrape targets
Map every flow Prometheus needs: to kubelets for node and container metrics, to the kube-apiserver for discovery and control plane metrics, and to etcd endpoints if you scrape etcd directly. Confirm that network policies, host firewalls and any mutual TLS requirements allow these flows from the namespace where Prometheus runs. Test each path with a short-lived pod before deployment, and document which endpoints need client certificates so the scrape configuration matches the cluster's authentication setup.
Alertmanager and Grafana placement
Decide where Alertmanager and Grafana run before writing alert routes. Alertmanager can run as a cluster for high availability, and Prometheus should be configured with the address of every Alertmanager instance, with no load balancer in front. Grafana documentation states that SQLite does not scale for production, so use MySQL or PostgreSQL for production and high availability. Confirm which team owns the on-call integrations, the Grafana database and whether these components sit inside the cluster or in a separate monitoring environment.
How Prometheus discovers Kubernetes targets
Prometheus collects metrics by pulling them over HTTP from each target on a schedule, so it must know which endpoints exist before it can scrape them. In a fixed fleet of virtual machines, a static list of addresses can stay accurate for a long time. A Kubernetes cluster does not work that way. Pods are scheduled, restarted and replaced constantly, and their IP addresses change with them. Kubernetes service discovery lets Prometheus query the Kubernetes API for the objects that exist and keep its target list current between scrapes. The main decisions are which discovery roles to enable, how to scope each one to namespaces and what each scrape job is meant to cover.
Each discovery role returns a different kind of target with its own metadata. The node role yields one target per cluster node and is the usual starting point for kubelet and node-level metrics. The service role yields targets for Service objects and suits probing with an exporter. The pod role yields one target per pod port and gives the finest granularity, which is why it is common for application metrics. The endpoints role lists the individual addresses behind a Service, and the ingress role lists the hosts and paths declared in Ingress objects. Because the metadata differs by role, a rule written for pod labels will often match nothing when applied to node targets. Confirm the exact role names and the metadata each one exposes against the Prometheus version you run and the Kubernetes API version in your cluster.
Relabeling rules decide what happens to each discovered target before Prometheus scrapes it. A keep or drop action filters targets by their metadata, so a job can be limited to pods with a specific annotation or to one namespace. A replace action copies or builds labels, for example turning pod metadata into job, namespace or instance labels that dashboards and alerts can rely on. Relabeling can also rewrite the scrape address and path. Discovery metadata labels carry a double-underscore prefix and are discarded after relabeling unless a rule copies them into a regular label. The rules run in order, so an early drop can make a later rename irrelevant. Teams should document the intended label set for each job before writing the rules.
Relabeling mistakes are a common source of both missing and duplicated series. A regular expression that does not match the metadata the cluster actually produces will silently drop an entire job. Two jobs that discover the same workload, for example through the pod and endpoints roles, can scrape it twice under different label sets, and dashboards then show doubled counters or gaps after a rename. To check the result, open the target and service discovery pages in the Prometheus web interface, verify the labels on a sample of targets and validate the configuration with promtool before each rollout. Include these checks in the acceptance criteria for any implementation work, whether your own team or a partner carries it out.
Kubernetes monitoring setup with Prometheus, step by step
This sequence takes a Kubernetes cluster from no metrics pipeline to a Prometheus deployment that discovers targets, evaluates alert rules, routes notifications and optionally ships data to long-term storage. Each step ends with a check you can run before moving on, so problems surface while the scope is small. The steps assume a plain Prometheus configuration, so confirm the mapping if you use an operator-based stack.
Create namespace and read-only discovery RBAC
Create a dedicated monitoring namespace, a service account for Prometheus and a ClusterRole that allows reading the objects Kubernetes service discovery needs, such as nodes, services, endpoints, endpoint slices and pods, along with the node metrics paths. Bind that role to the service account and keep it free of write verbs and of list or watch access to Secrets. Check: run an authorization check as the service account to confirm list and watch succeed on pods and nodes, while creating or deleting pods and reading Secrets is denied.
Deploy Prometheus with sized persistent storage
Deploy Prometheus as a single instance with a persistent volume mounted at its data directory, and set retention by time or size to match the history your team needs to query. The default retention is 15 days, so change it deliberately. Local storage is not clustered or replicated, so treat the volume like any single-node database and plan its backups and failure handling. Confirm the storage class and volume size with your storage team, because they depend on your ingest rate. Check: open the status page to confirm the retention flags, then restart the pod and confirm that stored data survives the restart.
Add scrape jobs for control plane and nodes
Add scrape jobs for the API server, kubelets, cAdvisor and etcd, each using the Kubernetes service discovery role that fits its endpoints. The API server and kubelet jobs use the service account token and the cluster CA, the cAdvisor job reads the kubelet's container metrics path, and the etcd job needs the client certificates your cluster was built with. Managed control planes may hide etcd or some API server endpoints, so confirm what your provider exposes. Check: in the targets view, each job lists its expected endpoints as up, and a control plane query returns series for every node.
Relabel pods that opt in to scraping
Add a relabeling rule to the pod job that keeps only pods carrying an agreed annotation, such as a scrape flag set by the application team. Map the port and path annotations into the target's address and metrics path. Write the annotation names into a short standard so every team uses the same convention. Pods without the annotation are dropped before they become targets, which keeps scrape load and series counts under control. Check: deploy one test pod with the annotation and one without it, then confirm only the annotated pod appears in the targets view.
Load alert rules from version-controlled files
Store alerting and recording rules as files in a Git repository, reviewed through the same process as application code. Mount them into Prometheus through a ConfigMap or the mechanism your deployment tooling uses, reference the rule files from the main configuration and reload after each change. Start with rules for scrape failures, control plane availability and etcd health, and link a runbook in each alert's annotations. Check: validate the rule files with promtool in your pipeline, then confirm the rules page shows each group as loaded with no errors.
Connect Alertmanager and send a test alert
Point Prometheus at every Alertmanager instance it should use, then configure grouping and routing for the teams that own each alert. Set up at least one receiver, such as your on-call tool or a chat channel. Alertmanager handles grouping, inhibition and silences. If you run it as a cluster for high availability, Prometheus should list each instance through its own address so every instance receives alerts. Check: in a non-production copy, fire a rule that is always true, confirm it appears in the Alertmanager interface and reaches the receiver, then confirm the resolved notification arrives.
Add remote write for long-term storage
Add a remote write section only if local retention cannot meet your history requirement, and choose a receiver that supports the protocol along with your retention and access model. Remote write sends samples to external storage, while the local instance remains the source for live alerting, so keep the local retention window in place. Load credentials through the secrets mechanism you already use, tune queue settings with the receiver owner, and record who owns the receiver. Check: query a recent series in the receiver, then confirm Prometheus's remote write metrics show samples sent with a failure count that is not climbing.
Which control plane, node and workload metrics to collect
The control plane tab covers the components that accept, validate and store cluster changes. The API server matters first, because every other component depends on its responsiveness. etcd matters next, because it holds all cluster state and a slow or unhealthy member affects the whole cluster.
- API server requests and latency: request counts by verb, resource and response code, plus request duration histograms. These show error spikes and slow reads or writes.
- API server health: readiness and liveness signals, and any admission-related latency your version exposes.
- etcd leadership and health: whether members have a leader and whether leader changes are increasing over time.
- etcd database size: the database size reported by etcd, tracked against the backend quota you configured.
- etcd scrape endpoints and certificates: in a self-managed cluster, etcd serves its own metrics endpoint, so Prometheus needs a dedicated scrape job and client certificates that match how etcd was started. Managed control planes often hide etcd, so this work does not apply there.
Before writing alert rules, confirm the exact metric names and labels against the Kubernetes and etcd versions running in your cluster, because they change between releases.
The node tab covers whether each machine has capacity and whether the kubelet on it is working. Two sources matter here. The kubelet exposes its own operational metrics and per-container resource usage, and a node-level exporter reports host metrics that the kubelet does not cover, such as disk, filesystem, network and system service state.
- Kubelet operation: pod lifecycle counts, kubelet operation latency and error counters, which show a node that is up but struggling to run its pods.
- Container CPU and memory: container usage and working set metrics served through the kubelet, which support capacity planning and CPU throttling analysis.
- Node-level exporter: a DaemonSet such as node_exporter collects host CPU, memory, disk, filesystem, network and systemd metrics from every node, so the same dashboards and alerts apply across the fleet.
- Disk and filesystem pressure: free space on the root and image filesystems, which often triggers pod evictions before any other alert fires.
- Node conditions: Ready and pressure conditions exported as object state, which show what the scheduler sees when placing pods.
Before relying on these names, confirm which kubelet and container metrics your Kubernetes version exposes, and check whether your distribution already deploys a node exporter so you do not collect the same series twice.
The workload tab covers metrics from the applications and from the objects that run them. An application exposes metrics on an HTTP path, and Prometheus has to be told where that path is through service discovery. Two patterns are common, and the choice depends on whether the team already runs the Prometheus Operator.
- Annotation-based discovery: the pod carries annotations that name the port and path, and a scrape job built on the Kubernetes pod role reads them. Prometheus does not act on these annotations by default, so the scrape configuration must be written to use them.
- ServiceMonitor pattern: with the Prometheus Operator, a ServiceMonitor resource selects Services by label and defines the endpoint, which keeps scrape configuration alongside the application manifests.
- Application metrics: request rate, errors and duration, plus queue depth and business counters the team defines, exposed on a stable path with bounded label values.
- Object state: Deployment, StatefulSet and Job state from kube-state-metrics, showing desired versus available replicas and failed jobs.
- Access to metrics endpoints: decide whether each endpoint needs TLS or a token, especially in namespaces shared by several teams.
Confirm the annotation names or label selectors your discovery configuration uses and the operator version you run, then test one target end to end before rolling the pattern out to every application.
How Kubernetes labels map to Prometheus labels and series
This table maps common Kubernetes object metadata to Prometheus labels, showing where each value comes from, which discovery role and relabel action produce it, and why it matters for alert routing and dashboards. Keep a label when someone routes alerts or filters panels with it, and drop it when it only adds unbounded series. Team and cost-center labels need an explicit copy step because pod discovery does not carry namespace annotations by default.
| Prometheus label | Kubernetes source | Discovery role and relabel action | Why it matters |
|---|---|---|---|
| namespace | metadata.namespace on pods, services and endpoints | pod, service or endpoints role; __meta_kubernetes_namespace copied to namespace | Routes alerts to the owning team and scopes dashboard variables |
| pod | metadata.name of the pod behind the scraped target | pod role; __meta_kubernetes_pod_name copied to pod, then reviewed for cardinality | Pinpoints a failing replica, but each replacement pod starts new series |
| container | Container name in the pod spec | pod role; __meta_kubernetes_pod_container_name copied to container | Separates sidecar metrics from application metrics in per-container panels |
| node | spec.nodeName on the pod, or the node object for kubelet and cAdvisor metrics | pod role adds __meta_kubernetes_pod_node_name; node role scrapes kubelet endpoints | Connects host pressure alerts to the workloads running on that node |
| service | Service name behind an endpoint or endpointslice | endpoints or endpointslice role; __meta_kubernetes_service_name copied to service | Gives a stable grouping across replicas for latency and error-rate alerts |
| workload (app label) | Labels such as app.kubernetes.io/name on pods | pod role; labelmap on __meta_kubernetes_pod_label_ with an agreed allowlist | Groups replicas under one owner; copy only label names the teams agree to keep |
| team or cost-center | Annotations or labels on the namespace | Needs an explicit copy, for example through a kube-state-metrics annotation allowlist or a join; confirm for your versions | Drives alert ownership and showback reports for shared clusters |
| High-cardinality values | Pod names, container IDs, request paths, user IDs | Drop with metric_relabel_configs or limit scrapes before samples reach storage | Each unique value adds series, raising storage and query cost over time |
| Label names over releases | Label names referenced in alert rules and dashboards | Rename only through a planned migration that keeps old and new series for a transition period | Silent renames break alert routes and dashboard queries without an error |
Alert rules, Alertmanager and long-term storage for Kubernetes
Alerting and retention are two separate decisions in a Prometheus deployment for Kubernetes, and both should be made deliberately before the first dashboard goes live. Prometheus evaluates alert rules on a schedule and sends each firing alert to Alertmanager. Alertmanager then decides what happens next: it groups related alerts so one incident does not produce dozens of pages, applies inhibition rules so a downstream symptom is suppressed when its root cause is already firing, and honors silences that operators create for planned work. Rules for Kubernetes typically cover node pressure, pods stuck in restart loops, unavailable deployments and control plane health. Your team should own those rules as code, reviewed like any other change.
Alertmanager can run as a cluster for high availability through its --cluster flags, which let instances share notification state so a single alert is not sent twice. Prometheus should be configured with the address of every Alertmanager instance so it delivers each alert to all of them. Do not place a load balancer in front of Alertmanager for this traffic. A load balancer would send alerts to one instance at a time and can hide an instance that has failed. Confirm the exact flags, ports and cluster peer settings for your Alertmanager version against the Alertmanager documentation, and test a failover by stopping one instance during a controlled alert.
Local storage is the default, and its limits are the first thing to understand. Prometheus writes its time series database in two-hour blocks, backed by a write-ahead log, and keeps data for 15 days unless you change the retention setting by time or size. Local storage is not clustered or replicated. Each Prometheus server holds its own data, so a failed disk or node means lost history unless you have a backup or a remote copy. Treat the Prometheus volume like any single-node database: size it for your series count and retention, monitor free space, and plan the recovery steps before you need them.
Retention is a trade-off between cost, query needs and recovery. Longer local retention increases disk use and can slow compaction and queries, so it is usually paired with a plan for larger volumes. Remote write sends samples from Prometheus to an external long-term storage system, and remote read lets Prometheus query that storage. Remote storage adds network traffic, another system to operate, and a dependency that can fall behind during an outage. Ask your team which questions need history beyond 15 days, how long compliance or incident review requires data, and what it would cost to rebuild data that was never retained. Write those answers down as the retention decision for the platform.
Securing Prometheus scrapes, endpoints and credentials
Prometheus holds detailed data about your clusters, and its scrape configuration reaches into kubelets, etcd and application endpoints. The controls below decide who can read that data, what Prometheus can reach and where credentials live, so review them as one design before the first scrape job goes live.
Least-privilege RBAC for discovery
Prometheus's Kubernetes service discovery reads Pods, Services, Endpoints, Nodes and Ingresses through the Kubernetes API, so its service account needs list and watch permissions only. Scope them with a ClusterRole only where cluster-wide discovery is required, and use Roles in individual namespaces otherwise. Confirm in your own cluster that the account cannot read Secrets, and review any wildcard verbs added during troubleshooting.
Authenticated scraping of control plane endpoints
Kubelet and etcd metrics endpoints can require client certificates or bearer tokens, and the scrape configuration must present them. Keep certificate files and tokens in mounted Kubernetes Secrets referenced by path, not inline in the configuration file, and rotate them on the same schedule as the certificates they belong to. Confirm which control plane endpoints your distribution exposes without authentication before you rely on that assumption.
Network access to metrics endpoints
Metrics endpoints often expose internal names, versions and workload behavior, so they should not be reachable from every namespace. Use Kubernetes NetworkPolicy to allow ingress to exporters and application metrics ports only from the Prometheus namespace, and confirm that your network plugin enforces these policies. Test the rules from a pod outside the allowed source to verify that the block works.
Protecting the Prometheus web UI and API
The Prometheus web UI and HTTP API have no built-in user accounts, so access control has to come from the layer in front of them. Place them behind an authenticating proxy or ingress with SSO, and do not expose the service directly to users or public networks. Read the Prometheus security model documentation to see what the project protects and what it leaves to you, and review the lifecycle and admin endpoints before publishing the service.
Alertmanager receiver credentials in Secrets
Alertmanager receivers need webhook URLs, SMTP passwords and API tokens that grant access to paging and chat systems. Store them in Kubernetes Secrets and reference them from the Alertmanager configuration, then limit get and list access on those Secrets to the Alertmanager service account and the people who manage it. Because Secrets are stored unencrypted in etcd by default, confirm that encryption at rest is enabled for your cluster.
Common Kubernetes Prometheus failures and how to fix them
This table lists the failures teams most often hit when running Prometheus against a Kubernetes cluster, with the behavior behind each one and the fix to check first. Read each row as a symptom you can search for in targets, logs and storage, followed by a fix you can verify. The main takeaway is that most gaps come from discovery permissions, scrape configuration or storage design, not from Prometheus itself.
| Failure | Cause | Fix |
|---|---|---|
| Targets missing from discovery | The Prometheus service account lacks RBAC list and watch permissions on pods, services, endpoints or nodes, so kubernetes_sd_configs returns nothing for those roles. | Grant read-only list and watch on the resources each service discovery role needs through a ClusterRole and ClusterRoleBinding, then confirm targets appear on the Prometheus targets page. |
| Kubelet or etcd scrapes fail with TLS errors | Scrape jobs reference the wrong CA bundle, client certificate or server name, or the kubelet serving certificate does not match the hostname Prometheus connects to. | Set tls_config with the cluster CA and a valid client certificate, use the node's correct server name or skip verification only as a temporary diagnostic, and check the scrape error text on the targets page. |
| Relabeling drops every target | A keep action regex does not match the label values produced by discovery, or a drop rule matches a common label such as a namespace or app name used by all workloads. | Test each relabel_configs rule against sample discovery labels, start with one job, and review the dropped targets list on the Prometheus status page before expanding the rules. |
| Duplicate series from overlapping jobs | Two scrape jobs, such as a pod-level job and a service-level job, collect the same endpoint, or the same target is discovered through more than one role without distinguishing labels. | Keep one job per metric source, use the annotation or label filters to select each target once, and compare series counts per job after changes. |
| Disk exhaustion from long retention | Retention is set by time or size larger than the volume can hold, and high series cardinality grows blocks faster than expected. | Size the volume from observed ingestion rate and retention, set a size-based limit alongside time-based retention, and alert on free disk and on the head series count. |
| Alerts fire twice through a load balancer | Prometheus sends alerts to a load balancer in front of Alertmanager, so each Alertmanager instance receives the alert through a different path and cluster deduplication does not apply as intended. | Point Prometheus at every Alertmanager instance through the alerting configuration or service discovery, and let the Alertmanager cluster handle deduplication with its cluster settings. |
| Gaps after Prometheus restarts | Storage is an ephemeral container filesystem or an unmounted volume, so the write-ahead log and blocks are lost or not recovered when the pod is rescheduled. | Mount a persistent volume for the data directory, verify the data path inside the pod, and test a restart while watching whether recent samples are still queryable. |
Questions buyers ask about Kubernetes monitoring with Prometheus
How much does Kubernetes monitoring with Prometheus cost to run?
Prometheus has no license fee because it is Apache-2.0 licensed and the project has no paid enterprise edition. Costs come from four places: compute and storage for the Prometheus servers and Alertmanager, long-term storage if you add an external system, the implementation work to design scrape configuration, alert rules and dashboards, and the people who run it after handover. Infrastructure and storage recur and grow with series count, scrape interval and retention. Implementation is mostly one-time, although rule tuning and upgrades recur. Ask any quote to separate one-time build from recurring operations, name who pays for each item, and state the assumed series count, retention period and number of clusters. If you also use Grafana for dashboards, price its hosting and license separately.
How long does a Prometheus monitoring implementation take?
Duration depends on scope: the number of clusters, whether long-term storage is included, how many teams own alerts, and how much existing monitoring must be migrated or retired. A single cluster with control plane, node and workload scraping, plus a first set of alert rules, is a much smaller job than a multi-cluster platform with remote write, Alertmanager routing to on-call tools and dashboards for several teams. Access timing often matters as much as build effort. Cluster credentials, approved network paths to scrape targets and retention decisions can wait on other departments. Ask the written scope to list milestones, the client tasks each milestone depends on and the acceptance checks for each one. DeployOpen engagements are fixed-scope, so the agreed scope and timeline are written before work starts.
What do we need to provide before Prometheus monitoring work starts?
Before work starts, you typically provide read access to the clusters and a named platform contact. You also need a decision on retention: Prometheus keeps local data for 15 days by default, and you must decide whether longer history goes to an external long-term storage system through remote write. Name an owner for each alert and the receiver it routes to, such as an on-call tool, a chat channel or a ticketing queue. Network paths from the Prometheus servers to scrape targets and to Alertmanager receivers must be approved, and the hosting platform needs storage sized for the chosen retention. Alerts without owners tend to be written and then ignored. Ask the quote to list each input, its owner and the date it is needed.
Can our internal team set up Prometheus monitoring alone?
Many internal teams can build a working setup, since Prometheus is open source and its documentation is detailed. Success depends on skills and time. The common gaps are Kubernetes service discovery roles and relabeling, sizing memory and storage as series grow, designing alert rules that page only on symptoms people can act on, configuring Alertmanager for high availability, and planning long-term storage. Local storage is not clustered or replicated, so a failed Prometheus server creates a monitoring gap unless you design redundancy on purpose. Ask whether someone on your team has run a time-series database in production and can own alerting after handover. Many teams keep operations in-house and request outside help only for the design and the first rule set.
How are Prometheus upgrades handled across new minor releases?
Upgrades follow the Prometheus release cycle. A new minor release cycle begins every 6 weeks, and a minor release generally stops receiving bug fixes after those 6 weeks. Teams that want fewer upgrades can run the long-term support line, which receives bug fixes for one year. The current LTS line is 3.13, supported until July 31, 2027. Upgrade work covers reading release notes for breaking changes, testing scrape configurations and alert rules in a staging cluster, upgrading Alertmanager and exporters where needed, and checking compatibility with the Kubernetes version, since Kubernetes itself ships roughly three minor releases a year. Ask the quote whether upgrades are part of the build, who runs them and whether they fall under a retainer.
Where do Prometheus monitoring setups usually fail?
Most failures come from scale, labels and alert design. High-cardinality labels, such as request IDs or pod names in fast-churning workloads, grow memory and disk use until queries slow down. Relabeling mistakes can drop targets silently, so dashboards look healthy while a namespace has stopped reporting. Alert rules without owners or runbooks get muted, and misconfigured Alertmanager routes send pages to the wrong team. Because local storage is not replicated, a lost volume can mean lost history unless backups or remote write exist. Useful checks include target health per job, series count trends, a dead-man alert that proves the alert path works, and a test alert for each receiver after every configuration change.
What does a Prometheus monitoring engagement with DeployOpen include?
DeployOpen plans, implements, configures, documents and supports Prometheus in infrastructure you control, including your cloud accounts, data centers or air-gapped networks. The agreed scope can cover Kubernetes service discovery roles and relabeling, scraping for control plane, node and workload metrics, alert rules with routing through Alertmanager, long-term storage design if you choose it, and handover documentation. Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts. DeployOpen does not publish a price list, so a quote follows from the written scope. Ongoing support, upgrades and continuous testing are available as retainers. DeployOpen can prepare monitoring evidence for compliance assessments, but independent auditors issue any audit reports.
How DeployOpen scopes and delivers Kubernetes monitoring
DeployOpen plans and builds Prometheus monitoring for Kubernetes inside your own clusters and infrastructure, starting from the questions your teams need answered. Each engagement is fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. Ongoing support, upgrades and testing are available as a retainer.
How we scope
- Inventory of clusters, namespaces, workloads and the services teams need to watch
- Service discovery roles, relabeling rules and scrape targets written down per cluster
- Written scope covering alert rules, Alertmanager routing, retention and long-term storage choices
How we build
- Prometheus scrape configuration for node, control plane and workload metrics, tested against live targets
- Alerting rules tested against sample data before they are allowed to page anyone
- Alertmanager cluster layout reviewed, with every Prometheus pointing at each Alertmanager instance
How we hand over
- Runbook for storage sizing, retention settings and what happens when local disk fills
- Documentation of local storage limits, since a single Prometheus server does not replicate its data
- Walkthrough of PromQL queries, dashboards and alert ownership with your platform and on-call teams
How we support
- Retainer options for upgrades that follow Prometheus LTS and minor release cycles
- Continuous testing of alert rules and scrape configuration as clusters change over time
- Ongoing review of retention, series growth and Alertmanager health as workloads grow

