Kubernetes monitoring with the Prometheus Operator: what it does and who it suits
A platform team running several clusters needs scrape targets, alert rules and retention settings that anyone on the team can review and reproduce. The Prometheus Operator meets that need by managing Prometheus, Alertmanager and their configuration through Kubernetes custom resources. ServiceMonitor and PodMonitor resources describe what Prometheus scrapes, and PrometheusRule resources hold the alert rules. These objects can be stored in Git, reviewed like any other change and applied to each cluster in the same way. Without this, scrape configuration tends to drift between clusters as teams add exceptions by hand. This section explains what the Operator does, what it costs to run and which teams are likely to benefit from it.
Prometheus is open source under the Apache-2.0 license, and there is no paid enterprise edition of the project itself. That means the software carries no license fee, and the cost sits in everything around it. You pay for compute for Prometheus and Alertmanager replicas, storage for the time series database, the implementation work to design selectors, namespaces and alert routing, and the engineers who upgrade, watch and troubleshoot the stack over time. Implementation is largely a one-time cost, while compute, storage growth and operator time recur. When you request a quote, ask who builds the rule library, who owns response to alerts, and how storage growth is estimated and reviewed as clusters are added.
This approach suits teams that already operate Kubernetes, manage cluster configuration through Git and need the same monitoring standard across several clusters. It is a weaker fit for a single small cluster where a few static scrape jobs would do, or for an organization with no one who can read Kubernetes resources and troubleshoot a failed rollout. Two properties of Prometheus shape the decision. Its documentation says it values reliability over precision and is not a good choice where 100% accuracy is required, such as per-request billing. Its local storage is also not clustered or replicated, so the design must account for how long data is kept locally and whether remote write sends samples to a long-term store.
This page does not cover the current Operator or kube-prometheus versions. Confirm both against the project's own release notes before you build, and check which Kubernetes versions each Operator release supports, since your clusters follow their own upgrade cycle. Prometheus minor releases move quickly: a new minor release cycle begins every six weeks, and a minor release generally stops receiving bug fixes after those six weeks. Long-term support releases receive bug fixes for one year, which gives a longer window to plan upgrades. Pin the versions you test, record them in the same repository as your manifests, and name who reviews each release before it reaches production clusters.
Before you commit, run three checks. First, list the clusters and the services in each that need scrape targets, and decide which namespaces the Operator will watch. Second, write down retention in both time and size, and confirm the default of 15 days fits your review and investigation needs. Third, decide who answers alerts and how Alertmanager routes them, because the Operator automates delivery of configuration but not the judgment behind it. If the answer to any of these is unknown, the design and operating choices later on this page will be difficult to scope accurately.
What the cluster and team need before the build starts
Monitoring with the Prometheus Operator depends on three things being ready before any Helm chart is applied: a cluster the team can run, storage sized for the retention you choose, and a named owner for every alert that pages someone. The cards below cover each one and the checks to make before choosing a stack version.
A supported and administrable cluster
Check the cluster's Kubernetes minor version against the current supported releases before you pick a kube-prometheus or operator version. The project maintains only the three most recent minors, and the version skew policy limits how far kubelets may lag the API server. Confirm your distribution is conformant and that your team can upgrade the control plane and nodes on a schedule.
Cluster-admin access for the initial install
The Prometheus Operator installs custom resource definitions and cluster-scoped RBAC, so the first install needs cluster-admin or an equivalent role. Plan to narrow access afterward so that day-to-day changes to ServiceMonitor, PodMonitor and PrometheusRule objects run under namespace-scoped roles. Record who holds the elevated role and how it is granted and revoked.
Network paths to scrape targets and Alertmanager
Prometheus pods must reach every scrape endpoint on the port and path each monitor declares, and must reach every Alertmanager instance directly rather than through a load balancer. Check NetworkPolicy rules, service mesh sidecar mTLS settings and any cluster egress controls before the first scrape. Test each path with a short-lived pod in the same namespace as Prometheus.
Persistent volumes sized for the chosen retention
Prometheus writes to a local time series database that is not clustered or replicated, so each replica needs its own persistent volume. Size the volume from the ingested sample rate and the retention you set by time or size, which default to 15 days. Confirm that the storage class supports volume expansion and snapshots, and decide who monitors disk growth.
Retention and replica decisions made in advance
Decide retention, replica count and whether long-term storage will use remote write before the first manifest is committed. Replicas improve availability of queries and alerting, but they do not share data, so each replica holds its own copy. Write these choices into the Prometheus custom resource as code so reviewers can compare them with the storage budget.
A named owner for alert routing and on-call
Assign one team to own Alertmanager routes, receivers, silences and inhibition rules, and name the people who receive pages outside working hours. Without a owner, alert rules multiply and nobody decides which alerts page and which go to a channel. Agree how receiver credentials are stored, who rotates them, and how a test alert is sent after each routing change.
How ServiceMonitor and PodMonitor discovery turns into scrape targets
A ServiceMonitor or PodMonitor is how an application team declares its scrape targets without editing the Prometheus configuration file. The Prometheus Operator watches these custom resources, resolves each one against the Kubernetes Services or Pods whose labels it matches, and renders the result into the scrape configuration of the Prometheus instance it manages. The Operator does not collect any metrics itself. Prometheus performs the scraping, and the Operator keeps its configuration current and reloads it when objects change. Platform owners should be able to trace the full chain for any target: the monitor object, the selector it matches, the Service or Pod endpoints behind it, and the resulting job in the generated configuration.
The Prometheus resource decides which monitors are picked up at all. Its serviceMonitorSelector and podMonitorSelector fields are label selectors that filter monitor objects, so a monitor without the expected labels is ignored even when its manifest is valid. Platform teams should agree on a labeling convention, decide who owns it, and document it next to the Prometheus manifest. Confirm how the Operator version you adopt treats an empty selector, since older and newer behavior has differed on whether an empty selector matches everything or nothing. Test that behavior in a non-production cluster before the first team onboards.
Namespace selection controls which teams can add targets to the monitoring estate. The serviceMonitorNamespaceSelector and podMonitorNamespaceSelector fields on the Prometheus resource limit the namespaces where Prometheus looks for monitor objects. Restricting these selectors to namespaces carrying an agreed label lets the platform team decide which namespaces can contribute targets. Kubernetes RBAC still governs who can create monitor objects in each namespace, so the two controls should be designed together. Every new target adds series and scrape load to Prometheus, so admission guardrails and per-target limits on samples per scrape are worth agreeing before teams are given self-service access.
The most common failure is a monitor that matches no Service or Pod labels. The selector finds nothing, no scrape job is produced for it, and the team sees empty dashboards and no alerts without a clear error in its own namespace. Port naming causes a similar problem: the port named in the monitor must match a named port on the Service or Pod, and a mismatch silently yields zero targets. Check the Prometheus targets page, the Operator logs and the generated configuration after each new monitor is applied. Build an alert for targets that disappear, so a silent selector mistake is caught before it hides an outage.
Kubernetes monitoring with the Prometheus Operator: the setup sequence
This sequence takes a Prometheus Operator monitoring stack from target inventory to production in eight steps, with a pass or fail check closing each one. Each check is designed to catch a specific failure, such as targets that never scrape, rules that fail to load, or alerts that reach the wrong receiver. The result is a monitoring setup the team can reproduce, test and promote from version-controlled manifests.
Inventory targets, labels and namespaces
The platform team lists every workload that exposes metrics, the port and path each one uses, the labels that identify its owning team and environment, and the namespaces that will hold monitoring objects. The labels recorded here become the selectors that ServiceMonitor and PodMonitor objects match on, so label mismatches are a common reason a target never appears. Store the inventory in the same repository as the manifests. Pass check: every target has a named owner, a metrics endpoint and a selector label that you can confirm with a Kubernetes label query across the namespaces you plan to scrape.
Install the Operator and kube-prometheus from manifests
The platform team installs the Prometheus Operator and the chosen kube-prometheus stack into a non-production cluster, applying manifests kept in version control through a repeatable process. Retention, replica count and any remote write settings are declared in the Prometheus resource in the same change, so they are code from the first deployment. Set retention explicitly, because the default is 15 days. Pass check: the Prometheus and Alertmanager pods reach ready state, the retention, replica and remote write values appear in the running Prometheus configuration, and those values are unchanged after a pod restart.
Add ServiceMonitor and PodMonitor for one application
Pick one application from the inventory and create a ServiceMonitor or PodMonitor that selects it by label and names its metrics port. Choose an application with a stable metrics endpoint so that any first scrape failure points to the monitor definition, not to the application itself. Confirm that the metrics endpoint is reachable only from the monitoring namespace, using network policy if your cluster supports it. Pass check: the target appears on the Prometheus UI targets page with state up, carries the labels you expected, and shows no scrape errors.
Add PrometheusRule objects and verify evaluation
The team writes PrometheusRule objects for the recording and alerting rules it agreed on, starting with a small set tied to the application from the previous step. Each alerting rule needs a named owner and a runbook reference in its annotations. Validate the rule files with promtool, the Prometheus command-line tool for checking rule syntax, before the change is applied. Pass check: the Rules page in the Prometheus UI lists each rule with no evaluation error, and an alerting rule enters pending or firing only when its controlled test condition is met.
Configure Alertmanager receivers and test routes
The team defines receivers, grouping keys and routes in the Alertmanager configuration and stores it with the other manifests. Receiver credentials go into Kubernetes Secrets, which are stored unencrypted in etcd by default unless encryption at rest is configured for the cluster. If high availability is required, point Prometheus at each Alertmanager instance, because the Alertmanager documentation advises against placing a load balancer in front of them. Send a labeled test alert through each route. Pass check: each test alert reaches its intended receiver once with the grouping and labels designed for it, and no test alert reaches a receiver it should not.
Apply security settings from the security section
The platform team applies the access controls chosen in the security section. These include TLS for Prometheus and Alertmanager endpoints with client certificates where needed and a minimum of TLS 1.2, basic authentication only with bcrypt-hashed passwords and only over TLS, and an authenticating reverse proxy in front of the web UI and API for single sign-on. Leave the admin API disabled unless there is a documented need, and confirm that the enable flags for admin API and lifecycle endpoints are absent from the deployed arguments. Pass check: unauthenticated requests to the UI and API are refused, and the deployed container arguments match the manifests exactly.
Run a restore drill on Prometheus data volumes
Local Prometheus storage is not clustered or replicated, so the team manages its data volumes like a single-node database and backs them up accordingly. Snapshot a volume, restore it into a separate namespace and start a Prometheus instance against the restored data. Before the snapshot, record a fixed set of PromQL queries over a fixed time window, then run the same queries against the restored instance. Write the procedure down and name the operator responsible for it. Pass check: the restored instance returns the same query results for that window, and the restore steps are documented and have been completed by someone other than the author.
Promote identical manifests through the delivery pipeline
The team promotes the same version-controlled manifests to production through its existing delivery pipeline. Only environment values change, such as cluster endpoints, storage classes and references to receiver credentials. Gate the promotion on the earlier checks passing in the non-production cluster, and keep the Prometheus and Operator versions identical to those tested. Run the same test alert procedure after the production rollout. Pass check: a diff between the promoted and tested manifests shows only approved environment values, and production targets, rules and routes match the state recorded in the test cluster.
Which Kubernetes objects map to which Prometheus settings
This table maps each Kubernetes object your team writes to the Prometheus or Alertmanager setting it controls, the owner who should approve changes, and the field to check after a change. The label column shows which labels must stay consistent across teams so that routing, rules and dashboards keep matching. Read each row as a change control entry: before merging a manifest, confirm the approver and check the listed field in the running configuration.
| Kubernetes object | Prometheus or Alertmanager setting it controls | Owner who approves changes | Field to check after a change | Labels that must stay consistent |
|---|---|---|---|---|
| ServiceMonitor | Scrape targets for Services, including endpoints, port names, scrape interval and metric relabeling | Platform team, with the owning application team consulted | Generated scrape job in the Prometheus targets page shows the expected endpoints as up | Namespace, the release label the Prometheus selector matches, and the port name |
| PodMonitor | Scrape targets for Pods that have no Service, such as batch jobs or sidecar metrics endpoints | Platform team, with the owning application team consulted | Targets page lists the expected pods, and no series carry unexpected high-cardinality labels | Namespace, the release label the Prometheus selector matches, and the pod label used for job names |
| PrometheusRule | Recording rules and alerting rules evaluated by Prometheus, including thresholds, for durations and severity | Service owner for the alert logic, with the on-call lead approving severity and routing | Rules page shows the group loaded without errors, and the alert appears in the expected state after a test | severity, team or service label used by Alertmanager routes, and the rule group names used in dashboards |
| Alertmanager configuration | Receivers, routing tree, grouping, inhibition rules, and silences that apply to alerts from Prometheus | Monitoring lead, with the security team reviewing receiver endpoints | Alertmanager status page shows the expected config hash, and a test alert reaches the intended receiver | The team and severity labels that routes match on, and the receiver names referenced by dashboards |
| Prometheus resource (retention and replicas) | Local retention by time or size, replica count, storage volume size, and the Alertmanager endpoints Prometheus sends to | Platform team, with the capacity owner approving storage changes | Runtime flags and storage metrics show the new retention and that each replica is running and sending alerts | External labels that identify each replica and cluster, which remote write systems and dashboards depend on |
| Secret or ConfigMap holding receiver credentials | Credentials for Alertmanager receivers and any basic authentication or TLS material that Prometheus or Alertmanager reads | Security team, with the on-call lead for receiver access | Pod restarts or reloads after the change, and a test notification is delivered with the new credential | Secret names referenced in the Alertmanager configuration, and the namespace the pods read them from |
How PrometheusRule alerts reach people through Alertmanager
Prometheus evaluates the alert rules in your PrometheusRule objects and sends every firing alert to Alertmanager. Alertmanager is the part that decides who hears about an alert and when. It groups related alerts into a single notification, applies inhibition rules so that a symptom alert is suppressed while a broader cause alert is already firing, and honors silences that operators create for planned work. Prometheus does not notify anyone directly, so a team that reviews its rules but not its Alertmanager routing has only checked half of the path from a metric to a person.
Alertmanager can run as a cluster for high availability. Its instances find each other through the --cluster flags, which configure the listen address, the peers to join and gossip settings, and they share notification and silence state so that a restarted instance does not resend alerts already handled elsewhere. The documented rule for this design is that Prometheus should be configured with every Alertmanager instance, not with a load balancer in front of them. Each instance receives every alert, and the cluster deduplicates notifications. Placing a load balancer in front of Alertmanager can split alerts across instances that do not share the same view, so confirm this wiring in your deployment rather than assume the Operator produces it.
The Prometheus Operator manages this wiring through an Alertmanager custom resource and the Prometheus resource that references it. Before you rely on either, confirm in the documentation for the Operator version you intend to run how cluster settings, replica counts, peer discovery and the Alertmanager endpoints given to Prometheus are exposed as fields. Field names and defaults have changed across releases, and a configuration that looks correct in a values file may not produce the cluster you expect. Ask your platform team to check the rendered configuration in the running cluster, not only the manifest.
Test routing before any alert is trusted. Send synthetic alerts with labels that match each receiver, and confirm that the right team gets the notification through the intended channel with the expected grouping. Check that inhibition rules suppress only the alerts they are meant to suppress, and that a silence created on one Alertmanager instance is visible on the others. Create a silence in a test namespace, confirm it expires as intended, and record who is allowed to create silences in production. Repeat these checks after each change to routing or receivers, because a mistake in a label matcher can send critical alerts to a channel nobody watches.
Plan for failures in the alert path itself. Alertmanager cannot report that it has stopped, so a common safeguard is a rule that always fires and is routed to an external heartbeat service that pages when it goes silent. Decide who owns the receiver credentials, where they are stored, and how they rotate. Finally, document the expected behavior of each instance restart so that on-call engineers know whether a notification may repeat after a rolling update. These decisions are easier to make at design time than during the first incident.
Security settings to apply to the Prometheus and Alertmanager endpoints
Prometheus and Alertmanager expose HTTP endpoints, admin interfaces and configuration objects, and each needs a deliberate setting before go-live. Work through these controls in order, then check the monitoring namespace against Pod Security Standards and the cluster against the CIS Kubernetes Benchmark, which serves as a reference for the cluster as a whole.
TLS and client certificate authentication
Serve the Prometheus and Alertmanager web endpoints over TLS with client certificate authentication, and set TLS 1.2 as the minimum version, as the Prometheus security model describes. Basic authentication sends credentials in cleartext when TLS is absent, so it should never be the only control on an endpoint. Prometheus stores basic authentication passwords as bcrypt hashes, but the hash does not protect credentials on the wire. Check the web configuration for each Prometheus and Alertmanager instance, and confirm that every scraper and Alertmanager peer presents a valid client certificate.
Authenticating reverse proxy for user access
Prometheus has no built-in single sign-on, so user access to the web UI and API normally goes through an authenticating reverse proxy connected to your identity provider. The Prometheus security model also notes there is no built-in CSRF protection on mutating endpoints, so the proxy and network policy must limit which clients and origins reach them. Confirm that the proxy enforces authentication on every path, that Grafana and other tools use dedicated service accounts, and that network policy blocks direct access to the Prometheus pods from application namespaces.
Admin and lifecycle endpoints kept disabled
Prometheus disables its admin HTTP APIs by default. The --web.enable-admin-api flag enables destructive operations such as deleting series, and --web.enable-lifecycle enables reload and shutdown endpoints. Leave both disabled unless a documented runbook requires them. Because the Prometheus Operator generates pod arguments from the custom resources, check the arguments on the running Prometheus and Alertmanager pods after each change, and restrict any reload trigger to the pipeline or operators who need it.
RBAC for monitor and rule objects
ServiceMonitor, PodMonitor and PrometheusRule objects decide what Prometheus scrapes and what it alerts on, so editing them changes your monitoring posture. Use Kubernetes RBAC to limit create, update and patch rights on these resources to the platform team, and grant application teams read access or a narrow namespace role only where they need it. Review any ClusterRoleBinding that covers these resources, and confirm the Prometheus service account holds only the list and watch rights that discovery requires.
Receiver credentials in Kubernetes Secrets
Alertmanager receivers need credentials for email, chat and paging systems, and these are typically referenced from Kubernetes Secrets. Secrets are stored unencrypted in etcd by default, so confirm with the cluster team that encryption at rest is configured for Secret resources and ask for the encryption configuration as evidence. Restrict list and watch access to Secrets in the monitoring namespace to the smallest set of identities. Where policy requires it, load receiver credentials from an external store through the Secrets Store CSI Driver.
Exporter targets taken from URL parameters
Exporters such as SNMP and Blackbox take their targets from URL parameters. The Prometheus security model warns that anyone with HTTP access to these exporters can make them send requests to arbitrary endpoints, including internal services the exporter's network position can reach. Restrict access to exporter services with network policy and authentication, limit which Prometheus jobs can call them, and define allowed modules and targets in the exporter configuration before any probe job runs in a shared namespace.
Setting retention and replicas as code without assuming Prometheus is clustered
Retention and replica counts belong in version-controlled manifests, but they only make sense once you know what Prometheus actually does with data. Prometheus writes samples to a local time series database organized into two-hour blocks, with a write-ahead log protecting data that has not yet been compacted into a block. The default retention is 15 days, and you can configure retention by time, by size or both. Before you commit values to Git, confirm which of these limits your chosen Prometheus Operator version exposes on the Prometheus custom resource and how it applies them when both are set.
Replicas need the most careful reading. The Prometheus storage documentation states that local storage is not clustered or replicated and should be managed like any other single-node database. Setting a replica count in the Operator therefore starts more Prometheus servers, each with its own local storage. It does not create a shared dataset, automatic data repair or the consistency guarantees you would expect from a database cluster. Confirm in your version how the Operator assigns external labels to each replica, how queries and alerts handle the duplicate series that result, and what happens to a replica's history if its volume is lost.
Long-term storage is a separate decision. Prometheus can send samples to external systems through remote write and read from them through remote read, so history can outlive the local retention window. Choose that external system before you finalize local retention, because it determines who operates the history, how long it is kept and whether alert rules still depend on local data during an outage of the remote path. Ask the vendor or project behind the remote system how it handles out-of-order and duplicate samples from replicated senders, and record the answer alongside the manifests.
Volume sizing follows directly from the retention values you set. Since retention can end on time or on size, the volume must hold the larger of the two outcomes plus headroom for compaction and the write-ahead log. Make the storage class, volume size and reclaim policy explicit in the same repository as the retention settings, and review them whenever either changes. Then test the design: restart a replica, delete its pod, and confirm that the data and the external labels survive as expected in your environment. Record the results so a later retention change is a reviewed edit, not a surprise on a full disk.
Common failures in Prometheus Operator monitoring and how to fix them
This table lists the failures teams most often hit when running Prometheus through the Prometheus Operator, with the probable cause, the check to run and the fix for each. Work down the rows in the order your monitoring pipeline runs: discovery, scraping, rules, alert delivery, storage and replication. The main takeaway is that most silent failures trace back to a label mismatch, a missing secret reference or a volume that filled without an alert.
| Failure | Probable cause and check | Fix |
|---|---|---|
| ServiceMonitor matches no targets | The selector in the ServiceMonitor does not match the labels on the Service, or the Service port name does not match the endpoint port in the ServiceMonitor. Check the Prometheus targets page and compare labels with kubectl get svc --show-labels. | Align the selector and the named port with the Service. Confirm the Prometheus resource's serviceMonitorSelector and namespace selector also include the ServiceMonitor's namespace. |
| Target is up but returns no expected metrics | The endpoint answers but the application exposes different metric names, or the path or scheme in the endpoint is wrong. Fetch the endpoint from inside the cluster and compare the output with the expected names. | Correct the path, scheme or port in the endpoint definition. Use metricRelabelings to keep or rename series only after confirming the exact metric names the application emits. |
| PodMonitor selects no pods | The pod selector or the named container port does not match the running pods, often because a label was changed in a new Deployment revision. Check the labels on pods with kubectl get pods --show-labels. | Update the selector to the current pod labels and the named port. Add a check to the release pipeline that compares selectors with rendered pod labels. |
| PrometheusRule fails to load | A YAML or PromQL syntax error, or an invalid rule group name, stops the rule from loading. The operator logs and the Prometheus rules page show the error. Validate rules with promtool before applying them. | Fix the syntax and resubmit. Run promtool check rules in CI so a broken rule fails the merge, not production. |
| PrometheusRule loads but never fires | The expression returns no series because of a wrong label name, a metric renamed in a new exporter version, or a for duration that is longer than the expected event. Test the expression in the Prometheus expression browser. | Correct the label matcher or metric name, then add a test in promtool unit tests that asserts the alert fires for sample input. |
| Alertmanager receiver fails silently | The receiver uses an incorrect API key, webhook URL or SMTP password, or the secret it reads is missing or in another namespace. Check Alertmanager logs for notification errors and confirm the secret name and key in the alertmanager configuration. | Recreate the secret in the namespace the Alertmanager resource uses, reference the exact key, and send a test alert through the routing tree. Alert on failed notification counts from Alertmanager metrics. |
| Prometheus pods restart when volumes fill | The persistent volume reaches capacity, the write-ahead log and head blocks cannot be written, and the container exits. Check volume usage and the storage size against the configured retention. | Increase the volume size if the storage class allows expansion, and reduce retention or set a size-based retention limit below the volume size. Alert on free space before it reaches the limit. |
| Data disappears earlier than expected | Retention is set by time or size, and the size limit is reached before the time limit, so the oldest blocks are deleted first. Default time retention is 15 days unless changed. Compare the retention settings in the Prometheus resource with the actual oldest sample. | Set retention in both time and size deliberately, sized to the volume. For longer history, configure remote write to an external long-term storage system, since local storage is not replicated. |
| Alerts duplicate because Prometheus misses an Alertmanager | Prometheus is not configured with every Alertmanager instance, or it is pointed at a load balancer. Check the Alertmanager discovery status on the Prometheus status page and confirm the instance list includes each replica. | Let the operator discover all Alertmanager replicas through the Alertmanager resource, and do not place a load balancer in front of them. Deduplication in the cluster depends on every replica receiving the same alerts. |
Kubernetes monitoring with the Prometheus Operator: questions buyers ask before hiring help
How much does it cost to hire help for Prometheus Operator monitoring?
Prometheus has no license fee because the project is licensed under Apache-2.0, so the cost comes from four places: infrastructure, storage, implementation and the people who run it. Implementation is usually a one-time cost that covers design, Operator and kube-prometheus setup, ServiceMonitor and PodMonitor discovery, PrometheusRule alerts and Alertmanager routing. Running costs recur: compute and disks for Prometheus and Alertmanager, any long-term storage, version upgrades, and staff time for on-call and rule changes. DeployOpen provides a quote that follows a written scope, so ask for each cost line to be listed separately, with the one-time items, the recurring items and the party responsible for each.
How long does a Prometheus Operator monitoring project take?
Duration depends on the scope agreed in writing, so there is no single figure that applies to every cluster. The main variables are the number of clusters, how many teams need monitors added, how complex the alert routing is, whether retention and remote storage must be designed, whether a security review is required, and whether the cluster already runs a compatible stack. One cluster with an existing kube-prometheus install and simple routing will move faster than several clusters that need new receivers and a formal security review. Ask for a schedule in the scope document that shows milestones, review points and the dependencies on your team, such as access approvals and receiver credentials.
Which factors make a Prometheus Operator monitoring project larger?
The largest factors are the number of clusters, the number of teams adding monitors, the complexity of alert routing, and the retention and remote storage design. Each additional cluster adds Operator installs, RBAC and dashboards to verify. Each team that adds ServiceMonitor or PodMonitor objects creates label and naming decisions that need review. Alert routing grows with the number of receivers, escalation paths, inhibition rules and silences. Retention matters because Prometheus keeps 15 days of local data by default, and local storage is not clustered or replicated, so longer retention or remote write changes both the storage and the design. A security review adds time when access, TLS and exporter exposure must be assessed. A cluster that already runs a compatible stack needs less rebuilding. Confirm each factor in your environment before the scope is written.
What must our team provide before a monitoring engagement starts?
Before work starts, the client normally provides cluster access for the agreed environments, a named owner who can decide on scope, labels, alert thresholds and routing, the receiver endpoints for paging, chat and ticketing, and any existing alert runbooks. Access should be limited to the clusters and namespaces in the agreement, with a clear path for installing the Operator and its custom resource definitions. Receiver endpoints need the credentials and TLS certificates that Alertmanager will use. Existing runbooks show which alerts matter and who acts on them, so they shape the PrometheusRule set. If a runbook does not exist, the scope should name who writes it. Confirm these items during scoping, because missing receivers or an unnamed owner usually hold up the schedule.
Can our own platform team set up Prometheus Operator monitoring alone?
An internal team can do this alone if it already runs Kubernetes in production and has people who can work across the stack. The skills needed include Kubernetes administration (custom resources, RBAC, namespaces and rollouts), PromQL for writing and testing recording and alert rules, Alertmanager routing and receiver configuration, storage and retention planning, TLS and secret handling, and upgrade planning that respects the Kubernetes version skew policy. Someone must also own on-call practice, since alert quality depends on it. A practical test is whether two or three people can each explain the design and rebuild it from code. If the team lacks PromQL or Alertmanager experience, a partner can shorten the learning curve, and the team keeps ownership of the configuration as code.
Does DeployOpen issue audit reports for our monitoring setup?
No. DeployOpen does not issue audit reports, attestations or certifications. Independent auditors and certification bodies issue those. DeployOpen can prepare the systems, controls and evidence that an assessment examines, such as configuration as code, documented alert ownership, access records, retention settings and change history for the Prometheus Operator deployment. Ask your auditor which evidence they expect before the build starts, so the monitoring design produces it. Where your framework refers to a specific control, the scope should name the control ID and the evidence that supports it. Your auditor decides whether the evidence is sufficient.
How do we secure Prometheus and Alertmanager endpoints in Kubernetes?
Start with the endpoints. Prometheus supports TLS and client certificate authentication with TLS 1.2 as the minimum, and HTTP basic authentication with bcrypt-hashed passwords. Basic authentication without TLS exposes credentials in cleartext, so pair it with TLS. Prometheus has no built-in single sign-on, so access to the web UI and API is usually placed behind an authenticating reverse proxy. Admin APIs are disabled by default, and lifecycle endpoints should stay off unless a reload workflow needs them. Exporters such as SNMP and Blackbox take targets from URL parameters, so anyone with HTTP access to them can make them send requests to arbitrary endpoints. If receiver credentials live in Kubernetes Secrets, those are stored unencrypted in etcd by default, so encryption at rest and tight RBAC on Secrets belong in the review. Confirm each control in your cluster before go-live.
What usually goes wrong with Prometheus Operator alert routing?
Most failures come from design gaps. A single Prometheus instance holds the only copy of its recent data, and local storage is not clustered or replicated, so losing that node means losing history unless the design accounts for it. Alertmanager must run as a cluster where high availability matters, and Prometheus should point at every Alertmanager instance rather than a load balancer. Retention left at the 15-day default may not match incident review or audit needs. Selectors on ServiceMonitor and PodMonitor objects can match nothing after a label change, which leaves a target unmonitored without an obvious error. Upgrades need a plan too: Prometheus minor releases stop receiving bug fixes after six weeks, so stable operations usually follow a long-term support line. Ask for these behaviors to be verified against the versions you run.
How DeployOpen scopes and delivers Kubernetes monitoring with the Prometheus Operator
DeployOpen plans, implements, documents and supports Prometheus on Kubernetes inside infrastructure you control, using the Prometheus Operator and kube-prometheus as the configuration model. The work is agreed as a fixed scope in writing before it starts, and ongoing support is available as a retainer once the monitoring stack is handed over.
How we scope
- The written scope lists which clusters, namespaces and workloads get ServiceMonitor or PodMonitor coverage.
- Retention period, storage sizing and replica count are agreed as settings in version-controlled manifests.
- Alertmanager receivers, routing trees and PrometheusRule ownership are defined before any alert is built.
How we build
- Operator, kube-prometheus and Prometheus custom resources are deployed and kept in your Git repository.
- Discovery selectors are tested against real labels so only intended targets are scraped in each cluster.
- Alert rules are written in PrometheusRule objects and checked with promtool before they reach the cluster.
How we hand over
- Your team receives the manifests, runbooks for alert triage and a record of every retention and replica decision.
- Restore of Prometheus data is walked through with your operators, since local storage is not replicated by default.
- Walkthroughs cover adding a new ServiceMonitor, silencing an alert and reviewing an Alertmanager configuration change.
How we support
- Ongoing support covers Prometheus Operator and kube-prometheus version upgrades planned against Kubernetes version support windows.
- Retainers can include reviews of alert noise, scrape health and storage growth on a schedule you agree.
- Changes to the monitoring stack follow the same written change process as the rest of your platform.
