Kubernetes monitoring with Grafana: Setup Guide

How Grafana connects to Kubernetes monitoring data

Operators running Kubernetes need cluster and workload metrics and container logs in one place, so that a node pressure event, a crash-looping pod and the log lines around it can be read side by side. Grafana is the visualization and alerting layer in this design, not the store. Metrics are kept in a Prometheus-compatible store and logs in Loki, and Grafana queries both through its built-in data sources: the Prometheus data source for metrics and the Loki data source for logs. Grafana itself keeps only its configuration and dashboards, which is why its database choice and its data source choices are separate decisions. The topic sources cover Prometheus, Alertmanager and Loki, which means the connection is made through these general-purpose data sources rather than a dedicated Kubernetes plugin. Confirm this with your own team before design starts, because it changes what you build. Plan the Prometheus data source URL, the Loki data source URL and any tenant header you need, then check that each data source returns results from a test query before you build dashboards on it. For Loki with auth_enabled set to true, the tenant is identified with the X-Scope-OrgID header, so the data source configuration must match the tenant the log shipper writes to.

Alerting works through the same chain. Prometheus evaluates alert rules and sends firing alerts to Alertmanager, which groups, inhibits and silences them. Grafana Alerting can also route notifications, so decide early which system owns each alert, because duplicate notifications from two places are a common consequence of leaving that unclear. Prometheus should point at every Alertmanager instance rather than at a load balancer. Also decide who can edit data source connections, since a Grafana user with broad access can reach every store Grafana is connected to. Ask your implementation partner or platform team to document the query path, the owner of each data source and the test queries used to verify it.

What you need before collecting Kubernetes data

Monitoring a cluster with self-hosted Grafana depends on decisions made before the first scrape: which Kubernetes version you run, which Prometheus and Loki versions will store the data, and how the Grafana instance is built. Each card below is a prerequisite to confirm with your platform team before any collector is deployed.

Kubernetes version and support window

Kubernetes maintains only the three most recent minor releases, so the cluster version sets how long your monitoring agents, kube-state-metrics and dashboards can rely on a stable API. As of September 2026 the supported lines were 1.37, 1.36 and 1.35, with end-of-life dates in 2027. Confirm your current minor version, its end-of-life date and the planned upgrade window, because a monitoring change made during an upgrade is harder to validate. Check the version skew rules before upgrading control plane and nodes separately.

Prometheus and Loki versions

Pin the Prometheus and Loki versions before writing scrape configs and retention rules, because each project ships on its own cadence. Prometheus minor releases stop receiving bug fixes six weeks after release, while its LTS line gets one year of fixes, so an LTS line is usually the safer choice for a long-lived cluster. Loki's simple scalable mode is being deprecated and removed in Loki 4.0, so plan the deployment mode around that date rather than around today's default.

Grafana edition and production database

Grafana stores its configuration in SQLite by default, and the installation docs state that SQLite does not scale for production. Choose MySQL 8.0 or later, or PostgreSQL 12 or later, before go-live, and decide whether you need high availability with two or more active-active servers. Confirm early whether you need SAML, team sync or SCIM, because those features are available only in Grafana Enterprise and Grafana Cloud, which changes the identity design for operators and dashboard owners.

Object storage for Loki

Loki keeps compressed log chunks and its index in object storage, so the bucket is a production dependency that needs its own owner, encryption and lifecycle rules. Amazon S3, Google Cloud Storage and Azure Blob Storage are the recommended production stores, while MinIO and local filesystems are supported but not typically recommended for production. Retention is disabled by default, which means logs accumulate indefinitely until you set retention_enabled and define rules per tenant or stream, so confirm the retention policy with the data owners before ingestion starts.

Network paths from clusters to central stores

Each cluster needs outbound paths to the central Prometheus-compatible store and Loki endpoint, and the Grafana instance needs paths to both. Prometheus pulls metrics from targets inside the cluster, so decide whether scraping stays local with remote write to a central store, or whether a central Prometheus reaches across network boundaries. Confirm firewall rules, TLS certificates, DNS names and egress limits for each cluster, and put Loki behind an authenticating reverse proxy, because Loki has no built-in authentication layer.

Control plane and etcd access plan

Monitoring agents need read access to the API server and kubelet endpoints, and the cluster already has etcd, control plane nodes and credentials that must stay protected. Decide who may create ServiceAccounts and ClusterRoles for collectors, and restrict access to Secrets with least-privilege RBAC, because Secrets are stored unencrypted in etcd by default. Confirm that etcd snapshot backups exist and are tested, and that the monitoring design does not require direct etcd access, since that access would expand the attack surface without adding much visibility.

Configuration sequence for the first cluster

This sequence brings up the monitoring stack in dependency order, so each component is verified before the next one depends on it. The result is a self-hosted Grafana that queries cluster metrics from Prometheus, routes alerts through Alertmanager, and shows container logs from Loki.

Deploy Grafana on a production database
Install Grafana with a MySQL 8.0 or later or PostgreSQL 12 or later database for configuration, not the SQLite default, because the installation docs say SQLite does not scale for production. Your platform team or DeployOpen configures the Helm chart or container image, the database connection, and the admin sign-in path. The step is done when you can sign in, the health endpoint responds, and a test change survives a Grafana pod restart.
Deploy Prometheus with cluster scrape targets
Deploy Prometheus in the cluster with scrape jobs for the API server, kubelets, cAdvisor, kube-state-metrics, and node exporters, using Kubernetes service discovery. Confirm the scrape targets are up in the Prometheus targets view and that workload pods carrying your scrape annotations or ServiceMonitor objects appear. Add a data source in Grafana pointing at Prometheus, then run a simple query such as a pod count. The check passes when the query returns series for every node and namespace you expect.
Configure Alertmanager routing
Deploy Alertmanager with grouping, inhibition and receiver settings agreed with the on-call team, and point Prometheus at every Alertmanager instance. If you run more than one instance, enable the cluster flags so silences and notifications are shared. Send a test alert from a dummy rule and confirm it reaches the receiver once, not once per replica. Grafana Alerting can use an external Alertmanager, so decide which system owns routing before you write rules in both places.
Size and deploy Loki with object storage
Choose the Loki deployment mode by expected daily log volume, using the monolithic mode for smaller volumes and simple scalable or microservices modes as volume grows, as described in the Loki deployment modes docs. Back the chunks and index with a supported object store such as Amazon S3, Google Cloud Storage, or Azure Blob Storage. Enable retention deliberately, because retention is disabled by default and logs are kept indefinitely until it is set. The check is that the Loki ready endpoint reports ready and a test write returns successfully.
Ship container logs with Grafana Alloy
Deploy Grafana Alloy as a DaemonSet to collect container logs from every node, attach Kubernetes labels such as namespace, pod, and container, and push them to Loki. Keep label cardinality low, since Loki indexes labels rather than log content. If you run multiple tenants, set the X-Scope-OrgID header on each request, and remember Loki has no built-in authentication, so place an authenticating reverse proxy in front of it. Add Loki as a data source in Grafana. The check is a log line from a known pod visible in Explore.
Provision data sources and dashboards as code
Define the Prometheus and Loki data sources, folders, and dashboards in YAML or JSON provisioning files loaded at Grafana startup, so a rebuilt instance matches the last reviewed configuration. Store the files in version control and deploy them through your normal pipeline. Some teams also use Terraform or Git Sync, which the provisioning docs list as further options. The check is that you can delete the test data source in a throwaway instance, restart Grafana, and see it return from the files.

Choosing the metrics store and Loki deployment mode

The first decision is whether a single Prometheus server is enough or whether you need a store that outlives one cluster. A single Prometheus server keeps its samples in a local time series database, and that storage is not clustered or replicated, so it should be managed like any other single-node database. Its default retention is 15 days, which is often shorter than the window a capacity review or an incident post-mortem needs. Confirm your retention requirement with the teams that use the data, then decide whether remote write to long-term storage is needed, or whether a Prometheus-compatible store that serves several clusters from one query endpoint fits better. Each option adds a component you must back up, upgrade and size, so write that cost into the plan before choosing.

Cardinality is the constraint that shapes both choices. Every unique combination of label values creates a separate time series, so labels such as pod name, request path or user ID multiply storage and query cost quickly. Prometheus documents that it values reliability over precision, so a small gap in a sample is acceptable for dashboards and alerts, but it is not suitable where every event must be counted exactly. Keep labels to bounded sets such as namespace, workload and cluster, and check series counts from the Prometheus status pages before adding a new scrape job. For logs, the same discipline applies to Loki labels. Its monolithic mode runs as a single binary and suits up to about 20 GB of logs per day. Simple scalable mode, which separates read, write and backend targets and approaches 1 TB per day, is being deprecated and will be removed in Loki 4.0. For new deployments, plan around microservices or a supported successor path, and confirm the current guidance in the Loki deployment modes documentation before committing.

Single cluster or multi-cluster views in one Grafana

A single cluster with one Prometheus-compatible store is the simplest setup. Grafana queries one data source, dashboards need no cluster selection logic beyond a single variable, and alert rules evaluate against local data. Most of the design effort goes into scrape configuration, retention and the dashboards operators open first, not into topology.

One Prometheus data source: Point Grafana at the Prometheus server and keep the data source name stable, because dashboards and alert rules refer to it.
Local retention limits: Prometheus keeps data locally for a configured period, and local storage is not clustered or replicated, so plan backups and size retention for the cluster you have.
Optional cluster label: Add a cluster label even with one cluster, so a later second cluster does not require relabeling existing series.
Alerts stay local: Alert rules and Alertmanager routing run against the same store that Grafana reads.

Before committing, confirm with your platform team whether a second cluster is planned within your support window, since adding the label later changes existing queries.

Several clusters with a Prometheus server in each is a common pattern when clusters are operated by different teams or sit in separate networks. Each Prometheus scrapes only its own cluster, and Grafana adds one Prometheus data source per cluster. The price is that cross-cluster questions require querying several data sources, and no single store holds the full history.

External labels: Set external_labels in each Prometheus configuration with a unique cluster value and, where useful, a region or environment value. These labels are attached to series sent to remote systems and make the origin of each series clear.
Cluster label convention: Agree on lowercase, stable cluster names before rollout and use the same key everywhere, including alert labels and dashboard variables.
Dashboard variables per data source: Build dashboards that select a data source variable, so one template works for every cluster.
Alerts per cluster: Each Prometheus evaluates its own rules, so an outage of one cluster's Prometheus silences only its alerts. Monitor each Prometheus for being up.

Confirm how your teams will name clusters and whether any cross-cluster panel is needed, because that decides whether this tab is enough.

Several clusters with a long-term store place a query layer over multiple Prometheus servers, so Grafana can present one global view. Remote write sends samples from each Prometheus to external storage, and Grafana reads from that store as a single Prometheus-compatible source. The sibling topic on Thanos and Mimir covers their architecture, components and configuration, so this tab only covers what changes around Grafana.

Consistent external labels: Global queries depend on the cluster label being identical in naming and format across all Prometheus servers, so validate it before enabling remote write.
Deduplication for replicas: If Prometheus servers run in high availability pairs, the long-term store needs a replica label to merge duplicate series. Define that label in the same convention.
Single alerting point: Decide whether rules run in each cluster for locality or centrally against the global store, and document the choice, because central rules depend on the store's availability.
Retention outside the cluster: Long-term retention moves to the external store, so the storage budget and backup plan change.

Ask your platform team which store is in scope and confirm how the Grafana data source will authenticate to it before dashboards are built.

Alerting is where the three topologies differ most in operational risk. The question is which component evaluates a rule, which Alertmanager receives the alert, and what happens when a link in that chain fails. Grafana Alerting and external Alertmanagers follow the same logic, and the sibling topic on Grafana Alerting covers routing details.

Local evaluation: Per-cluster Prometheus rules keep alerts working during a cross-cluster network failure, but each rule set must be maintained for every cluster.
Central evaluation: Rules against the global store are easier to keep consistent, but a store outage stops every alert at once.
Cluster label in routing: Route notifications using the cluster label so the on-call team sees which cluster raised an alert.
Duplicate notifications: Replicated Prometheus or Grafana instances can send the same alert twice unless deduplication and HA settings are configured.

Run a test alert from each cluster and from the global view before go-live to confirm routing reaches the correct team.

Mapping Kubernetes identity, labels and log streams to Grafana

This table compares how Kubernetes objects, access roles and tenant boundaries map onto Prometheus labels, Loki stream labels, Grafana roles and Loki tenant IDs. Read each row as a decision: the label or identity choice that keeps queries fast and access scoped. The main takeaway is that label cardinality and tenant boundaries are hard to change later, so settle them before the first dashboard is built.

Mapping	What to map	Design choice and watch-outs
Namespace and workload labels	Namespace, workload name, app label from pod metadata	Keep namespace and app as stable labels; avoid pod name or container ID as a label, since each new pod creates a new series.
Node and service labels	Node name, service name, cluster label	Add a cluster label to every series from every cluster so multi-cluster queries can filter cleanly. Confirm relabeling rules with your scrape configuration.
Loki stream labels	Namespace, pod, container, app	Loki indexes labels only, so use a small set of low-cardinality stream labels and keep log content in the line, not in labels.
Kubernetes RBAC to Grafana roles	ClusterRole and namespace RoleBinding groups	Grafana roles are not generated from RBAC automatically. Map each group to a Grafana role and confirm the mapping through the identity provider you use.
Group mapping in open source	OAuth, Entra ID, Okta, GitHub, GitLab, Google, LDAP, auth proxy or JWT	SAML, team sync and SCIM are Enterprise or Cloud features. Plan group claims through a supported open-source option and confirm claim names with your provider.
Loki tenant IDs	X-Scope-OrgID header per cluster or business unit	Enable multi-tenancy with auth_enabled and assign one tenant per cluster or unit. An authenticating reverse proxy must set the header, since Loki has no built-in authentication.
Data source scoping	Grafana data source per tenant or cluster	Give each team a data source that sends only its tenant header. Confirm that dashboards cannot switch tenants through shared data sources.

Dashboards and alerts operators need first

Operators need a small set of views before they build anything else. The cards below cover the starting views and alerts that show whether the control plane, workloads, storage and logs are healthy, and they connect back to the Prometheus metrics and Loki logs described elsewhere on this page. Treat each one as a starting point to adapt, not a finished dashboard.

Node and control plane health

Start with node readiness, kubelet availability and control plane component status, including kube-apiserver, kube-controller-manager and kube-scheduler. Alert on nodes that stop reporting Ready and on control plane endpoints that stop responding, because every other view depends on them. On stacked or external etcd topologies, confirm which control plane targets your scrape configuration actually reaches, since a missing target can look like a healthy cluster.

Workload restarts and readiness

Chart container restart counts, pods stuck in pending or not-ready states, and deployments with fewer available replicas than desired. A rising restart count on one workload is often the first sign of a bad release or a failing dependency. Group these panels by namespace so the owning team can act without searching the whole cluster.

Resource saturation per namespace

Compare CPU and memory usage against resource requests and limits for each namespace. Namespaces that run near their limits, or whose workloads have no requests set, make scheduling and eviction behavior hard to predict. Use this view to decide where capacity conversations should start, and check that the metric names match your kube-state-metrics and cAdvisor versions.

Persistent volume and etcd health

Track persistent volume capacity and claim status alongside etcd leader changes, commit latency and database size. Kubernetes stores Secrets and cluster state in etcd, so slow or failing etcd members affect the whole cluster. Confirm that etcd metrics are scraped from every member and that the database size alert sits well below the quota your etcd deployment uses.

Log error rates per namespace

Use Loki and LogQL to count error-level log lines per namespace and per workload over time, then link that panel to the restart and saturation views. Label design matters here: namespace, pod and container labels should come from the collector that ships logs, and high-cardinality labels should stay out of the index. Confirm log retention in Loki, since retention is disabled by default and logs are otherwise kept indefinitely.

Alerts routed through Alertmanager

Define alert rules in Prometheus and send them to Alertmanager, which handles grouping, inhibition and silences. Prometheus should point at every Alertmanager instance rather than a load balancer, and Alertmanager needs its cluster settings if you run it for high availability. Test each route end to end, and confirm that the receivers and escalation paths match your on-call model before you rely on them.

Common failures in Kubernetes monitoring pipelines

This table lists the failures most often seen when Kubernetes metrics and logs flow into self-hosted Grafana, with the likely cause and the first check to run for each. Read it as a triage order: confirm the collection path before tuning storage, and confirm tenancy and retention settings before assuming data is lost.

Failure	Likely cause	First check
Targets missing from Prometheus	Service discovery selectors or scrape annotations do not match the workload labels, or RBAC blocks discovery	Open the Prometheus targets page and compare discovered labels with the expected selectors
Cardinality blow-up in the metrics store	Unbounded label values such as pod names, request IDs or user IDs added to metrics	Query the active series count and list the highest-cardinality metric names before changing retention
Loki rejects log pushes	auth_enabled is true but the shipper does not send the X-Scope-OrgID header, or no authenticating proxy sets it	Send a test push with and without the tenant header and read the HTTP status and error body
Loki retention never deletes data	Retention is disabled by default, so the compactor keeps logs until retention_enabled is set	Confirm retention_enabled in the compactor configuration and check compactor logs for retention runs
Duplicate alert notifications in HA Grafana	Alerting in two or more active Grafana servers without the extra high availability configuration the docs describe	Confirm each Grafana node shares the same database and alerting HA settings, then count notifications per alert
Grafana Live features behave inconsistently	Grafana Live has documented limitations in high availability setups behind a load balancer	Test live panels with sessions pinned to different nodes and decide whether live updates are needed at all
Dashboards stop loading after a Grafana change	Database backend or auth tokens not shared across nodes, so sessions fail when routed to another node	Check that all nodes use the same MySQL or PostgreSQL database and that users can reach any node

Questions teams ask before building Kubernetes monitoring

What drives the cost and effort of Kubernetes monitoring with Grafana?

Storage, retention, label cardinality, the number of clusters and the skills of the team running it drive most of the cost. Metrics and logs both grow with cluster size and workload churn, so storage and object store spend recur every month. Prometheus is licensed under Apache-2.0 with no paid enterprise edition, so the cost is infrastructure, storage, implementation and the people who run it. Loki's retention is disabled by default, which means logs are kept indefinitely until you set it, so retention is a cost decision to make early. Ask any quote to separate one-time build work from recurring infrastructure and support, and to state the retention periods and expected series counts it assumes.

How long does a Kubernetes monitoring build take?

Duration depends on scope: how many clusters you bring in, whether logs go to Loki as well as metrics to Prometheus, how many dashboards and alert routes you need, and whether the store must serve several clusters from one view. A single cluster with metrics, a small set of dashboards and basic alerts is the smallest scope. Adding Loki with object storage, identity integration and multi-cluster views widens it. DeployOpen does not publish durations; a fixed-scope written plan sets the timeline, so ask for milestones tied to clusters, data sources and acceptance checks.

What must our team provide before the build starts?

Plan to provide the following before work begins:

Read access to the clusters, with a kubeconfig or service account scoped to the monitoring namespaces
An object storage bucket for Loki if logs are in scope, since Loki's recommended production stores include Amazon S3, Google Cloud Storage and Azure Blob Storage
An identity provider for Grafana sign-in, such as Entra ID, Okta or Google OAuth, which all editions support
On-call routing and contact points for Alertmanager, including who receives which severity
A decision on which Kubernetes distribution and version each cluster runs

Confirm these with your platform and security teams early, because access and identity approvals often set the start date.

Can our in-house team build and run this without a partner?

Yes, many teams do, and the open-source components are free to use. The question is capacity and skills: the team needs Kubernetes operations, Prometheus and PromQL, Loki storage and retention, Grafana administration and alert design, plus someone who owns upgrades. Prometheus local storage is not clustered or replicated and should be managed like any other single-node database, so high availability is design work your team must own. A partner fits when the build must land on a fixed date or when your team lacks these skills. Either way, write down who will answer pages at 2 a.m. and who performs each upgrade before choosing.

Does Grafana Cloud change the trade-off for Kubernetes monitoring?

It changes who runs the stack and what you pay for. Grafana Cloud has a free tier, a Pro tier starting with a 19 US dollar monthly platform fee plus usage, and an Enterprise tier with a minimum commitment of 25,000 US dollars per year, according to the Grafana pricing page. A managed service shifts storage operations and upgrades to the vendor, but usage charges scale with data volume and you still own cluster-side collection and alert design. Self-hosting keeps data inside your own infrastructure, which matters for data residency. Compare total cost by modeling your metric series and log volume against the pricing page, and ask any partner quote to state both options side by side.

How does Kubernetes version support affect the monitoring plan?

Kubernetes releases come about three times a year, and the project maintains the three most recent minor releases, so your monitoring must keep working as the API and cluster change. The version skew policy requires kube-apiserver to be upgraded one minor version at a time and allows kubelets up to three minor versions older. Plan collection agents and exporters for the versions you run now and for the next upgrade. Prometheus minor releases get bug fixes for only about six weeks, so choose a long-term support line where you can. Confirm the Kubernetes version and Prometheus or Loki release for each cluster in writing, and ask the quote to include an upgrade procedure for the monitoring stack.

How to confirm the monitoring works before handover

Handover should rest on checks you can run and repeat, not on a dashboard that looks populated. Start with Prometheus scrape target health: every node exporter, kube-state-metrics endpoint, and application target your design lists should show as up in the targets view, and any target that is missing should be traced to its service discovery labels or static configuration. Then run a query for each cluster that returns current values for node CPU, memory, pod restarts and the control plane, and confirm each cluster label appears. A cluster that is missing from a multi-cluster view is a failed check, even if the other clusters look healthy. Your team should confirm the exact metric names and label set against the exporters and version of Prometheus it deploys.

Logs and alerts need their own checks. In Grafana, run a Loki query filtered by namespace for at least one workload in each namespace the design covers, and confirm that log lines are recent and carry the expected labels, including the X-Scope-OrgID tenant if multi-tenancy is enabled. For alerting, trigger a test alert deliberately, for example with a rule that fires on a non-production workload, and follow it from Prometheus through Alertmanager to the contact point your on-call team actually uses. Check grouping, silencing and resolution notices as well as the first notification. In a high-availability setup, stop one Grafana server behind the load balancer and confirm sessions continue, dashboards load and alerts still notify exactly once, since the Grafana HA documentation flags duplicate notifications as a risk. Finally, delete a provisioned dashboard in a non-production instance, redeploy from the stored files, and confirm it returns unchanged. Record the results, the versions tested and who signed off, and ask your partner or internal team to reproduce each check during handover.

How DeployOpen scopes and delivers Kubernetes monitoring

DeployOpen plans, builds, documents and supports self-hosted Grafana monitoring inside the infrastructure you control, including the clusters, the metrics store and the log store. We start from a written scope that names the clusters, workloads, alert recipients and retention you need, so the Prometheus-compatible metrics, Loki logs and dashboards are sized to that scope.

How we scope

We list every cluster, namespace and workload class that needs metrics, logs and alerts before any build starts.
We confirm which Kubernetes versions run in each cluster, since the version skew policy governs how the collector and exporters are installed.
We record retention targets for metrics and logs, plus where Loki object storage and Prometheus local storage will live.
Engagements are fixed-scope, with deliverables and timelines agreed in writing before work begins; ongoing support is available as a retainer.

How we build

We deploy Prometheus and Grafana with provisioned data sources and dashboards defined as code, so the setup can be rebuilt.
We configure service discovery, scrape jobs and recording rules for control plane, node and workload metrics.
We set up Loki in the deployment mode that matches the agreed log volume, with auth and a tenant header design agreed up front.
We test alert rules against known failure conditions and confirm notifications reach the right on-call routes.

How we hand over

You receive the configuration repository, dashboards, alert rules and runbooks written for your operators.
We document the Grafana database choice, backup steps for dashboards and Prometheus data, and restore procedures.
We walk your platform and on-call teams through dashboards, alert triage and multi-cluster views in a handover session.
The written record covers what was built, what was verified, and which items remain open for your team to confirm.

How we support

Retainers can cover Grafana, Prometheus and Loki patch updates, plus planned upgrades as Kubernetes and component versions change.
We can extend dashboards and alert rules when new workloads or clusters join the monitoring scope.
Periodic checks review storage growth, scrape health and alert noise, and report changes to your team.
Support terms, response expectations and included work are defined in the retainer agreement before it starts.

Kubernetes cluster monitoring with Grafana