How to monitor ClickHouse with Prometheus and Grafana
Prometheus scrapes the ClickHouse metrics endpoint, while Grafana handles dashboards and Alertmanager routes the alerts, so the replication, merge and memory signals you care about reach a person before users notice. A team running self-managed ClickHouse replicas needs that early warning because ClickHouse rarely fails loudly at first: a lagging replica, a growing merge queue or a memory ceiling creeps up over hours. Prometheus is open source under the Apache-2.0 license, so the software itself costs nothing to license. Your costs are the hosts and disks for Prometheus, the storage for metric history, Grafana's own database and hosting, and the engineering time to write alert rules and keep them tuned. Prometheus keeps local data for 15 days by default and is not replicated, so decide early whether a second instance or remote write to long-term storage is part of the design.
Choose this stack over the built-in /dashboard when you need alerts that page someone, history beyond a few hours, or one view across several clusters and other services. The built-in dashboard suits a single engineer checking a live node. ClickStack suits deep, query-level investigation of the same tables. Prometheus and Grafana are the better fit when the job is watching trends and getting notified. Confirm the metric names and the Prometheus endpoint settings for your ClickHouse version in the server configuration and the official ClickHouse documentation before you write rules, because exposing and enabling that endpoint is a server setting you must verify in your own environment.
Before the first alert fires, decide who owns the Alertmanager routing, how Prometheus reaches every Alertmanager instance, and which Grafana database you will run. The Grafana installation docs state that SQLite is not suitable for production, so plan MySQL or PostgreSQL from the start. Check the current Prometheus and Grafana release lines against your upgrade policy, since both projects retire minor releases on fixed schedules. Keep the alert rules and dashboards in version control, using Grafana provisioning, so a rebuilt host returns to the same monitoring state.
Choosing the storage, dashboard and alerting options
This table compares the four decisions that shape a ClickHouse monitoring stack: Prometheus storage, Grafana's database, where Grafana runs and how dashboards are managed. Read each row for the fit and the operational cost it adds, then decide which costs your team can carry.
| Decision | Option | Fit | Operational cost it adds |
|---|---|---|---|
| Prometheus storage | Local retention (default 15 days, set by time or size) | Short incident look-back on a single server, with metrics kept close to the cluster | The Prometheus server is not clustered or replicated, so you manage backups, disk sizing and restarts as for any single-node database |
| Prometheus storage | Remote write to external long-term storage | Longer retention and history beyond local disk limits | A second storage system to run, plus remote write configuration, capacity planning and query path ownership |
| Grafana database | SQLite (default) | Evaluation or a single small server with few users | Simplest setup, but Grafana's docs say SQLite does not scale for production or high availability |
| Grafana database | MySQL 8.0+ or PostgreSQL 12+ | Production use and any high availability design | A database to provision, back up and upgrade, plus the database-backed setup that high availability requires |
| Grafana hosting | Self-hosted Grafana | Teams that must keep dashboards and metrics inside infrastructure they control | Servers, upgrades on Grafana's release cadence, authentication setup and high availability design, all run by your team or partner |
| Grafana hosting | Grafana Cloud tiers (free, Pro, Enterprise) | Teams that accept a managed service and want less platform work | Recurring platform fees and usage charges; Enterprise has a minimum commitment of 25,000 US dollars per year (Grafana pricing page) |
| Dashboard management | Provisioning from YAML or JSON files at startup | Repeatable environments and dashboards reviewed like code | Upfront work to write and store provisioning files and keep them in step with the running instance |
| Dashboard management | Manual dashboards built in the UI | Fast exploration and one-off investigation | Dashboards drift between environments and are hard to audit or restore after a loss |
Which ClickHouse signals deserve an alert
The cards below group the conditions that most often turn a ClickHouse incident into data loss, failed inserts or slow dashboards. Each category links a symptom to the component that causes it, so alert rules can point operators at the right place. Confirm metric names and thresholds against your ClickHouse version and the output of your own metrics endpoint before writing rules.
Replica health and sync lag
A replica that falls behind or drops out after a node loss leaves queries reading stale or partial data until it catches up. Replicated tables resync missing data from a healthy replica once the node returns, but a replica that never recovers is a silent loss of redundancy. Alert on replicas that are read-only, not active, or whose replication queue keeps growing. Confirm the exact replica and queue metric names in your endpoint output.
Merge backlog and part count
Frequent small inserts create data parts faster than background merges can combine them. When the count climbs, inserts slow down and can be rejected with a too many parts error, which stops ingestion for logs and telemetry. Track the number of active parts per table and the merge queue depth, and alert on a sustained upward trend rather than a single spike. Confirm thresholds with your insert batch sizes and version.
Memory usage near limits
Large aggregations, joins and merges can push server memory toward the configured ceiling, and queries that exceed it fail. Memory pressure on one node also affects its replication and merge work, so it can cascade. Alert on server memory use approaching the limit for a sustained period, and on memory tracked by in-flight queries. Confirm the metric names for total, tracked and cache memory against your endpoint.
Failed and slow query rates
Query failure and latency rates show whether users and dashboards are being served, even when servers look healthy. A rising share of failed queries often points to memory limits, timeouts or a schema change that broke a dashboard. Alert on the ratio of failed queries and on latency percentiles for the query classes you care about, not on raw counts alone. Confirm which query event and duration metrics your version exposes.
Keeper availability and leader stability
ClickHouse Keeper coordinates every ReplicatedMergeTree table, so a Keeper outage or repeated leader elections block inserts and replica coordination across the cluster. The replication guide recommends at least three Keeper nodes on dedicated hosts in production. Alert when a Keeper node stops responding, when the ensemble loses quorum, or when leader changes repeat within a short window. Confirm the Keeper metric names and health commands for your release.
Backup job success
A backup that fails silently leaves no recovery point, and replication does not protect against deletion or corruption. Alert on failed or missing scheduled BACKUP jobs and on the age of the last successful backup. Pair the alert with periodic restore tests, because a backup that has never been restored may not restore when needed. Confirm how your backup jobs report status, since they may not expose a Prometheus metric by default.
Setting up scraping, alert rules and dashboards in order
This sequence takes ClickHouse from an enabled metrics endpoint to alerting and dashboards that your team can rebuild from files. Each step closes with a check, so a gap in scraping, alert routing or Grafana setup shows up before it matters. The result is monitoring that covers every replica and can be reviewed like other infrastructure code.
Enable the ClickHouse Prometheus endpoint
Turn on the Prometheus metrics endpoint in the server configuration on every ClickHouse node. Confirm the port and path your version exposes in the ClickHouse documentation for that release, and restrict access with network rules or authentication so metrics are not open to anyone who can reach the host. From the Prometheus host, request the endpoint on each replica and compare the output. The step is done when every replica answers and returns the same metric families.
Add scrape targets to Prometheus
Define a scrape job in prometheus.yml that lists each ClickHouse replica, either as static targets or through service discovery that matches your platform, such as Kubernetes or your cloud provider's inventory. Prometheus pulls metrics over HTTP, so the Prometheus host needs network reachability to every replica's endpoint. Add labels for cluster, shard and replica so dashboards and alerts can filter by node. Reload the configuration and check the Prometheus status page for the new job. The step is done when the job lists every intended replica and no unexpected hosts.
Confirm every target is up
Open the targets view in the Prometheus web interface and confirm that each ClickHouse replica shows as up with a recent last scrape. Run a PromQL query against one metric family that you confirmed in your own endpoint output, and check that it returns one series per replica. If a target is down, read the error shown for it, which usually points to a network rule, a wrong port or an authentication setting on the endpoint. The step is done when all expected targets are up and the query returns a series for each replica.
Write and test alert rules
Write alert rules for conditions operators must act on, such as a replica that stops answering scrapes, a replica falling behind its peers, or memory and disk use trending toward limits. Set thresholds from your own baseline of normal load and record the reason for each rule. Test each expression in the Prometheus expression browser, then trigger a safe condition on a non-production cluster to confirm the alert fires and reaches a receiver. Prometheus values reliability over precision, so treat alerts as signals that start an investigation. The step is done when each rule has been seen firing once in testing.
Point Prometheus at every Alertmanager
List each Alertmanager instance in the alerting section of the Prometheus configuration. Prometheus should send alerts to every instance it knows about, so do not place a load balancer in front of them. If you run more than one instance, confirm they form a cluster using the cluster settings in the Alertmanager documentation, and check that grouping, inhibition and silences match your on-call process. Stop one instance during a test window and confirm a test alert still reaches a receiver. The step is done when the test alert arrives through the surviving instances.
Install Grafana and set its database
Install Grafana from a supported package for your operating system or from its container image, following the installation documentation for the version you plan to run. For production, move Grafana's configuration database from the default SQLite to MySQL 8.0 or later or PostgreSQL 12 or later, because the installation docs say SQLite does not scale for production. If you plan two or more active-active servers behind a load balancer, follow the high availability guide and configure alerting to avoid duplicate notifications. The step is done when the server starts, reports the database backend you chose, and you can sign in.
Provision the data source and dashboards as code
Define the Prometheus data source, dashboards and alerting resources in YAML or JSON provisioning files, which Grafana loads at startup. Store the files in version control and deploy them through your existing pipeline so every dashboard change is reviewed. Grafana also documents Terraform and Git Sync as as-code options, which your team can evaluate against its change process. Start Grafana from the files in a test environment and confirm the data source and dashboards appear without manual steps and that a panel returns Prometheus data. The step is done when a fresh install reproduces the same setup.
Keeping Prometheus, Alertmanager and Grafana healthy
Local Prometheus storage is not clustered or replicated, so each Prometheus server must be operated like any single-node database. The Prometheus storage documentation says to manage it that way, and the practical consequence is that a disk filling up or a server failing silently removes your ClickHouse visibility at the moment you need it most. Plan disk headroom against the default 15-day retention or the size and time limits you configure, take backups of the data directory on a schedule you test, and, if ClickHouse alerts are critical, run a second scraper so that a single server outage does not stop detection. Remote write to long-term storage is an option for retention, but it does not by itself give you high availability for alerting.
Alertmanager can run as a cluster for high availability using its --cluster flags, and Prometheus should be configured to send alerts to every Alertmanager instance rather than through a load balancer. Grafana high availability works differently: two or more active-active Grafana servers sit behind a load balancer and share a MySQL or PostgreSQL database, which is required for production and high availability because SQLite does not scale. Be aware that alerting in Grafana needs extra configuration to avoid duplicate notifications across nodes, and Grafana Live has limitations in HA. Confirm both behaviors in a test environment before relying on Grafana alerts for paging.
Plan upgrades on the vendors' cadences. Prometheus minor releases arrive on a 6-week cycle, and a minor release generally stops receiving bug fixes after that window, so the LTS line matters for production: the current LTS 3.13 line is supported until July 31, 2027. Grafana ships minor releases every other month and patch releases at least monthly, with each minor supported for 9 months, so schedule a recurring patch window rather than waiting for a major upgrade. Assign an owner for each component and confirm the support dates against the vendors' pages before each planning cycle.
Sending the same ClickHouse metrics to Datadog through OpenMetrics
Teams that keep Prometheus and Grafana for alerting and dashboards usually add Datadog as a second consumer of the same ClickHouse metrics. Prometheus stays the system of record for alerts, so the existing scrape job does not need to change, and Datadog collects through its own path. The platform team that runs Prometheus keeps ownership of the ClickHouse endpoint, while the Datadog account owner decides which checks and dashboards the organization uses.
- Scrape configuration: Leave the Prometheus scrape job as it is. Give Datadog its own collection configuration, and do not point it at the Prometheus server unless Datadog confirms that setup is supported.
- Agent ownership: Name one owner for the Datadog collector who is separate from the Prometheus alerting owner, so a change to one does not silently break the other.
- Metric names and types: Export a sample of what the ClickHouse endpoint returns and check each name and type (counter, gauge or histogram) against what the Datadog OpenMetrics check expects.
- Cost and cardinality: Ask Datadog how metrics collected this way are counted toward billing, and decide which high-cardinality labels to drop before they multiply the series count.
Before enabling the second consumer, confirm with Datadog which OpenMetrics check settings apply to the ClickHouse version you run and whether a ClickHouse-specific integration is available to your account.
Teams replacing Prometheus with Datadog as the primary system need to decide what happens to alert rules, stored history and dashboards. Prometheus keeps local data in a single-node store with a default retention of 15 days, so any history you need for baselines must be exported or accepted as lost. Plan the cutover so that alerting never has a gap in coverage.
- Parallel running: Run both collectors for a period and compare series for the same ClickHouse metrics before Prometheus is retired. If you consider remote write from Prometheus into external storage, confirm with Datadog whether it accepts that data path.
- Alert inventory: List every Prometheus alert rule and Alertmanager route, then recreate each one as a Datadog monitor with the same thresholds, grouping and notification targets.
- Dashboards: Grafana dashboards do not move automatically. Rebuild the panels your operators use during incidents first, and retire the rest later.
- Ownership at cutover: Assign the Datadog-side owner before Prometheus is decommissioned, so someone is accountable for ClickHouse alerts at every point during the change.
Before retiring Prometheus, confirm with Datadog the metric names and types for your ClickHouse version and keep the Prometheus scrape running until the comparison shows matching series.
Teams that already run the Datadog agent can scrape the ClickHouse metrics endpoint through the agent instead of adding Prometheus to the path. The agent runs on or near the ClickHouse hosts, so network reachability to the endpoint and any authentication requirements are the first things to check. Ownership usually sits with the observability team that runs the agent, while the ClickHouse operators own the endpoint itself.
- Scrape configuration: Add an OpenMetrics check for the ClickHouse endpoint in the agent configuration, and limit access to the endpoint so only the agent host addresses can reach it.
- Authentication: Confirm whether your ClickHouse configuration requires authentication on the metrics endpoint, and store any credentials in the secrets mechanism your agent deployment already uses.
- Change control: Agree in writing who edits the check file, who approves changes and who is paged when the check stops reporting.
- Metric review: Compare the metrics the check returns with the ones you plan to alert on, and remove high-cardinality labels in the agent configuration before the data reaches billing.
Before relying on the check, confirm with Datadog which OpenMetrics options and metric types apply to your ClickHouse version, and do not assume a ClickHouse-specific integration exists unless Datadog documents one.
Securing the metrics endpoints and dashboards
Metrics are sensitive. Query patterns, table names, host names and error details can show how your ClickHouse is used, so treat the scrape path and every dashboard as production data access. The five controls below cover the ClickHouse endpoint, Prometheus exposure, Grafana browser settings, Grafana metadata exposure and identity choices.
Limit the ClickHouse metrics endpoint
The ClickHouse Prometheus endpoint should answer only the scraper addresses that need it. Place Prometheus on a monitoring network or allow-list the scraper addresses in host and network firewall rules, and do not publish the port through a public load balancer. Confirm in your ClickHouse version's documentation whether the endpoint can require credentials. If it cannot, network controls are your only barrier, so test the listener from outside the scraper network before go-live.
Follow Prometheus exposure guidance
The Prometheus security model page sets out the assumptions the project makes about who can reach the server and its API. Read it before exposing any Prometheus address beyond the monitoring network. If remote users need the web UI, put Prometheus behind an authenticating reverse proxy instead of publishing it directly, and keep the query API closed to anyone who does not need it. Review these settings again after each Prometheus upgrade.
Harden Grafana browser sessions
Grafana's hardening guide recommends secure and SameSite cookies, a Content Security Policy and Trusted Types. These settings limit what an injected script or hijacked browser session can do with a dashboard. Enable them in the server configuration, then test every data source panel, plugin and embedded view, because a strict policy can break plugins that load inline content. Store the settings in your configuration repository so they survive upgrades.
Hide version and protect metrics
Grafana can hide its version from unauthenticated visitors and can require authentication for its own metrics endpoint. Enable both, because a visible version helps an attacker choose known weaknesses to target. Also turn on the domain verification setting so Grafana rejects requests sent through DNS rebinding. Confirm the exact configuration keys for your release in the hardening guide before you change production.
Choose SSO features by edition
Grafana authentication supports LDAP, generic OAuth, Entra ID, Okta, GitHub, GitLab, Google OAuth, auth proxy and JWT in every edition. SAML and SCIM provisioning need Grafana Enterprise or Grafana Cloud, and team sync and active sync follow the same rule. If your identity provider requires SAML or SCIM, include that edition in the design and confirm current licensing with Grafana before you choose the platform.
Limits to confirm before you commit to this stack
This table lists the facts an operator must confirm in their own environment before committing to Prometheus, Grafana and Datadog for ClickHouse monitoring. Read each row as a check to close before design work starts, and assign the confirming party named in the last column.
| Item | Why it matters | Who confirms it |
|---|---|---|
| ClickHouse version and metrics exposure | Metric names and availability differ between release lines, so the version in production decides which exporter settings and dashboards apply. | Platform team, against the release notes for the exact line in use |
| Endpoint configuration and port | The Prometheus endpoint must be enabled in server configuration and reachable from the scraper, and firewall rules must allow only the approved scraper. | Platform team and network security |
| Metric names and label sets | Alert rules and dashboards depend on exact names and labels, and a rename or label change silently breaks them. | Monitoring owner, checked against a test scrape of the target release |
| Grafana mix-in dashboard fit | A community or vendor dashboard may target a different ClickHouse version, so panels can show missing or wrong data. | Monitoring owner, by comparing panel queries with scraped metrics |
| Datadog OpenMetrics coverage | Not every scraped metric may map cleanly through the OpenMetrics integration, and custom metric volume can change cost. | Observability lead, with the Datadog account owner and a test agent run |
| Prometheus cardinality and retention | Series count and label combinations set memory, disk and retention needs, and local storage is not clustered or replicated. | Platform team, using measured series counts from a pilot scrape |
| Alertmanager high availability | Alertmanager clustering and sending to every instance affect whether alerts fire during a node failure. | Monitoring owner, verified with a failover test |
| Grafana Enterprise features needed | SAML, team sync, active sync and SCIM are limited to Grafana Enterprise and Grafana Cloud, so identity needs may force a licensing decision. | Identity lead and procurement, against the current Grafana authentication documentation |
| Grafana backend database for high availability | SQLite does not scale for production, so MySQL or PostgreSQL is required for high availability setups. | Platform team, sized against the chosen Grafana version |
How much effort and cost does ClickHouse monitoring with Prometheus and Grafana take?
Do Prometheus and Grafana cost anything to license?
Neither the Prometheus project nor the open-source Grafana edition carries a license fee. Prometheus is licensed under Apache-2.0 and has no paid enterprise edition of its own. Grafana's open-source edition is licensed under AGPLv3 after Grafana Labs relicensed it in April 2021, and Grafana Enterprise is a separate commercial option. The real costs sit elsewhere: the servers and disks that run Prometheus and Grafana, the storage for metric history, the Datadog subscription if you forward metrics there, the implementation work, and the operator time to keep alerts and dashboards current. Ask any partner to separate these lines in the quote so you can see which ones recur each year.
What drives the effort of monitoring ClickHouse up or down?
Effort rises with the size of the cluster and the expectations placed on monitoring. The main drivers are:
- The number of ClickHouse replicas and shards, since each host adds scrape targets, dashboards to check and alerts to tune.
- The number of ClickHouse Keeper nodes. The replication guide recommends at least three Keeper nodes on dedicated hosts in production, and each one needs health checks.
- Retention. Prometheus keeps local data for 15 days by default, and longer history means remote storage or a long-term store to design and run.
- High availability. Running two Prometheus servers or an Alertmanager cluster, as the Alertmanager docs describe, adds configuration and testing.
- Datadog. Sending the same metrics through OpenMetrics means a second pipeline, its own agent configuration and its own cost model.
Can our own team set up ClickHouse monitoring without a partner?
Yes, a team with Linux, Kubernetes or systemd experience and some Prometheus knowledge can usually build a working scrape, a dashboard and a first set of alerts. The harder part is the ongoing judgment: choosing alert thresholds that do not flood on-call staff, keeping the Prometheus and Grafana versions current, protecting the metrics endpoint and the Grafana login, and testing that alerts reach the right people. Teams without a Prometheus owner often find that the first incident exposes gaps in the setup. Decide who owns the stack after go-live before you decide who builds it.
What do we need to provide before monitoring work can start?
The work moves faster when the following are in place before scoping begins:
- Network paths from the Prometheus host to each ClickHouse server's metrics port and to each Keeper node.
- Read access to the ClickHouse hosts and configuration files, so the metrics endpoint can be enabled and checked.
- A named owner for each alert, with an agreed escalation path.
- A test environment with representative data, where scrape settings and alert rules can be tried before production.
- Access to your Grafana database if you run Grafana in high availability, along with your authentication choice.
- A decision on whether Datadog is in scope, and the Datadog account and API access that implies.
How long does a ClickHouse monitoring implementation take?
Duration depends on scope, and no fixed timeline applies to every environment. A single cluster with one Prometheus server, a standard dashboard and a short list of alerts is a much smaller piece of work than a multi-site deployment with high-availability Prometheus, Grafana behind a load balancer, provisioned dashboards as code and a Datadog pipeline. Ask for the scope to list each component, the environments it covers, the alert set and the handover tasks. Timelines should follow from that written scope and the approvals your change process requires. DeployOpen agrees scope, deliverables and timelines in writing before work starts.
What does a quote for ClickHouse monitoring include?
A quote follows from the written scope, so DeployOpen produces one after the scope is agreed, not before. Ask that the quote lists the clusters and hosts covered, the Prometheus and Grafana topology, whether high availability is included, the alert set and dashboards to be delivered, the Datadog work if it applies, testing and handover, and any ongoing support or upgrade retainer. Confirm which items are one-time implementation and which are recurring retainer or infrastructure costs. DeployOpen does not publish a price list.
Which recurring costs should we budget for after go-live?
Recurring costs usually fall into four groups: infrastructure for Prometheus, Alertmanager and Grafana, storage for metric history, the Datadog subscription if used, and people time for upgrades and alert tuning. Upgrades recur on a schedule. Grafana's upgrade guidance describes minor releases every other month and patch releases at least monthly, and Prometheus minor releases arrive every 6 weeks, so someone must test and apply them. Ask any partner to state whether upgrades, alert review and on-call support are included in a retainer or billed as separate work.
How DeployOpen scopes and supports ClickHouse monitoring
DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure, including the Prometheus and Grafana layer that watches ClickHouse. Engagements are fixed-scope, with the metrics, dashboards, alert rules and handover agreed in writing before work starts.
How we scope
- We confirm which ClickHouse clusters, Keeper nodes and hosts need scraping and where Prometheus will run.
- We agree which alerts, dashboards and retention periods the written scope covers, and who receives each alert.
- We record whether Datadog, Grafana Cloud or another destination is in scope, and what each one costs to run.
How we build
- We configure scrape targets, exporter endpoints and labels against your clusters and test them in a pre-production environment first.
- We provision Grafana data sources and dashboards as code so they can be reviewed, versioned and rebuilt.
- We test alert rules with simulated conditions and confirm Alertmanager routing reaches the people named in the scope.
How we hand over
- We document scrape configuration, alert thresholds, dashboard ownership and the runbook for each alert.
- We walk your operators through the dashboards and the Keeper and replication signals they should watch first.
- We hand over access, configuration files and a list of open items, with each item assigned to an owner.
How we support
- Ongoing support, upgrades and continuous monitoring reviews are available as a retainer after handover.
- We review Prometheus, Grafana and ClickHouse version changes against the monitoring stack before you upgrade.
- Retainer scope, response expectations and review cadence are written into the agreement, so you can confirm them before signing.
