Monitoring ClickHouse with Prometheus and Grafana
Scraping the ClickHouse Prometheus endpoint, the metrics worth alerting on, the Prometheus and Grafana mix-in dashboards, and sending the same metrics to Datadog through OpenMetrics.
ClickHouse monitoring and operations
Watching and troubleshooting self-managed ClickHouse: Prometheus metrics, the built-in dashboard, system tables, Keeper health, tracing and the errors operators hit most.
Using the ClickHouse built-in observability dashboard
What the /dashboard page shows, how to protect it, which charts flag merge, memory and parts problems early, and when to move to Prometheus or ClickStack.
Analyzing ClickHouse system.query_log
Finding slow, failed and expensive queries in system.query_log: useful columns, retention with TTL, sampling with log_queries_probability, and cluster-wide queries across replicas.
Monitoring ClickHouse Keeper health
Checking Keeper with ruok, mntr and lgif, watching leader changes, latency and snapshot sizes, and the Keeper warnings that precede replication problems.
Tracing ClickHouse queries with OpenTelemetry
Propagating trace context into ClickHouse, storing spans, and correlating application traces with query execution to find where time goes.
Fixing the ClickHouse too many parts error
Why ClickHouse rejects inserts with too many parts, how to confirm the cause, and the fixes: batching, async inserts, partition key changes and merge settings.
Troubleshooting ClickHouse memory limit exceeded errors
Finding which queries, caches and merges use memory with system.processes and memory metrics, and fixing memory errors with limits, spilling and schema changes.
Finding and cancelling stuck ClickHouse mutations
Monitoring system.mutations for queued and failing mutations, cancelling them with KILL MUTATION, and preventing large rewrites from hurting production.
Self-monitoring ClickHouse with ClickStack
Pointing the HyperDX UI at ClickHouse system tables for deep debugging: the bundled Selects, Inserts and Infrastructure dashboard, Lucene and SQL search, and keeping it separate from alerting.
