Ceph monitoring with Prometheus, Grafana and Alertmanager: what cephadm deploys and when you need it
If your Ceph cluster is in production and its alerts need to reach an on-call team, you need a metrics store, dashboards and a routing layer, and cephadm can deploy all three for you. The monitoring stack that cephadm sets up is Prometheus for collecting and storing time-series metrics, Grafana for dashboards, and Alertmanager for turning alert rules into notifications. Each has a distinct job. Prometheus pulls metrics over HTTP from targets it discovers, keeps them in its local time series database, and evaluates alert rules written in PromQL. Grafana queries Prometheus and renders the panels your storage and operations teams read every day. Alertmanager receives firing alerts from Prometheus and applies grouping, inhibition and silences before sending them on, so one failing host does not page the team once per OSD.
Prometheus local storage is single-node. The Prometheus storage documentation states that it is not clustered or replicated and should be managed like any other single-node database. Data is held in two-hour blocks with a write-ahead log, and default retention is 15 days, configurable by time or size. If the Prometheus host is lost, the history goes with it unless you planned for it. Remote write can forward samples to external long-term storage, but that is a design decision you should make before go-live, not after an incident. Alertmanager can run as a cluster for high availability, and Prometheus should be configured to reach every Alertmanager instance rather than a load balancer in front of them.
Licensing and cost deserve a clear view. Prometheus is licensed under Apache-2.0 and has no paid enterprise edition, so its cost is infrastructure, storage and the people who run it. Grafana changed course in 2021: on April 20, 2021, Grafana Labs relicensed Grafana, Loki and Tempo from Apache License 2.0 to AGPLv3, and offers Grafana Enterprise as a non-AGPL commercial option. Confirm with your legal team whether AGPLv3 terms fit how you will run and modify Grafana, and check the Grafana Labs licensing page for the current position. Cephadm deploys the open-source components, so the decisions that drive your budget are sizing the metrics retention, redundancy for the monitoring hosts and the time your team spends maintaining them.
Choosing between the cephadm stack and an existing Prometheus
This path uses the Prometheus, Grafana and Alertmanager services that cephadm deploys alongside the Ceph cluster. cephadm places and configures these containers, so the monitoring stack follows the same upgrade cycle as Ceph and sits close to the daemons it watches. The trade-off is that the storage team owns another stateful system on the cluster's own hosts.
- Storage and retention: Prometheus keeps its time series on local disk on the host where cephadm places it. Plan the disk, retention period and size limit before go-live, because local storage is not clustered or replicated.
- Alert rules: The Ceph team usually owns the default rules that ship with Ceph and any custom rules added to the stack. Changes are made in the cluster's configuration, so the rules are tied to the Ceph version in use.
- Grafana access: Operators reach the Grafana instance that cephadm deploys, so network paths, TLS and login methods must be set for that host. Confirm which users can reach it from the management network.
- Placement and versions: Confirm which hosts run the Prometheus, Grafana and Alertmanager services, and that they have enough CPU, memory and disk. Confirm that the monitoring images match the Ceph release you run.
- Alert delivery: Alertmanager routes alerts to receivers you configure, including webhooks, so the on-call path must be tested from this instance.
Confirm the host placement, the retention limit and the Grafana access model with your Ceph and monitoring owners before you decide to keep this stack.
This path scrapes Ceph's metrics endpoints into a Prometheus server that your organization already runs, and routes alerts through your existing Alertmanager. The central monitoring team keeps its retention, storage and alert policies, and the Ceph team contributes scrape targets and rules. This suits organizations that already run Prometheus at scale or need one alerting policy across many systems.
- Storage and retention: Ceph metrics are added to your current retention and storage design. Confirm that the volume of Ceph series fits your capacity plan and that long-term storage is handled by your existing remote write setup, if you have one.
- Alert rules: Your monitoring team typically owns the rule files and routing. Agree in writing which team reviews changes to Ceph alert rules, how rules are versioned, and who gets paged for Ceph alerts.
- Grafana access: Ceph dashboards are imported into the Grafana instance your organization already uses. Confirm the data source name, the access model and whether the Ceph dashboards need changes for your labels.
- Placement and versions: Confirm the network path from your Prometheus servers to every Ceph manager and exporter endpoint, and confirm the Ceph release so the metric names and labels match the dashboards and rules you import.
- Alertmanager integration: Prometheus should point at every Alertmanager instance, so confirm your Alertmanager cluster is configured that way before you add Ceph receivers.
Confirm the metric names for your Ceph release, the scrape network path and who owns each rule file before you start.
The right path depends less on the software than on who must answer alerts, pay for storage and change rules. A storage team that wants Ceph monitoring to move with Ceph upgrades may prefer the cephadm stack, while a platform team with mature Prometheus operations may prefer scraping. Both paths need the same tests, so the decision should be recorded before build work starts.
- Accountability: Name the team that answers a Ceph alert at night and the team that changes its threshold. These are often different people in a shared model.
- Cost components: Infrastructure, storage and the staff time to run each stack are the main costs. Ask each option's owner to estimate those in the scope, since DeployOpen does not set internal costs.
- Change control: Rules and dashboards need a review process whichever path you take. Decide whether they live in version control or in the cluster configuration.
- Upgrade coupling: A cephadm stack follows Ceph upgrades. A shared Prometheus follows your own release schedule, so check that it can scrape the metric set of your Ceph release.
- Audit evidence: Record retention, access and alert history in whichever system you choose, so reviewers can see how alerts were handled.
Before you choose, ask each owner to confirm the on-call path, the retention requirement and the upgrade plan they will accept.
In air-gapped or tightly restricted networks, the monitoring choice also depends on how container images and packages reach the site. The cephadm stack needs the Prometheus, Grafana and Alertmanager images available in the site's internal registry before deployment. An existing Prometheus needs a network path from its scrapers to Ceph, which some segmented networks restrict.
- Images and packages: Confirm that the monitoring images for your Ceph release are mirrored internally, and record their versions for the change record.
- Network segmentation: Confirm which firewall rules allow scraping of Ceph exporters and which allow Alertmanager to reach webhook receivers inside the site.
- Grafana authentication: Confirm which login methods your environment supports, since some features depend on Grafana edition. Check the authentication options against your identity provider before you commit to a design.
- Data residency: Confirm where metrics and alert payloads are stored, especially if alerts include host or user identifiers.
- Recovery: Confirm how the monitoring stack is restored if its host fails, since local Prometheus storage needs its own backup approach.
Confirm image mirroring, firewall paths and the authentication model with your security team before you start the build.
Configuration options and when each fits
This table compares the main configuration choices for the monitoring stack around a Ceph cluster, including Prometheus retention, remote write, Alertmanager clustering, the Grafana database backend, webhook receivers and dashboard provisioning. Read each row for the condition that makes the option right and the effort it adds to build and run.
| Option | When it fits | Effort to set up and run |
|---|---|---|
| Prometheus retention by time or size | Short incident history on the cephadm-deployed Prometheus is enough. The default is 15 days, so extend it when capacity reviews need several months of OSD and pool trends. | Low to set, but disk use grows with the number of OSD and pool series. Confirm the disk sizing with a test scrape before changing retention. |
| Remote write to long-term storage | Compliance or capacity planning requires metrics kept beyond the local window, or you already run a long-term metrics platform. | Moderate. Someone must own the remote endpoint, its retention and its availability, because local Prometheus storage is not clustered or replicated. |
| Alertmanager cluster mode | Alerts must keep routing if one Alertmanager instance fails, and the operations team needs silences and grouping to hold across nodes. | Moderate. Enable the cluster flags on each instance and point Prometheus at every Alertmanager instance, not at a load balancer. |
| Grafana database backend: SQLite | A lab or proof of concept where one Grafana server is acceptable and no one depends on its dashboards for on-call. | Lowest effort, but the Grafana installation docs say SQLite does not scale for production, so plan a migration before go-live. |
| Grafana database backend: MySQL or PostgreSQL | Production use or high availability with two or more active-active Grafana servers behind a load balancer. | Higher. You operate the database, backups and upgrades, and alerting needs extra configuration to avoid duplicate notifications. |
| Webhook receivers | Alerts must reach a ticketing, chat or on-call tool that accepts HTTP posts from Alertmanager. | Low for a single receiver. Every receiver needs authentication, retry behavior and a test path, and the payload format must be mapped to your tool. |
| Dashboard provisioning as code | Dashboards and data sources must be reviewed, versioned and rebuilt identically in each environment. | Moderate upfront to write YAML or JSON files and load them at startup. It lowers drift later, but manual edits made in the UI will be overwritten by provisioned files. |
Setting up Ceph alert routing to webhooks step by step
This sequence takes a cephadm-managed Ceph cluster from its built-in monitoring services to a tested alert path, where a firing Ceph alert reaches a webhook receiver and noise is controlled with silences and inhibition. Each step ends with a check you can run before moving on.
Confirm cephadm monitoring services and placement
Start by listing the monitoring services cephadm runs on the cluster, which typically include Prometheus, Alertmanager, Grafana and the node exporter, and note the hosts each one is placed on. Your cluster administrator runs the orchestrator commands that show service status and placement, and records which host runs Alertmanager and which runs Prometheus. Confirm that each service reports as running and that its placement matches your intended monitoring hosts. Done means you have a written inventory of the monitoring daemons, their hosts and their ports, which you will reuse in every later step.
Verify Prometheus scrapes the Ceph manager exporter
Check that Prometheus has a scrape target for the Ceph manager's Prometheus module, which exposes cluster metrics such as health status and OSD state. Open the Prometheus targets view on its web port, or query the targets API, and confirm the Ceph manager endpoint shows as up. Also run a simple query for a core Ceph metric to confirm samples are arriving. If the target is missing or down, fix the scrape configuration or the manager module before adding any rules, because alerts built on absent data will never fire. Done means the Ceph target is up and the test query returns recent values.
Add custom alert rules to Prometheus rule files
Create or extend a rule file that Prometheus loads, with groups of alerting rules based on PromQL expressions over Ceph metrics. Give each rule a name, a expression, a for duration, and labels such as severity, plus annotations that describe the problem and the first response. Reference the rule file from the Prometheus configuration and reload the server. Keep the first set small, for example cluster health and capacity conditions, so the team can judge their quality. Done means Prometheus reports the rule group as loaded without errors and the rules evaluate on the rule page.
Configure Alertmanager receivers for webhooks and grouping
Edit the Alertmanager configuration to define a webhook receiver with the URL of your incident or chat endpoint, and set a route tree that groups alerts by labels such as alertname and cluster. Choose a default receiver and add child routes for severity levels if you need different destinations. Validate the file with the amtool check-config command before applying it, then reload Alertmanager. Confirm that Prometheus lists every Alertmanager instance in its alerting section, pointing at each instance directly rather than at a load balancer. Done means the configuration validates and Prometheus shows every Alertmanager as active.
Test a firing alert end to end
Trigger a test alert in a controlled way, for example with a rule whose threshold you temporarily set to match current data, or by sending a synthetic alert to Alertmanager through its API with a clearly labeled test name. Watch the alert move from pending to firing in Prometheus, then confirm Alertmanager groups it and that the webhook receiver gets the expected payload. Check the receiver logs and the Alertmanager status page. Restore the original threshold and confirm the resolved notification arrives. Done means you have seen one firing and one resolved notification reach the webhook, with the grouping labels you expected.
Set silences and inhibition rules
Once routing works, reduce noise by adding inhibition rules so that a cluster-wide alert suppresses the per-host or per-OSD alerts that follow from it, and create silences for planned work such as drive replacement or host maintenance. Define inhibition in the Alertmanager configuration with source and target matchers, and create silences through the Alertmanager interface or API with an owner and a reason. Record each silence in your change process so it expires or is removed. Done means a test with a parent and child alert produces only the parent notification, and a silence created for a test label suppresses matching alerts until it expires.
Writing custom alert rules that page the right people
Custom alert rules in Prometheus are PromQL expressions evaluated on a schedule, and each rule fires only after its condition holds for the duration set in the for field. Start with the health alerts that already ship with Ceph, since they reflect the cluster's own view of its state, then add rules for the conditions those alerts do not cover: OSD commit or apply latency sustained above a threshold you have measured on your hardware, raw capacity used approaching the nearfull and full ratios, and the number of monitors in quorum falling below the majority your cluster needs. Before writing any expression, query the Prometheus expression browser against your own endpoint and confirm each metric name and label set exists in the Ceph release you run. Metric names and labels have changed between releases, so a rule copied from a blog post can silently never fire. Write alert tests with promtool test rules so a broken rule fails in review rather than during an incident.
Noisy rules are usually rules with no for duration, thresholds set below normal peak load, or conditions that flap during planned recovery and rebalancing. Use for windows long enough to ignore brief spikes, and use recording rules to smooth expensive expressions. Prometheus values reliability over precision, so a rate computed over a short window may be approximate, which is acceptable for paging on sustained degradation but not for billing or exact accounting. Decide routing before writing rules: a monitor quorum loss, a full OSD or a cluster in HEALTH_ERR belongs with on-call, while a sustained rise in latency on one host or a capacity trend that will exhaust space in weeks belongs in a ticket queue. Express that split with a severity label and match on it in the Alertmanager routing tree, and confirm with your operations team which conditions they can act on at 3 a.m.
Route pages through Alertmanager with grouping so one failed host produces one notification, and use inhibition so that a host-down alert suppresses the OSD alerts it causes. Review every rule after each Ceph upgrade and after the first real incident it covered, and retire rules that have not produced an actionable page. Ask your partner or internal team to document each rule's owner, the runbook it links to and the test that proves it fires. Treat the alert set as code kept in version control, because an unreviewed threshold change is a common source of missed or excessive alerts.
Securing the monitoring stack and its endpoints
The monitoring stack holds a map of your storage cluster, so it needs the same access discipline as the cluster itself. The cards below cover the Grafana settings, network exposure, alert delivery and sign-in options that connect Prometheus, Grafana, Alertmanager and the Ceph dashboard into one reviewed design.
Grafana hardening settings
Grafana ships with browser and endpoint protections that should be reviewed rather than assumed. Check secure and SameSite cookies, a Content Security Policy, Trusted Types, hiding the Grafana version from unauthenticated users, and requiring authentication for the metrics endpoint. Also confirm domain verification against DNS rebinding for the URL users reach. Record each setting in the configuration so an upgrade does not silently reset it.
Prometheus and Alertmanager exposure
Prometheus has no built-in user authentication, so anyone who can reach its HTTP port can query metrics and read configuration. Place it behind a reverse proxy that enforces authentication, or restrict access with network controls so only Grafana, administrators and approved scrapers can connect. Apply the same rule to Alertmanager, because its API can create silences that hide real alerts.
Webhook credentials and TLS
Alert receivers such as chat or ticketing webhooks carry tokens that grant write access to other systems. Store those values in files or a secrets system rather than in plain configuration committed to a repository, restrict who can read the Alertmanager configuration, and use TLS for every hop. Confirm with each receiver's vendor how tokens are rotated and revoked, and test rotation before an incident requires it.
Grafana sign-in options
Choose the sign-in method against your edition and identity provider. LDAP, generic OAuth, Entra ID, Okta, GitHub, GitLab and Google OAuth are available in all editions, while SAML, team sync, active sync and SCIM provisioning require Grafana Enterprise or Grafana Cloud. If your directory only speaks SAML and you do not use Enterprise, plan an OAuth or LDAP bridge and confirm its support with your identity team.
Ceph dashboard SSO scope
The Ceph Dashboard supports single sign-on through SAML 2.0, and through OAuth2 when cephadm runs the Management Gateway and OAuth2 Proxy services. Confirm which option your cluster version supports before designing around it. Note that dashboard sign-in covers only the dashboard, so Grafana, Prometheus and Alertmanager still need their own access controls.
Limits to confirm before you rely on this stack
Is Prometheus local storage clustered or replicated?
No. Prometheus local storage is not clustered or replicated, so it should be managed like any other single-node database. Each Prometheus server keeps its own two-hour blocks and write-ahead log, and the default retention is 15 days, configurable by time or size. If you need longer retention or a copy of the data outside the host, plan for remote write to external long-term storage and test the read path. Confirm the disk sizing, backup approach and what happens to alerting when the single Prometheus host fails.
Do Prometheus and Alertmanager need every Alertmanager instance configured?
Yes. For Alertmanager high availability, Prometheus should point at every Alertmanager instance rather than at a load balancer in front of them. The instances form a cluster through the --cluster flags. Confirm that each Alertmanager node is listed in the Prometheus alerting configuration, that the cluster peers can reach each other, and that you test a notification after failing one instance. Routing and receiver settings must also match across nodes, so review them as a single change.
Does Grafana high availability send duplicate alert notifications?
It can, unless you configure alerting for high availability. Grafana high availability uses two or more active-active servers behind a load balancer, sharing a MySQL or PostgreSQL database, and the documentation notes that alerting needs extra configuration to avoid duplicate notifications. Grafana Live also has limitations in high availability setups. Confirm the alerting settings with a test alert, check that users can reach any node, and decide whether live streaming features are needed before you depend on them.
What database does Grafana need for production?
Grafana stores its configuration in SQLite 3 by default, but the installation documentation says SQLite does not scale for production. For production and high availability, plan for MySQL 8.0 or later, or PostgreSQL 12 or later. Confirm which database your team already backs up and restores, and include it in your recovery test, because dashboards, data sources and alert configuration live there.
How long is each Prometheus and Grafana release supported?
Prometheus minor releases start a new six-week cycle, and a minor release generally stops receiving bug fixes after those six weeks. Prometheus long-term support releases get bug fixes for one year. The current long-term support line is 3.13, supported until July 31, 2027. Grafana minor releases are supported for nine months, and the last minor release of a major version gets 15 months. Plan an upgrade cadence that stays inside these windows, and confirm the dates against the release pages before you commit.
Which Ceph release will my cephadm version deploy?
The Ceph release is set by the container image that your cephadm version uses, so it can differ between clusters and between installs. Cephadm was introduced in v15.2.0 (Octopus) and does not support older releases. Confirm the exact Ceph image version and the Prometheus, Grafana and Alertmanager images that cephadm pulls for your cluster, then check the Ceph active releases page for end-of-life dates. Record these versions in your change log so an upgrade or audit uses the actual running images.
Can we use our existing Prometheus instead of the one cephadm deploys?
Yes, but confirm it before you design around it. Cephadm deploys its own monitoring stack, and integrating with an existing Prometheus means scraping the Ceph exporters, adding the alert rules and making sure the labels match your dashboards. Check the scrape interval, the retention you need, who owns the rule files, and whether your existing Alertmanager receives Ceph alerts without duplicates. Test the integration in a non-production cluster first.
Does Grafana alerting cover the same features as Prometheus Alertmanager?
Not necessarily, so confirm which alerting path your team will use before building rules in two places. Alertmanager handles grouping, inhibition and silences for alerts sent by Prometheus. Grafana has its own alerting configuration, and its high availability setup needs additional settings to avoid duplicate notifications. Decide on one primary alert path for Ceph health, document who owns each rule, and test that a single failure produces one notification.
Running monitoring after go-live: upgrades, retention and dashboards
Plan the monitoring stack's upgrades on its own calendar, separate from Ceph's. Prometheus publishes a new minor release cycle every 6 weeks, and a minor release generally stops receiving bug fixes after those 6 weeks, so a team running a non-LTS line must upgrade on that rhythm or accept unpatched software. Prometheus long-term support lines receive bug fixes for one year, which suits a team that wants fewer upgrade events. Grafana works differently: major releases arrive once a year, minor releases every other month, and each minor is supported for 9 months. Decide which Prometheus line and which Grafana minor track you will run, write the upgrade window into your operations calendar, and confirm the current support dates on the Prometheus release cycle page before you commit.
Keep dashboards, data sources and alert rules in Git rather than editing them in the UI. Grafana provisioning loads data sources, dashboards and alerting resources from YAML or JSON files at startup, so a rebuilt Grafana host comes back with the same configuration. For Prometheus, store prometheus.yml and every rules file in version control, and run promtool check rules in CI before a change reaches production. Reviewing these files also gives you a record of who changed a threshold and why, which auditors and on-call engineers both ask for later.
Sizing and retention need the same discipline. Prometheus keeps local data in two-hour blocks with a 15-day default retention, configurable by time or size, and its storage is single-node, not clustered or replicated. Set retention explicitly, watch disk growth against the number of Ceph OSD and RGW series you scrape, and send long-term history to remote storage if you need it. For Grafana, back up configuration by database: SQLite is the default, but the installation guidance says it does not scale for production, so a team that moves to MySQL or PostgreSQL should back up that database on a schedule and test a restore. Finally, repeat your alert tests after every Ceph upgrade, because metric names and health codes can change between releases.
Trade-offs between built-in and external monitoring
Cephadm can deploy its own Prometheus, Grafana and Alertmanager, but many organizations already run a monitoring platform. The four options below differ in who operates the metric store, where the data sits and who gets paged, so the choice should be made before alert routing is built.
Cephadm-managed monitoring stack
This is the Prometheus, Grafana and Alertmanager set that cephadm deploys alongside the cluster as containers. It fits small to mid-sized clusters where the storage team owns monitoring end to end and wants dashboards that match the Ceph release. The team must patch these containers with the cluster, size the Prometheus disk for retention, and decide whether the single Prometheus instance is acceptable. Confirm that metrics and dashboards stay on hosts inside your boundary and who holds Grafana admin access.
Existing enterprise Prometheus
Here Ceph exporters are scraped by the Prometheus your platform team already operates, and alerts route through its Alertmanager. It fits organizations that want one alerting policy across servers, Kubernetes and storage. The cost is in scrape configuration, label conventions and keeping Ceph dashboards and rules in your own repository. Ask who owns the Prometheus, how long it retains storage metrics, and whether its Alertmanager cluster can reach the webhooks your on-call tooling expects.
Hosted Grafana Cloud option
A hosted service stores metrics and runs dashboards outside your data centers, which removes local storage sizing and upgrades from your team. It fits teams with limited operations staff and no regulatory bar on sending infrastructure telemetry to a third party. Pricing has a fee structure that scales with usage, so ask for a quote based on your series volume. Confirm the region where data is stored, what Ceph metric names leave your network, who can read them and how exporter credentials are handled.
Separate observability platform
A dedicated observability product, often run by a central team, may ingest Ceph metrics, logs and traces together. It fits organizations that already standardize on one platform for correlation across applications and storage. The team must map Ceph exporters into that platform's data model, keep Ceph-specific alert rules working after vendor changes, and confirm that its storage and retention meet your policy. Check where raw data is kept, which administrators can export it, and whether Ceph health alerts still reach the storage on-call rota if the platform is unavailable.
