Fixing Ceph monitor clock skew

Why Ceph monitor clock skew needs fixing before it becomes an outage

Ceph monitors must agree on time within 0.05 seconds, so a MON_CLOCK_SKEW warning should be fixed at the host time source, where the drift starts. Raising the allowed drift or muting the warning hides a symptom that stays present while the monitors depend on synchronized clocks. You are most likely reading this because health output names a monitor that has drifted, or because the cluster was built on hosts where no time service was ever configured. Virtual machines are a common source of drift. A guest that is paused, migrated or restored can resume with a clock that no longer matches its peers, and a host rebuilt without its time configuration can start with the wrong offset from the beginning.

This work is worth scheduling in four situations: when you stand up a new cluster, before the first monitor is deployed; after a host reboot or a VM migration that changed how a monitor host keeps time; when ceph health detail first reports the warning; and during any review of cluster host configuration. Confirm in your own environment which time sources each host uses, whether those sources are reachable from every monitor host, and whether a hypervisor time sync setting is overriding the guest clock. Those answers decide whether the fix is a configuration change on each host or a change to how your virtual machines are managed.

The result is a cluster whose monitors agree on time by design, with one reviewable time configuration on every host and a health view that stays clean without exceptions. The work is also easy to verify. A monitor that remains in agreement after a planned reboot shows the fix holds, and a warning that returns after a migration points to a hypervisor or host setting that still needs correcting. Before any change, ask your team or a partner to record the time source for every host in the cluster, so the before and after states can be compared later.

Which time source fits your Ceph hosts

This table compares five common ways to keep Ceph monitor and cluster host clocks aligned, from chrony and ntpd to systemd-timesyncd, an existing enterprise NTP service and a local time source for air-gapped sites. Read each row against your host operating system, your network path to time sources and how tightly your monitors must agree. The main takeaway is that the tool matters less than whether every monitor host can reliably reach a source that stays accurate.

Time source	When it fits	What to confirm	Main trade-off
chrony	New or rebuilt Ceph hosts where you want a maintained daemon that tracks and corrects drift continuously	Your host OS packages it and supports it, the hosts can reach at least one reachable upstream source, and you can read its tracking output on every monitor host	Another daemon to configure and monitor on every host, and its behavior depends on source selection being set up correctly
ntpd	Hosts already standardized on ntpd by your platform team, where the existing configuration is audited and maintained	Your OS still ships and supports ntpd, the configuration matches your org's standard, and monitor hosts agree with each other after changes	Older tooling that many teams are moving away from, so ongoing ownership and skills must be planned
systemd-timesyncd	Small or lab clusters on a stock OS where a simple SNTP client is enough and no dedicated time daemon is wanted	The OS includes it, the upstream servers are reachable from every monitor host, and you have checked that its accuracy suits monitor skew limits	Simpler client with fewer controls, so it is a weaker fit for monitors that must stay tightly aligned under network jitter
Existing enterprise NTP service	Organizations with a central, governed time service that all data center hosts already use	Every monitor host can reach the service over the network path you use, firewall rules allow it, and the service's own accuracy has been reviewed by your team	Ceph depends on a service you do not operate, so an outage or misconfiguration upstream can cause skew across the cluster
Local time source for air-gapped sites	Isolated networks with no route to public or corporate time servers	The local reference clock or GPS or similar hardware is supplied and maintained, and you can confirm how it is synchronized and monitored with your vendor	Requires dedicated hardware or a designated internal reference, plus a documented process for verifying and recovering that source

Why the 0.05 second limit matters and why more tolerance is not the fix

The 0.05 second limit exists because Ceph monitors do more than store the cluster map. They run Paxos to agree on each map change, and they use timestamps when they hold leases and judge the health of their peers. When one monitor's clock runs ahead of or behind the rest of the quorum, those time-based decisions can stop lining up across monitors. Ceph raises a clock skew health warning once the measured offset passes the allowed drift, so the check acts as an early signal before the disagreement affects quorum. The threshold is tight because a fraction of a second is enough to matter for those comparisons, and small time errors are easy to overlook until they show up as elections or quorum trouble.

The tempting fix is to raise the allowed drift, most often through the mon_clock_drift_allowed option. Confirm the option name, its current default and whether it still controls the warning in the Ceph version you run before you change anything, because values from older documentation or forum posts may no longer apply. A higher value silences the warning, but it does not correct the offset. The usual causes sit on the host: chronyd or ntpd not running, no reachable upstream source, a virtual machine whose clock is paused or stepped by the hypervisor, or a monitor host that is too busy to discipline its clock. Those faults continue to affect anything else on the host that depends on accurate time, including log correlation and certificate validity.

Treat a clock skew warning as a host problem first. Check the time daemon status on every monitor host, compare the offsets it reports with the health detail from Ceph, and trace the path to the upstream time servers, including any firewall or routing change that could block them. Record the measured offset before and after each fix so you can confirm the trend is flat. If you ever decide to change the drift setting, treat it as a documented, time-limited exception, and keep an alert on the underlying offset so the host fault stays visible after the warning is muted.

What else breaks when Ceph hosts drift apart

Monitor clock skew is the visible symptom, but the same drift quietly affects other parts of a storage platform. The cards below cover the areas where a skewed host causes confusion or failures, and what to check on each host before you assume the problem is elsewhere.

Log correlation across hosts

During an incident, you rely on timestamps to line up monitor, OSD, manager and gateway logs. If hosts disagree about the time, events appear out of order and the real sequence of a failure becomes hard to reconstruct. Compare the clock on each host against the same reference, and confirm that your central logging platform records the original event time rather than the time it received the message. Check this before you trust a timeline built from several hosts.

Certificate validity windows

TLS certificates carry not-before and not-after dates. A host whose clock sits behind may see a certificate as not yet valid, while a host ahead may see a valid certificate as expired. This affects dashboard access, RGW HTTPS endpoints and any certificate used for encrypted messaging. Check the certificate dates with your own tooling, compare them to the clock on the host presenting the certificate, and confirm the chain on each endpoint your clients use.

Signed requests from clients and gateways

S3 clients commonly sign requests with a timestamp, and the gateway rejects requests whose timestamp falls too far from its own clock. Authentication tokens with short lifetimes behave the same way. Clients that run on hosts outside your Ceph nodes can fail even when the cluster itself looks healthy. Check the clock on application servers, backup tools and CI runners that talk to the Object Gateway, not only the storage hosts.

Scheduled jobs that depend on timestamps

Scrubs, deep scrubs and lifecycle rules decide what to run and when by reading time values and interval settings. If a host's clock jumps, scheduling can shift or repeat work at unexpected times, which can add load during business hours. Lifecycle expiry rules that compare object age with the current time can also behave differently across gateways. Review the scrub schedule and lifecycle configuration after a clock correction, and confirm that the timing matches what you intended.

Gaps and jumps in monitoring data

Prometheus and Grafana store samples against the timestamps the scrape or exporter reports. A host with a drifting clock can produce points that sit outside the expected window, which shows up as gaps, overlaps or spikes on dashboards. Alert evaluation can then fire late, fire twice or miss a short event. Check dashboard timestamps for each host, and verify that alert rules still fire as expected once the clock is corrected.

How to fix Ceph monitor clock skew with chrony

This sequence confirms that clock skew is the cause of a monitor warning, brings every monitor and OSD host onto one internal time source with chrony, and ends with offsets that are inside Ceph's tolerance and a cluster that reports healthy. Run the checks on each host in order so that no host is left out.

Confirm the skew in cluster health output
Start from the cluster, not the hosts. Run ceph health detail from a node with an admin keyring and look for a MON_CLOCK_SKEW entry that names the monitors whose clocks are out of step, and note the offset Ceph reports for each. Record the output before you change anything, so you can compare it after the fix. The step is done when you have a written list of the affected monitors and their reported offsets. Confirm the exact warning text and wording for your Ceph release in the Ceph documentation for your version, because output format can change between releases.
Check the time service on every host
On each monitor host and each OSD host, identify which time service is running, for example chronyd, systemd-timesyncd or ntpd, and whether more than one is active. Two running time daemons can fight over the clock, so disable the extras before going further. Run systemctl status chronyd or the equivalent for your distribution, and record the result per host. The step is done when every host has one known time service, or a clear list of hosts that still need one. Confirm the unit names on your distribution, since packaging differs.
Point every host at one internal source
Choose one reliable internal time source, such as a pair of stratum-appropriate NTP servers your organization already operates, and add the same server lines to the chrony configuration on every monitor and OSD host. Do not mix several unrelated public pools, because differing sources are a common cause of drift. Edit the chrony configuration file for your distribution, usually /etc/chrony.conf or a file under /etc/chrony/, and confirm the source address with your network team. The step is done when every host's configuration names the same sources in the same way.
Enable and start chrony on each host
Enable the chrony service so it starts at boot, then restart it so the new configuration takes effect. Run the restart on one monitor host first, check it, and then repeat on the rest, so a configuration mistake does not affect the whole quorum at once. Confirm the service unit name and any firewall rule for NTP (UDP port 123) on your distribution. The step is done when the service reports active on every host and the configuration survives a reboot.
Verify tracking and sources on each host
On every host, run chronyc tracking and check the system time offset and the reference the host is synchronized to. Then run chronyc sources -v and confirm that your intended internal source is marked as selected or preferred, and that its reachability is nonzero. A source that is listed but never selected means the host is not synchronizing, even though the service is running. The step is done when each host reports the same reference and an offset that is small and stable across several checks. Confirm the exact field names and the meaning of the source markers for your chrony version.
Confirm Ceph clears the warning
After the offsets settle, return to the cluster and run ceph health detail again. Monitor skew warnings can take time to clear because Ceph checks the reported offsets on its own schedule, so repeat the check after a short wait rather than assuming the first result is final. Compare the output with the list you recorded in the first step. The step is done when the clock skew entry is gone and ceph health reports HEALTH_OK, or when any remaining warning is one you can explain and address separately. If the warning persists, check the affected monitor's offset again and confirm that the monitor host is using the same source as the others.
Record the configuration and monitor drift
Save the final chrony configuration, the chosen source list and the verified tracking output for each host in your change record or configuration management system. Add a check for time offset and synchronization state to your monitoring so that drift is noticed before Ceph warns about it. Review the time source after host builds and after any network or NTP server change, since a new host with a different source can reintroduce the problem. The step is done when the configuration is version controlled and an alert or check covers each monitor and OSD host.

Time sync on cephadm hosts versus air-gapped sites

Where hosts can reach a time service, the standard setup points each host at an internal NTP service that itself syncs to upstream sources. cephadm runs monitors, managers and OSDs as containers, but those containers read the clock of the host kernel. cephadm does not run its own time service, so the time daemon on each host, such as chrony or systemd-timesyncd, does the work. Check the host, not the container, and read the cephadm documentation for how your release deploys daemons.

Internal NTP first: Point hosts at two or more internal servers that sync to upstream sources, so one outage does not leave the cluster without a reference.
One time daemon per host: Run a single time service, usually chrony, and disable the others so they do not adjust the clock against each other.
Every monitor host: Confirm each host that runs a monitor is synchronized, not only the bootstrap host.
Firewall rules: Allow NTP traffic from cluster hosts to the chosen servers, and confirm your network team approves the change.
Consistent configuration: Manage time settings with the same tooling as other host settings so new hosts join with the right source.

After any change, check synchronization and offset on each monitor host with chronyc tracking and confirm the offset stays within the 0.05 second threshold across a reboot.

Protecting the time source and the change

Clock skew is often caused by the time path itself: which hosts serve time, how that service is authenticated, and who can change it. The cards below pair each control with a check you can run on your Ceph hosts before the next health warning appears.

Limit who can serve time

Every monitor host should sync from a short, named list of internal time servers, not from whatever pool a fresh install defaults to. Mixed upstream sources are a common reason two monitors disagree by a few hundredths of a second. Run chronyc sources -v on each monitor host and confirm that every source is one you approved, with the expected stratum and reachability. Anything else is a host to fix first.

Confirm time source authentication

Check whether your chrony or NTP version supports authenticated time (for example, NTS or symmetric keys) and whether your internal time servers are configured to use it. Unauthenticated time can be spoofed or quietly altered on the network path, which would move monitor clocks without any local change. Confirm the supported options with your time server vendor and your OS documentation, then run chronyc authdata to see what the local daemon has negotiated.

Control changes to time config

Treat /etc/chrony.conf on storage hosts as managed configuration, the same as the cephadm host spec and firewall rules. Record who changed it, when, and why, and roll changes out one monitor host at a time so quorum is never tested by a bad edit. After any change, verify with chronyc tracking and ceph time-sync-status. Unplanned edits are a frequent source of skew that shows up weeks later.

Alert on offset drift early

Monitor the system clock offset and the chrony tracking values on every Ceph host, not only the cluster health output. Set an alert well below the monitor skew threshold so you see a host trending away from its peers while the cluster is still healthy. Feed these metrics into the Prometheus stack you already run, or into the stack cephadm deploys, and check that the alert reaches a person. Ask your monitoring team which tool and threshold they use for this.

Watch hosts that drift after reboot

A host can sync correctly and then drift after a reboot, a VM migration, or a hypervisor move, if its time daemon is not enabled at boot or it is moved onto a host with a different time source. Check that chronyd is enabled, that it starts before the Ceph daemons, and that the VM tools or hypervisor time sync are not fighting chrony. Reboot a test monitor host in a maintenance window and confirm it returns to synchronized within the time your design allows.

What teams ask about Ceph monitor clock skew

How much effort does fixing Ceph monitor clock skew take?

The effort depends on the number of monitor and OSD hosts, whether time sync is already running and monitored, whether the hosts sit in an air-gapped network, and how strict your change windows are. A small cluster that already points at a corporate time source is usually a bounded change: confirm the time service on each host, correct the source, and verify offsets. An air-gapped site needs an internal time source designed first, which adds planning and approval steps. No duration is set here, so ask any partner to state effort per host and per site in the written scope, along with the assumptions behind it. Confirm whether the quote covers root cause investigation or only the time configuration change.

Can our own team fix Ceph clock skew without a partner?

In most environments, yes. The core work is running a time service on every Ceph host, pointing all of them at one agreed upstream source, and confirming that the cluster no longer reports skew. Teams that already run chrony or NTP across their fleet often need only to confirm that monitors use the same source as other servers. Bring in a partner when time comes from a virtualization layer, when hosts are isolated from public and corporate sources, or when skew returns after the fix. The deciding question is whether an engineer on staff can read offsets from each monitor and change time configuration under your change control process.

What do we need to give a partner for a clock skew engagement?

Prepare the access and facts the work depends on:

Shell access to every monitor host and to the OSD hosts, with sudo rights for time service changes.
The time sources in use today, including any corporate NTP servers, GPS appliances, or hypervisor time settings.
The cephadm-managed host list and the network paths from each host to the chosen time source.
The current ceph health detail output and any monitor logs from the incident.
A change window agreed with operations, with a rollback plan for the time service change.

A written scope should name who provides each item and what happens if an item is missing.

Is it safe to mute the Ceph clock skew warning?

Muting is safe only as a short, documented step while the time source is being fixed. A mute hides the warning but leaves the clocks wrong, and skewed monitors can still affect quorum behavior and make logs harder to correlate. Use ceph health mute with a time limit, an owner, and a written reason, and check the mute expires. Do not treat a muted warning as resolved, and do not let a mute carry over into routine maintenance. Your monitoring should still show the skew metric so the condition stays visible while the warning is silenced.

Does raising the clock drift setting fix monitor clock skew?

No. The mon_clock_drift_allowed option sets the threshold at which monitors report skew. Raising it changes when the warning appears, but the clocks stay out of step, and the monitors still disagree about time. Treat a higher threshold as a temporary measure only if an engineer has recorded why it was needed and when it will be reverted. The durable fix is to correct the time sources and confirm offsets are small on every host. Ask any partner to explain how they will handle the setting during the change and whether it will be reset afterward.

How do we confirm the clock skew fix has held?

Check at three levels. On each host, confirm the time service reports that it is synchronized to the expected source with a small offset, using the tracking and source commands for chrony. In Ceph, confirm that ceph health detail no longer lists a clock skew warning. Over time, watch the monitor clock skew signal in your monitoring stack and confirm it does not return after reboots, host maintenance, or network changes. Ask a partner to include these checks in the written scope, along with a post-change review date and an alert rule that fires if the warning comes back.

What else breaks in a Ceph cluster when clocks drift?

Beyond the monitor warning, drift makes log correlation unreliable, which slows incident response across monitors, OSDs, and gateways. Any authentication or token mechanism that depends on time windows can also fail in unexpected ways, and TLS certificate validation can reject connections when a host clock is far off. Check the Ceph documentation for the version you run to confirm which components depend on time in your deployment, and test authentication paths after the fix. Review application and client logs too, since they may timestamp events differently from the cluster.

What to confirm before you change time sources

Changing time sources on a running Ceph cluster touches the monitors that hold its quorum. Confirm the items below in your own environment before you edit any configuration, so the fix does not create a second incident.

Ceph release and drift settings

Confirm which Ceph release the cluster runs, then check the clock drift settings that release supports before you adjust any threshold. Monitor tolerance is a setting you should verify against the documentation for your release, not assume from older guides. Ask the cluster owner for the output of the health checks that report skew, so you have a baseline to compare after the change.

Host OS and time daemon version

Record the operating system and the chrony version on every monitor host, since they determine the available configuration keys and default sources. Mixed versions across hosts can produce different offset behavior, so confirm the image and package baseline with the platform or OS team. Check whether hosts already run a different time daemon that would conflict with chrony.

Enterprise NTP access to the storage network

Ask the network or infrastructure team whether the existing enterprise NTP service may be reached from the storage network. Some segments block outbound UDP port 123 by policy, and a monitor host that cannot reach a stable upstream will drift again after the fix. Confirm which internal servers are approved as upstream sources and who owns their stratum and monitoring.

Monitor and manager placement

Confirm which hosts run monitors and managers, and whether any monitor sits on a host that also runs heavy workloads. Changing time sources one monitor at a time keeps quorum intact, so the placement list defines the order of work. Ask the cluster owner to confirm the current quorum membership before any host is touched.

Time-dependent clients and gateways

Identify any client or gateway that depends on time-based signing, such as S3 request signatures or token validity windows, and confirm its own clock tolerance. A correction to a monitor clock does not fix a client whose clock is off. Ask the application or gateway owner to name the systems that validate timestamps against the cluster.

Change freeze and restart windows

Ask the change board whether a freeze covers restarting chrony, cycling monitors, or running recovery traffic. Monitor restarts can briefly affect quorum, and clock steps can trigger alerts, so the window should be agreed before work starts. Ask the operations lead to confirm the rollback steps and who approves a pause if health worsens.

How DeployOpen scopes and delivers clock fixes on Ceph

DeployOpen treats monitor clock skew as a fault in the host time layer, not as a Ceph setting to tune away. We plan the time sources, build and verify them on your Ceph hosts inside your own infrastructure, and hand over the configuration and checks your team needs to keep the monitors in agreement.

How we scope

We review current health output, monitor logs and host time status to confirm skew is the cause before changing anything.
The written scope lists which hosts, time sources and network paths are in scope, and which are excluded.
We agree the verification steps and acceptance checks for monitor health in the written scope before work starts.
Engagements are fixed-scope, with deliverables and timelines agreed in writing before the work begins.

How we build

We configure chrony or your approved NTP service on each Ceph host, using the time sources your environment allows.
We check that time sources are reachable across the monitor network and that firewall rules do not block them.
We test skew with the Ceph health checks and time status commands on every monitor host before closing the change.
We look for side effects, such as OSD heartbeat or log timestamp problems, and record anything found in the build notes.

How we hand over

You receive the final time service configuration, host inventory and the exact checks used to confirm synchronization.
We document how to add a new monitor host so it inherits the same time sources and passes the same checks.
We walk your operations team through the alert conditions and the steps to take if skew returns.
Runbooks are written for your environment so your team can repeat the checks without DeployOpen involved.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer for Ceph clusters you run.
A retainer can cover periodic review of time synchronization after host changes, network changes or Ceph upgrades.
Ask any provider to confirm in writing how they handle time source outages, escalation and access to your hosts.