When one CephFS metadata server becomes the bottleneck
Multiple active MDS daemons split CephFS metadata work across several ranks, each rank served by its own active metadata server. You need this when a single active MDS is saturated while handling metadata operations (opens, creates, lookups, attribute changes) from many clients, or when many separate directory trees are busy at once. The benefit is largest when the workload is spread across many clients or many directories. A single application running on a single client often sees little gain, because that one client's metadata requests still reach the same rank and the bottleneck stays where it was. Before changing anything, confirm the symptom: check whether the active MDS is CPU-bound or its request queue is growing during slow periods, and whether the slow operations are metadata rather than data I/O.
The main setting is max_mds, which controls how many ranks are active. The practical maximum for a highly available system is one less than the total number of MDS servers, because the remaining server must stand by to take over a failed rank. A cluster with only one spare daemon therefore has little headroom to add ranks without giving up failover. Plan the number of standbys as part of the change, not as an afterthought, and confirm the current count of daemons and standbys in your cluster before raising max_mds. Ranks also need the right client and workload profile to pay off, so a rise in rank count does not fix a metadata problem caused by one hot directory.
Once several ranks are active, you decide how directories are assigned to them. Static pinning with ceph.dir.pin places a chosen directory tree on a specific rank, which suits home directories or per-tenant trees with predictable load. Ephemeral pinning, with distributed or random policies, spreads subtrees automatically. The dynamic balancer, which moves load between ranks, is off by default, so you must enable and test it deliberately. Ask your team to measure per-rank load under real traffic before and after each change, and to document which trees are pinned and why. Your upgrade and maintenance plans should also cover how ranks are reduced or restarted, since those steps move metadata work between daemons.
Configuration options and when each fits
This table compares the main ways to spread CephFS metadata load across ranks and standbys, with the workload each suits and its main trade-off. Read each row against your client count, directory layout and tolerance for manual placement work. The main takeaway is that more active ranks only help when the load is spread across many directories or clients, and pinning or balancing choices decide whether that spread holds.
| Option | Best fit | Trade-off |
|---|---|---|
| max_mds greater than 1 | Many clients or many separate directories generating metadata load on one file system | Single-application workloads on one client often see little benefit; the practical maximum in a highly available setup is one less than the number of MDS servers |
| Plain standby MDS | Failover capacity when no rank is ever idle and you need a warm takeover target | Standby holds no cached metadata, so takeover after a failure can take longer than with standby-replay |
| Standby-replay MDS | Shortening failover for a specific rank that must recover quickly after a daemon fails | Consumes memory and resources on a daemon that serves no client traffic; confirm sizing with your vendor documentation and tests |
| Static pinning with ceph.dir.pin | Home directories or tenant trees where you want a known rank for each subtree and predictable behavior | Someone must place and review pins; a hot directory pinned to one rank stays on that rank regardless of load |
| Distributed ephemeral pin | Large trees with many subdirectories where automatic spread is wanted without manual pins | Placement is hash-based, so one very busy directory can still concentrate load on one rank |
| Random ephemeral pin | Workloads with many subtrees that should be scattered across ranks to avoid clustering | Locality is lost, so related directories may land on different ranks and cause more cross-rank traffic |
| Dynamic balancer | Changing workloads where you prefer the cluster to move load without redesigning pins | Off by default; enabling it moves subtrees automatically, so test its behavior under your load before relying on it |
How to raise max_mds safely in a running cluster
This sequence moves a CephFS file system from one active metadata server rank to several without losing access, leaving standbys ready to take over any failed rank. Each step ends with a check so the change can be stopped before it reaches clients.
Confirm the release and current MDS layout
Record the Ceph release running on every host, the file system name, the current max_mds value, and the number of active ranks and standby daemons. Run the status and file system dump commands on the cluster and save the output with the change record. Confirm the exact commands for the deployed release in the Ceph multiple active MDS documentation before you run them. The step is done when you can state the total number of MDS daemons, how many are active, and how many are standby, and when a second engineer has checked those numbers.
Provision standby metadata servers
Add MDS daemons so that the total count leaves at least one standby beyond the number of active ranks you plan to run. Where the cluster is managed by cephadm, deploy the daemons through the orchestrator service specification so that placement, restarts and upgrades follow the same path as the rest of the cluster. Place the new daemons on separate hosts where your layout allows it. The step is done when the new daemons report as standby and the cluster health output shows no new warnings about them.
Set max_mds within the safe limit
Set max_mds on the file system to a value no higher than one less than the total number of MDS servers, so that at least one standby is always available to take over a failed rank. Make the change in a quiet window and watch the rank state while it applies. Ranks start up in sequence, and clients may pause briefly during each transition. The step is done when the file system reports the new number of active ranks and each rank shows as active, with no rank stuck in a replay or resolve state.
Verify ranks and standbys are healthy
Check that every active rank is serving, that the standby daemons are registered and not reusing a rank that failed to start, and that the cluster health output has no metadata server warnings. Also confirm that client sessions are spread across the ranks and that no rank is carrying a disproportionate load. If a rank remains in a failed or replay state, stop here and resolve it before continuing. The step is done when the rank table and health output match the plan from step one.
Pin chosen directories to ranks
Apply ceph.dir.pin extended attributes to the directories that need a fixed rank, such as a large project tree or a tenant root, and leave other directories to the default placement. Set pins on one directory at a time and record the rank each one maps to. Be aware that pinning concentrates load on the rank that owns the directory, so choose pins that match the traffic you expect. The step is done when each pinned directory reports the expected rank value and a test client can read and write under it without errors.
Test failover of one active rank
Fail one active rank in a planned test, using the fail command for that rank, and confirm that a standby takes it over. Measure how long client operations stall and check that the taken-over rank replays its journal and returns to active. Repeat the test only after the cluster is healthy again. If no standby takes the rank, reduce max_mds before you go further and review the standby count. The step is done when the failed rank is active again under a standby or the original daemon, the file system is writable, and the test result is recorded.
Pinning choices for home directories and multi-tenant trees
Home directories produce many small, independent subtrees, one per user, which usually spread well across ranks. Distributed ephemeral pinning is normally the first option to test. Set it on the parent directory that holds the user folders so the MDS places each child subtree across ranks without an operator decision for every new account. Static pinning works, but it turns each new user into a placement task. Random ephemeral pinning also spreads load, but it moves subtrees unpredictably, which makes cache behavior harder to reason about for interactive users.
- Pin the parent directory: set
ceph.dir.pin.distributedon the folder that contains user homes so new homes are placed automatically. - Record per-rank load first: capture request rate, session count and cache use per rank with
ceph fs statusand the MDS performance counters before you change anything. - Look for heavy users: a few users running metadata-heavy jobs from their homes can still concentrate load on one rank, so check per-client session activity.
- Keep standbys available: pinning does not replace failover, so confirm that enough standby daemons exist to take over a failed rank.
- Remember the balancer is off by default: do not assume the dynamic balancer will correct skew unless you enable and test it.
Before the change, confirm the active rank count, the client mix and whether users run metadata-heavy jobs from their home directories, then compare latency and rank load after a representative period of normal use.
In a multi-tenant tree, each tenant owns one subtree, and the main goals are isolation and predictable performance for each tenant. Static pinning fits best when the number of significant tenants is small enough to assign deliberately to ranks, and when a tenant's workload is large enough to justify its own rank. Each tenant pin is then an explicit, auditable decision. Distributed ephemeral pinning suits the long tail of small tenants, where automatic placement is more practical than manual assignment. Many deployments combine both modes within one tree.
- Pin large tenants statically: set
ceph.dir.pinon each major tenant subtree to a chosen rank and record the mapping in your runbook. - Use ephemeral pins for small tenants: apply distributed pinning to the parent of the small tenant directories so they are placed automatically.
- Measure per-tenant metadata operations: track operations and latency by tenant so you can see whether one tenant affects others on the same rank.
- Plan for rank failure: a pinned subtree moves to a standby when its rank fails, so confirm that standbys can carry the largest tenant's load.
- Check the number of ranks against the tenant count: a static mapping with more tenants than ranks forces several tenants to share a rank, which should be a deliberate choice.
Before the change, document each tenant's metadata activity, then after the change confirm that no rank carries a disproportionate share of requests and that tenant latency stays within the agreed expectations.
A single large project directory is the case where pinning is least likely to help. Multiple active MDS daemons benefit most from many clients or many separate directories, and a single application on a single client often sees little gain. Pinning one directory to one rank moves the whole load to that rank without splitting it. Distributed ephemeral pinning can help only when the project contains many child directories that the workload touches in parallel. If the project is one flat directory driven by one client group, the first step is to examine the client pattern rather than the pin.
- Test the client pattern first: determine whether the load comes from many clients or from one application, because the answer decides whether extra ranks can help.
- Check the directory shape: pinning only spreads work across child directories, so a flat directory gains nothing from distributed pinning.
- Review application metadata calls: frequent
statoropencalls from a single job can often be reduced through client-side changes. - Keep `max_mds` at a level you can operate: adding ranks increases coordination and standby requirements, so increase it only when the measurements justify it.
- Confirm the limit: in a highly available setup, the practical maximum for
max_mdsis one less than the number of MDS servers.
Before any change, record requests per second and latency for the project, then compare them after the change to confirm that the work actually spread across ranks.
Security and operational considerations for MDS changes
Raising max_mds or adding pins changes who can reach which metadata and how much memory the daemons need. The cards below cover the four controls that keep a multiple active MDS layout secure and predictable: scoping client capabilities, sizing MDS hosts, watching rank and standby state, and governing changes to max_mds and pins.
Scope client capabilities per path
Every CephFS client holds a capability string that controls what it can read and write, and the MDS enforces it on each request. When you split a tree across ranks, a client with broad access can reach every pinned subtree, so one tenant's workload can touch another's metadata. Create one CephX identity per tenant or share, limit it to its own path, and review the caps with ceph auth get after each pin change.
Size MDS hosts for peak memory
An MDS holds a large part of its metadata cache in memory, and each active rank needs its own cache. Ceph's hardware guidance says to size memory for peak use rather than calm periods, so a rank that looks idle at night can exhaust memory during a backup or a large build. Place MDS daemons on dedicated hosts where possible, or confirm the co-located services leave room for the cache of every rank that could fail over to that host.
Monitor rank health and standbys
A rank without an active daemon stops serving its subtrees, and clients see hangs rather than errors. Track each rank's state, the number of standby daemons, and whether any rank has failed over recently. Alert when standby count drops below the number your failover design needs, so a single host failure does not leave a rank unassigned. Check ceph fs status as part of routine health review.
Keep standbys above the failover margin
The practical ceiling for max_mds in a highly available system is one less than the total number of MDS servers, because standbys take over failed ranks. Setting max_mds to the full server count leaves no spare for failover. Before raising max_mds, confirm the cluster has enough MDS daemons to keep at least one standby, and confirm that the standby is placed on a different host from the active rank it would replace.
Govern changes to max_mds and pins
Changing max_mds reassigns ranks and can move subtrees between daemons, and a wrong pin can concentrate a busy directory on one rank. Treat both as production changes: record the current ceph.dir.pin values and max_mds, test the change on a non-production file system, schedule it in a window, and keep a rollback command ready. Ask your partner or internal team to document who approves each change.
Questions teams ask before scaling CephFS metadata
Is one metadata server enough for our CephFS workload?
One active metadata server is enough for many CephFS deployments, and the answer depends on where the load comes from. Multiple active MDS daemons help most when many clients or many separate directories generate metadata requests at the same time. A single application running on a single client often sees little benefit, because that workload is served by one rank regardless of how many daemons exist. Before adding ranks, measure client count, metadata operation rates and MDS CPU and cache pressure over a representative period. Ask your team or a partner to confirm which of these actually limits you.
What does the team need to run multiple active MDS daemons?
You need someone who can read Ceph health output, change max_mds, and watch rank states during a change. You also need enough MDS daemons that standbys remain available after a failure. The practical maximum for max_mds in a highly available system is one less than the total number of MDS servers, because standbys take over failed ranks. Your team also needs a clear process for client mounts, directory pinning decisions, and rollback. Operations staff should know how to check ceph fs status and how to respond when a rank is stuck in replay or reconnect.
How long does an MDS scaling change window usually take?
The window depends on your workload, cluster size, client mix and how much testing you do beforehand, so no general duration applies. Raising max_mds is a short configuration change, but the work around it takes longer: baseline measurements, a test cluster run, pinning plans for directories, client behavior checks and a documented fallback. Ask any provider for a schedule that lists each step, the test environment used, and the conditions that would stop the change. Written scope and timelines are agreed before work starts on fixed-scope engagements.
Can our own team scale CephFS metadata without outside help?
Yes, many teams can, provided they already operate Ceph day to day and have a test environment that resembles production. The steps themselves are documented in the Ceph multiple active MDS documentation. The difficult parts are choosing the right number of ranks, deciding which directories to pin, and interpreting balancer behavior under real client traffic. If your team lacks recent Ceph operations experience or cannot test safely, an outside engineer can review the design and run the change with your staff so knowledge stays in-house.
What should a consultant's quote for MDS scaling include?
A useful quote lists the written scope, the assumptions and the deliverables. Ask for a baseline assessment of metadata load, the MDS server count required for your target ranks, the directory pinning design, a test plan, the change steps with rollback, and monitoring checks after the change. Also ask what is one-time work (design, testing, change execution) and what recurs (monitoring review, upgrades, support retainers). Ask which client environments are in scope and what the provider needs from you, such as access, maintenance windows and staff time.
How does the dynamic balancer decide which subtrees to move?
The dynamic balancer moves load between active ranks based on metadata activity it measures, aiming to even out the work each MDS carries. It is off by default, so you must enable it deliberately. Its decisions depend on observed load, which can change quickly, so it may not suit workloads where you need predictable placement. Static pinning with ceph.dir.pin gives fixed placement, while ephemeral pins with distributed or random policies spread directories automatically. Test the balancer against your real access patterns before relying on it, and confirm the exact behavior for your Ceph release in the official documentation.
Does scaling MDS ranks affect client mounts or data safety?
Changing the number of ranks affects metadata routing, so clients and their workloads can see pauses while ranks transition. Data stored in RADOS is not rewritten by changing max_mds, but metadata operations are the critical path. Before the change, confirm client kernel and FUSE versions, check for long-running metadata operations, and take a verified backup of the file system state your recovery plan depends on. Ask your provider how they will watch client sessions and stop the change if rank transitions stall.
What a scoped CephFS metadata engagement should include
A written scope for this work should name each component that changes the metadata layout, because that is what the client pays to have verified. Start with current workload measurements taken from the existing file system: client counts, the number of separate directory trees under load, metadata operation rates and where latency actually appears. The scope should then state the target max_mds and the standby count together, since the multiple active MDS documentation puts the practical ceiling for max_mds in a highly available system at one less than the total number of MDS servers. Standbys are what take over failed ranks, so a scope that sets max_mds without a standby plan has not met the availability goal. Ask the provider to show the arithmetic for the chosen numbers.
The pin design is the largest judgment call and should be a named deliverable with a directory list. It should say which directories receive static pins through ceph.dir.pin, which are left to distributed or random ephemeral pins, and how home directories or per-tenant trees are assigned to ranks. The benchmark plan should describe the workload replayed before and after the change, the metrics captured, and the acceptance criteria, and it should say whether the dynamic balancer is tested or left disabled, since it is off by default. The failover tests should cover killing an active rank under load and confirming a standby takes over. Documentation and handover should be listed as deliverables with named recipients.
Some items are one-time: measurements, pin design, benchmark runs, failover tests and handover. Others recur and belong in a separate, priced line: metadata monitoring, periodic review of pins and balancer behavior as directories grow, and rework after Ceph upgrades. Confirm with the provider which of these are included in the fixed scope and which move to a retainer, and confirm the Ceph release the work targets against the Ceph active releases page before the plan is signed.
