Why large Ceph RGW buckets slow down and when to plan for sharding
When a bucket grows toward hundreds of millions of objects and listings or writes start to lag, the usual cause is that one bucket index has outgrown the way it was laid out. Bucket index sharding is the mechanism that spreads that index across several RADOS objects so that updates and listings are no longer funneled through a single one, and planning the shard count before the bucket fills is what avoids a disruptive resharding later. The Object Gateway (RGW) keeps an index for each bucket that maps object names to their metadata. Every object written, overwritten or deleted adds an entry or changes one, so the index grows with object count, not with the bytes stored. A bucket holding many small objects can therefore have a heavier index than a bucket holding fewer large ones of the same total size.
Teams usually need this topic when the bucket is expected to reach very large object counts, when a workload writes or deletes continuously into one bucket, or when listings of the bucket are part of a production path such as batch jobs or analytics. Early symptoms include slower LIST calls, rising latency on puts and deletes, and index objects that become hot on a few OSDs. Before the first production write, ask the team for a realistic object count at twelve and at thirty-six months, the object size distribution, the write and delete rate, and whether the bucket will be replicated to another zone. Those figures decide the starting shard count and whether automatic resharding is enough. The exact thresholds for objects per shard and the default behavior depend on your Ceph release, so confirm them against the RGW documentation for the release you will run before setting values.
The rest of this topic covers the sizing and resharding decisions in order, so the practical questions to settle now are these: who owns bucket naming and creation, whether buckets can be split by workload instead of growing without limit, and which team will monitor index health once the bucket is in use. Changing the layout of a bucket that already holds data is possible but costs time and needs a window, so a short design review before go-live is usually cheaper than a resharding project after the first hundred million objects. A partner can run that review against your object profile and the cluster you have, but the numbers must come from your own workload measurements rather than from general guidance.
Configuration options for shard count and resharding behavior
This table compares four ways to control index shard count for a large RGW bucket: dynamic resharding, manual resharding, a fixed minimum shard count and prime shard counts. Read each row as a decision: when the option fits, what it changes in the cluster, and its main trade-off. Confirm every setting name and default against the Object Gateway documentation for your release before you change it in production.
| Option | When it fits | What it changes in the cluster | Main trade-off |
|---|---|---|---|
| Dynamic resharding | Buckets that grow unpredictably and where index latency can absorb occasional background reshards | RGW splits the bucket index into more shards as object counts cross configured thresholds, without an operator-run step | Reshard work runs during normal operation and can add load and latency while it progresses |
| Manual resharding | Known migrations or bulk loads where you can schedule the reshard outside peak hours | An operator triggers the reshard for a named bucket to a chosen shard count | Requires monitoring and a runbook, and a poorly timed reshard can affect client traffic |
| Fixed minimum shard count | Buckets you know will reach hundreds of millions of objects from day one | New buckets start with a higher shard count, so the index is spread across more objects before growth | Higher index overhead per bucket, which matters when many small buckets exist |
| Prime shard counts | Large buckets where a prime shard number reduces hash clustering across index shards | Sets the shard count to a prime value, which can spread keys more evenly across shards | Adds planning work and the chosen number must still fit your object-per-shard target |
| Object-per-shard target | Teams that want a predictable index size per shard and alert thresholds | Drives the shard count your reshard policy aims for as object counts rise | The target is a sizing choice, not a fixed Ceph default, so it must be validated on your hardware |
| Multisite behavior | Buckets replicated across zones where reshard events must stay consistent | Reshard activity coordinates with metadata sync between zones | Reshards can lag on secondaries and need checks before failover, so confirm the multisite steps for your release |
How to plan and verify resharding on a large bucket
This sequence takes a bucket from unknown object counts to a sized, verified index layout, so that listings and writes stay fast as the bucket grows. Each step ends with a check that tells the team whether it can move on.
Measure object counts and index health
The storage team records the current object count, shard count and index size for each large bucket, using the bucket statistics and index status commands in radosgw-admin. Confirm the exact subcommands and output fields in the radosgw-admin documentation for your release before running them on production buckets. Note which buckets already show large per-shard object counts or warnings about large omap objects in the cluster health output. The step is done when every large bucket has a recorded baseline in a shared document that the application owners have reviewed.
Choose the shard strategy before data lands
Before a new bucket receives its first large upload, the platform team agrees on an expected object count and a target objects-per-shard figure with the application owners. Use the sharding guidance in the Object Gateway documentation for your release to confirm the default thresholds and the minimum shard count you can set at bucket creation. Choose a shard count that is a prime number where the documentation recommends it, and record the reasoning next to the bucket's owner and expected growth. The step is done when the agreed numbers are written down and the bucket creation parameters match them.
Enable or tune dynamic resharding
The team checks whether dynamic resharding is enabled in the cluster configuration and whether the thresholds fit the baseline from the first step. Changes to the relevant configuration options are made through the central configuration store, and each option name and default must be confirmed in the Object Gateway documentation for the running release. Apply the change to a lab cluster first, and avoid tuning thresholds on production until the lab run has shown the expected behavior. The step is done when the effective settings are read back from the cluster and match the approved values.
Run a manual reshard in a test bucket
On a copy of a representative bucket in a non-production zone, the operator starts a manual reshard to the target shard count, then monitors the reshard queue and the index status until the operation completes. Record the time taken, the object count before and after, and any errors reported in the Object Gateway logs. Confirm the exact reshard commands, queue options and cancellation procedure in the radosgw-admin documentation for your release. The step is done when the test reshard finishes without errors and the team has a documented procedure that another engineer can repeat.
Verify listings and writes after the change
After the test reshard, the operator runs full and paginated listings, reads a sample of known keys, and writes and deletes test objects through the same S3 client libraries the applications use. The bucket's object count reported by the statistics command should match the count recorded before the change. Compare listing results and response times against the baseline, and check the Object Gateway logs for errors during the window. The step is done when the listings are complete, the sampled reads and writes succeed, and the owners sign off on the test result.
Schedule production resharding with a rollback plan
The team applies the verified procedure to production buckets in an agreed change window, one bucket at a time, starting with the bucket that has the highest objects-per-shard figure. Before each reshard, the operator captures the current statistics and confirms that enough free capacity exists for the temporary index copies, using the cluster capacity figures and the release documentation. Afterward, the same listing and write checks from the previous step are repeated. The step is done when each production bucket shows the target layout, its checks pass, and the baseline document is updated with the new values.
Resharding in multisite zones and what changes during the work
In a multisite configuration, a reshard is not only a local index change. Bucket index metadata is part of the zone configuration that replicates between zones, so the zone that performs the reshard and the zones that replicate that bucket all have to be considered before you start. Plan the work so that the bucket is treated as a change window in every zone that holds it, and confirm with the Object Gateway multisite documentation how your Ceph release handles reshard operations on replicated buckets, since behavior in this area has differed between releases. The Ceph releases index is the place to check which release you are on before reading that guidance.
The order matters more than the command. Start by recording the current shard count, the object count and the sync status of the bucket in every zone, using the sync status tooling described in the multisite documentation. Confirm that replication is caught up before the reshard begins, because a backlog makes it hard to tell later whether a discrepancy came from the reshard or from ordinary lag. Then reshard in the zone you intend to change, wait for the index to settle, and verify object counts and listings in the secondary zones before you consider the work finished. Keep a written rollback path for the window, and do not start a second reshard on the same bucket until sync has returned to a clean state.
Confirm the following with the Object Gateway multisite documentation before touching a production bucket: how the reshard interacts with sync for your release, whether writes to the bucket must pause or can continue during the operation, how the reshard status is reported in each zone, and what your team should monitor while sync catches up. Failover planning for a multisite zone is covered on a separate page, so keep this work to a single bucket reshard and do not combine it with a zone promotion or period commit. Test the full sequence on a non-production bucket in a lab zone pair that mirrors your topology before you schedule the change.
Security and operational risks during large-bucket changes
Resharding touches the bucket index that every read, write and listing depends on, so access, recovery, visibility and client behavior all need to be checked before and after the change. The cards below cover the controls that connect those risks and the tests that show whether a bucket is ready for a change in production.
Access policies through a reshard
A reshard changes how bucket metadata is laid out, but bucket policies, ACLs and IAM policies attached to users, groups and roles should be unaffected. Before the change, export the bucket policy and the policies of the principals that use it, then confirm that a test user with a known set of allowed and denied actions gets the same results afterward. Ask your team to record which IAM accounts and presigned URL holders depend on the bucket, because those paths fail in ways that look like application bugs.
Index state backup and restore
The bucket index maps object names to their storage, so losing or corrupting it can make objects appear missing even when their data is intact. Take a backup of the index objects, or at minimum a documented list of the bucket's index pool and shard layout, before a manual reshard. Decide in advance how you would restore that state and who approves the restore, and rehearse it on a non-production bucket with the same layout.
Slow requests and index growth
Watch request latency on the bucket, the number of objects per index shard and slow request warnings in the cluster health output during and after a reshard. A shard that keeps growing between checks is a sign the sharding plan no longer fits the bucket. Set alert thresholds with your own team based on a baseline you measured, rather than copying values from another cluster, and check that the alerts reach the team on call.
Change windows and load
A reshard moves index entries between shards, which adds I/O on the index pool and can slow client traffic on the same bucket. Schedule the change for a window when write volume is lowest, and agree a rollback point with the application owners before it starts. Record who can stop the operation and what state the bucket is left in if it is interrupted, because a partial reshard needs a planned recovery path.
Client retry behavior
Applications that retry failed S3 requests with short, fixed delays can add load during a reshard and turn a slow period into a backlog. Confirm that each client SDK uses exponential backoff with a retry limit, and that writers handle timeouts without creating duplicate objects. Test the actual SDKs and tools your applications use, since defaults differ between libraries and versions.
Testing with production-like counts
A reshard that completes in a test bucket with a few thousand objects tells you little about a bucket with hundreds of millions. Build a test bucket with a comparable object count, key naming pattern and listing workload, then measure reshard duration, latency and index behavior against it. Ask your team to document which differences remain between the test and production setups, since those gaps set the limits of what the test proves.
Questions teams ask before resharding large buckets
Who does the resharding work, and how much effort does it take?
The work splits between planning, execution and follow-up, and each part needs different people. Planning covers object counts per bucket, the current shard layout, client access patterns and whether any bucket sits in a multisite zone group. Execution is mostly waiting on background work, but engineers spend that time watching queues, client error rates and cluster load. Recurring effort comes from tracking bucket growth, because a bucket that outgrows its shard count needs another reshard. Ask for a quote that separates one-time planning and execution from any retainer for ongoing growth checks.
How long does a reshard take, and what drives the duration?
Duration depends on the bucket's object count, the number of shards involved, cluster load and how much client traffic continues during the change. No single figure applies to every cluster. The Ceph hardware recommendations advise benchmarking before buying because no two clusters are alike, and the same reasoning applies to a reshard window. Main drivers include index pool performance on SSD-backed metadata, whether recovery traffic and client I/O share the same network, and memory headroom during peak load. Ask for a timed rehearsal on a copy or a non-production bucket of similar size before you commit a production window.
Can an internal team reshard large buckets without outside help?
A team that already runs RGW under cephadm and has an engineer comfortable with Ceph administration can carry out a reshard. The main risk sits in preparation rather than in the command sequence. Teams need to know the current index layout, the RGW release in use, whether dynamic resharding is active and how client applications react to brief slowdowns. Outside help often adds value in the analysis, the rehearsal and the written rollback plan. Confirm internally whether anyone has run this procedure in production before and whether on-call staff can monitor the window.
What does the resharding work need from our team before it starts?
The work needs four inputs before it can be scoped properly:
- Object counts and growth rate for each bucket in scope, including the largest buckets.
- Access patterns: which applications write, list or delete, and whether they use versioning or lifecycle rules.
- Change windows, freeze periods and named contacts for each application owner.
- Cluster access, the RGW release, and the multisite topology if any zone group is involved.
Also confirm who can approve a pause if client error rates rise during the change. Delays from application owners often shift the schedule more than the technical steps do.
Is resharding reversible if something goes wrong during the change?
Treat the change as one you must be able to recover from, and plan that recovery before the window opens. Whether an interrupted reshard can be rolled back depends on the RGW release and the stage the operation has reached, so confirm that behavior for your release in the Ceph Object Gateway documentation. Keep a verified copy of critical data, record object counts and index state before and after, and rehearse the recovery steps on a copy of the cluster state where possible. Write down who makes the decision to stop and roll back.
How should we scope a quote for a bucket resharding project?
Ask for a written scope that names the buckets in scope, the target shard approach, the environments covered, the rehearsal and rollback work, change window support and the handover documents. Most DeployOpen engagements are fixed-scope, so scope, deliverables and timelines are agreed in writing before work starts, and the quote follows from that scope. Separate one-time work from recurring items such as growth monitoring or a support retainer for later reshards. Ask which client-side changes and approvals the quote assumes, so you can compare offers on the same basis.
How DeployOpen scopes and supports large-bucket work
DeployOpen scopes large-bucket work as a fixed-scope engagement agreed in writing before any change is made to the cluster. The starting point is measurement, not assumption: the object count per bucket, the growth rate, the ratio of listing and write operations, the number of bucket index shards currently in use, and the latency your applications see on list and put calls. Those figures determine whether the right answer is a manual reshard, a change to the default shard count for new buckets, a dedicated bucket layout for the largest workloads, or simply monitoring and a documented threshold. The written scope names which buckets and zones are covered, which settings change, the test plan, the rollback steps and the evidence the client receives at handover. Ask for the measurements to be listed in the scope document so the quote can be checked against them.
Delivery happens inside your own Ceph cluster, in your environment and under your change control. DeployOpen works with your storage, platform and application owners to apply the agreed changes, run reshard operations in a maintenance window you approve, and record the result, including the bucket index state before and after, the Ceph health output and the client tests that were run. Where multisite is involved, the scope should state how the secondary zones will be handled and who approves each step, because a reshard on a bucket that replicates across zones needs its own planning. Confirm with your Ceph version and vendor documentation which resharding behavior applies to your release before the work starts, and ask for that check to appear as a named deliverable.
Ongoing support is available as a retainer for clusters that keep growing after the first change. A retainer typically covers monitoring of bucket index growth, review of new large buckets before they go live, planned resharding during agreed windows and access to DeployOpen engineers when a bucket slows down. The cost of a quote depends on the number of buckets in scope, the number of zones, the amount of testing your applications need and how often your data grows. DeployOpen does not publish a price list, so a quote follows from the written scope. Ask each bidder to separate one-time work from recurring support and to state what a change request costs once the initial scope is accepted.
