When Ceph stretch mode fits a two data center design
Choose stretch mode when you want one Ceph cluster and one storage namespace spanning two data centers, and you can place a third, independent location for a tiebreaker monitor. If your applications can accept separate clusters and a recovery point that lags behind the primary site, two clusters with RBD mirroring is usually the simpler design. The deciding factors are how your workloads must behave when a site fails and the quality of the network between sites, because stretch mode depends on that link for synchronous writes. Ask your storage and application owners what must keep running, and for how long, if one building goes dark.
Stretch mode places monitors across the two sites and adds a tiebreaker monitor at a third location. When the sites lose contact, the tiebreaker helps decide which side keeps quorum, so the cluster does not become two writable halves. Pools in stretch mode use size 4, with copies placed in each data center. After one site is lost, the cluster can enter a degraded stretch mode, and once the failed site returns, it moves through recovery. Monitor placement and CRUSH site buckets are part of the design work, so confirm the exact steps for your release in the cephadm documentation and the stretch mode guidance it references.
Before you choose, list what each design asks of your team. Stretch mode needs a third location for the tiebreaker, a dependable low-latency link between the sites, and a tested procedure for entering and leaving degraded mode. Two clusters with RBD mirroring need a second cluster, failover and failback steps your team runs itself, and an agreed recovery point. Ask any implementation partner to show a written failover test for each option, run against your own link speeds and pool layout, before you accept either design.
Stretch mode, RBD mirroring or separate clusters
Stretch mode keeps one cluster spanning two sites, so client writes depend on the round trip between them. That makes inter-site latency the first measurement to take. Independent clusters with asynchronous mirroring keep client I/O local to each site and send changes to the other site afterward, so the link affects how far behind the second site runs rather than how fast local writes complete.
- Short, steady links: Stretch mode suits sites where the round trip is low and consistent enough that synchronous writes stay within the latency your applications accept. Confirm the threshold with application owners.
- Long or variable links: High or jittery latency slows every write in a stretch design, and the slowdown reaches end users. Asynchronous mirroring protects local latency but widens the gap between sites.
- Recovery traffic: After a failure, resynchronization competes with client traffic on the same inter-site link. Measure how much spare capacity that link has.
- Link redundancy: The Ceph hardware recommendations advise bonding network links across two switches. Apply the same redundancy to the inter-site path, since a single link failure can isolate a site.
Before choosing, collect latency and jitter measurements during peak business hours and compare them with the write latency your applications tolerate.
Both designs are meant to keep data available after losing one site, but they make different promises about what survives and how quickly service returns. Stretch mode depends on a tiebreaker monitor that must keep a majority when one site fails, so its placement is central. Mirroring leaves the second cluster holding whatever was replicated before the loss, and promotion and failback remain steps your team must run.
- Tiebreaker placement: The tiebreaker monitor should sit outside both data centers so that losing either site does not remove the monitor majority. Confirm the placement rules in the cephadm documentation for your version.
- Degraded operation: After a site loss, the surviving site runs in a degraded state until the failed site returns. Document who makes that call and how long the clients may run without the second site.
- Recovery point with mirroring: The secondary cluster holds data up to the last completed replication. Your recovery point depends on the measured replication lag, not on a guarantee.
- Manual steps: Mirroring requires you to promote the secondary, redirect clients and plan failback. Confirm which failover steps remain manual in stretch mode for your release.
Write down the failure scenarios you must survive, including losing the tiebreaker location, and test each scenario against both designs before you commit.
The core difference is when a write is considered complete. In stretch mode, data is placed across both sites under one pool, so a completed write exists at both locations. With independent clusters, a write completes on the primary cluster and reaches the secondary later through rbd-mirror or CephFS snapshot mirroring. The *RBD mirroring for disaster recovery* page covers the asynchronous options in depth.
- Synchronous placement: Writes in stretch mode are placed across both sites before the client sees them complete. Confirm the replica counts and minimum replica behavior for each pool with the stretch mode documentation.
- Asynchronous commit: With mirroring, writes made during the last replication interval may be missing at the remote site after a primary failure. Your applications must tolerate that loss window or you must reduce it.
- Application consistency: Databases and applications with their own consistency model should be tested for how they behave when writes stall during a link problem.
- Erasure coding limits: Stretch mode has limits on erasure-coded pools, so workloads that depend on erasure coding may need a different design. Confirm the current limits for your release.
In a lab, test write latency and application behavior during a controlled link disruption before you rely on either design in production.
A stretch cluster is one system to operate: one monitor set, one dashboard and one upgrade sequence. Two independent clusters double the monitors, managers, OSD hosts and upgrade work, but each cluster can be upgraded, maintained and failed on its own schedule. The overhead you accept shapes both hardware cost and the team time needed to run the design.
- Hardware and capacity: Each design needs headroom for its own failure scenarios. The Ceph hardware recommendations advise sizing memory for peak use and benchmarking before buying, which applies to either design.
- Upgrades: A single cluster means one coordinated upgrade across both sites. Two clusters allow staged upgrades but require two runbooks and a plan for mixed versions during the transition. Check the Ceph active releases page for support windows before scheduling.
- Monitoring and replication jobs: Mirroring adds replication jobs, lag alerts and peer configuration to watch. Stretch mode adds monitor placement and site-level health checks.
- Cost components: A quote should separate design, build, failover testing, and ongoing support, and state who owns hardware, licensing for any tooling, and monitoring. Ceph itself is free software, so most cost comes from hardware, network and labor.
Ask any partner to price the failover test and the first upgrade cycle separately from the initial build, because those tasks repeat over the life of the cluster.
Stretch mode configuration options and when each fits
This table compares the settings that shape a two-site stretch cluster: replica count, min_size, degraded and recovery behavior, CRUSH layout and tiebreaker placement. Read each row for the condition where it fits and the capacity or risk it adds. Confirm exact values and transition behavior for your Ceph release in the stretch mode documentation before you finalize the design.
| Option | When it fits | Capacity or risk trade-off |
|---|---|---|
| Size 4 pool split across two sites | Both sites must hold a full set of replicas for the same data | Usable capacity is about half of raw capacity. Confirm overhead with your CRUSH layout. |
| min_size 2 on size 4 pools | Writes should continue after one site is lost while two copies remain | Two surviving copies give less redundancy than four. Confirm the value for your release. |
| Lower min_size (such as 1) | Availability matters more than write durability during a double failure | Acknowledged writes may exist on one copy, raising loss risk if that copy fails. |
| Degraded stretch mode | A site is down and the surviving site must keep serving clients | Data runs with reduced redundancy until the failed site returns. Confirm the trigger and I/O behavior. |
| Recovery stretch mode | The failed site returns and data must resynchronize before normal operation | Recovery traffic competes with client I/O. Plan throttles and watch progress closely. |
| CRUSH buckets per data center | Both sites have comparable host and OSD counts and capacity | Uneven hosts or OSDs per site skew placement, capacity and recovery time. |
| Tiebreaker monitor in a third location | A third site with a network path to both data centers is available | Adds a site to operate. Latency and reachability to both sites affect quorum. |
| Tiebreaker monitor inside one data center | No third location exists and the design accepts reduced protection | If the host site fails, quorum can be lost. Confirm the failure scenarios you accept. |
How to set up a stretch cluster step by step
This sequence builds one Ceph cluster across two data centers with a tiebreaker monitor at a third location. Each step ends with a check, so you confirm monitor quorum, CRUSH placement and pool settings are correct before any production data is written to the cluster.
Prepare the third site or tiebreaker host
Before touching Ceph, decide where the tiebreaker monitor will run. It must sit outside both data centers, on a host that keeps its own network path to each site, so that losing either site does not also remove the tiebreaker. Confirm the host meets the cephadm prerequisites for the release you plan to run, and confirm the monitor placement rules with the Ceph documentation for that release. Done means the host is reachable from both sites, time synchronization is working, and the container runtime is installed and tested.
Bootstrap the cluster with cephadm
Bootstrap the first host with cephadm, which creates a single-node cluster with one Monitor and one Manager, then add the remaining hosts and let cephadm deploy the base services. Use the cephadm documentation for the release in use to confirm the bootstrap options and the exact commands, because these change between releases. Check that
ceph -sreports a healthy quorum with the expected monitor count and that every host appears in the orchestrator host list before you continue.Define CRUSH buckets for each data center
Create a CRUSH bucket of type datacenter for each site and move every host into the bucket for its physical location. This tree is what lets Ceph place replicas evenly across both sites and what the stretch rule will reference later. Review the layout with the CRUSH tree output and confirm each OSD host sits under exactly one datacenter bucket. Do not place the tiebreaker host in either data center bucket. Done means the tree shows two data center buckets with the expected hosts and OSDs under each, and the tiebreaker host is absent from both.
Deploy monitors and managers across both sites
Place monitors and managers so that each data center holds an equal share, and place the tiebreaker monitor at the third location. Use cephadm service specifications with host labels or explicit placements, and confirm the exact spec format for your release. Each monitor should also carry a CRUSH location that matches its data center, which the stretch mode logic depends on. Check the monitor map and quorum status after each placement change, and confirm that every monitor reports the location you set.
Enable stretch mode with the tiebreaker monitor
Enable stretch mode by naming the tiebreaker monitor and the CRUSH bucket type that separates the sites, then confirm the procedure and the command syntax for your release in the Ceph documentation. This step changes how monitors elect a leader and how the cluster reacts to a site failure, so the monitor locations from the previous step must already be correct. Check the cluster status and the monitor status output to confirm stretch mode is active, the tiebreaker is recognized, and both sites are listed as members.
Create size 4 pools on the stretch rule
Create the pools that will hold data and set their replication to size 4 so that each site keeps two copies. Assign the stretch CRUSH rule created for the two-site layout, and confirm that the rule places replicas across both data center buckets. Confirm the min_size value with the stretch mode documentation for your release, because it controls whether I/O continues during a site loss. Check the pool detail output for size, min_size and crush_rule, and confirm that placement groups reach an active and clean state.
Verify health and rehearse a site loss
Confirm the final state with a health check, the CRUSH tree, the monitor quorum and the pool settings, and record the output as your baseline. Then run a planned site-loss test in a lab or a maintenance window, observing how the cluster moves between degraded and recovery stretch modes and how it returns to healthy when the site comes back. Document the expected behavior, the commands used and the recovery steps. Done means the test results match your design and the runbook has been reviewed by the operations team.
What happens to clients during a site loss
When one data center fails, the outcome depends less on the lost OSDs than on which side still holds monitor quorum. The tiebreaker monitor, placed in a third location, is what lets the surviving site win that vote without the two sites deciding alone. Once the failed site is confirmed down, Ceph moves the cluster into *degraded stretch mode*, where the surviving site continues to serve I/O from its copies. When the lost site returns, the cluster moves into *recovery stretch mode* and resynchronizes data before it is treated as fully redundant again. Confirm the exact mode transitions, triggers and any manual steps for your release in the Ceph stretch mode documentation before you write runbooks.
Clients feel the failover as a pause, not as a clean switch. I/O to placement groups that were waiting on the lost site stalls while the monitors agree on the new state and the affected PGs re-peer. Block clients using RBD may see a latency spike or I/O errors, depending on guest timeouts and the mapping method, and S3 clients using the Object Gateway will typically see retries or temporary errors until the endpoint answers again. Ask the team that runs your hypervisors and applications which timeouts they use, because those settings decide whether a guest pauses or fails. Your application owners should also confirm that they retry idempotently, since a write that timed out may or may not have been acknowledged.
Recovery time and the write path are both governed by latency between the sites, because every replicated write must be acknowledged across the link before a client receives confirmation. A size 4 pool with two copies at each site makes that dependency explicit. Measure this in your own environment: build a staging cluster with the same inter-site latency and bandwidth as production, fail one site under synthetic client load, and record the stall length, the time to enter recovery mode and the resync rate. Repeat the test with the tiebreaker placed where you plan to run it in production, and with the monitors and managers in their intended positions. The numbers from your own test, not from general guidance, should set your recovery objectives.
Limits of erasure coding and device classes in stretch pools
Stretch mode spreads one cluster across two sites, so the limits on pool types, device classes, monitor placement and release support shape what the design can do. Each card names a constraint to confirm in the documentation for the exact release you plan to run before you commit to a layout.
Erasure-coded pools in stretch mode
Erasure coding uses less raw capacity than replication, but stretch mode designs are often built on replicated pools. Ask whether erasure-coded pools are supported in stretch mode for your release, and whether they can keep the site-level failure domain you need. If they are not supported, the capacity cost of replicated pools belongs in the budget from the start.
Device class placement across sites
Pools that target a device class, such as SSD or NVMe, depend on CRUSH rules that place copies in matching buckets. Confirm how the release handles pools that need matching device classes in both data centers, and what happens to writes if one site has fewer OSDs of that class. Mixed media across sites can leave a pool with fewer usable copies than its size setting implies.
Third site for the tiebreaker
The tiebreaker monitor decides which side keeps quorum when the two data centers lose contact, so it must sit outside both data centers. That usually means a third location with separate power and network paths. Confirm the monitor count, host placement and latency expectations in the stretch mode documentation, and test that the third site stays reachable during a partition between the other two.
Cost of size 4 replication
Size 4 keeps four copies of each object, so usable capacity is only a fraction of raw capacity. Budget for that ratio, plus free space that recovery needs after a site loss, before you choose drive counts. Ask your vendor or implementation partner how much headroom the design needs and what the inter-site link must carry while one site is degraded.
Release support and end of life
Stretch mode behavior and upgrade steps differ between releases, and cephadm manages clusters from v15.2.0 onward. Squid's estimated end of life is October 31, 2026, and Tentacle's is June 1, 2027, so the upgrade path belongs in the design before production. Confirm stretch mode support and recovery behavior for the exact release you deploy.
Questions buyers ask before building a stretch cluster
How much does building a Ceph stretch cluster cost?
Cost depends mainly on the hardware for two sites, the tiebreaker monitor, and the engineering time to design and test the layout. Hardware is one-time spend and includes OSD media, enterprise-class SSDs for metadata, and bonded network links across two switches, as the Ceph hardware recommendations advise. Design and deployment labor is also one-time and grows with the number of hosts, pools and failure domains. Recurring costs include power, space, support contracts, and staff time for patching and capacity growth. Ask the partner to price each component separately: the hardware list with who buys it, design and deployment hours by phase, site-loss testing, documentation and handover, and any ongoing retainer.
How long does a stretch cluster build take?
The duration is set by the written scope, which is agreed before work starts, so that document is the only reliable source for a timeline. The main drivers are inter-site network readiness, hardware lead times, the number of hosts at each site, and how many failure scenarios you require the partner to test. Site-loss, recovery and failback testing often takes longer than the initial deployment because each scenario needs a controlled run and verification. Ask for the proposal to list phases (design, procurement, cephadm deployment, pool and CRUSH setup, failure testing and handover), the expected duration of each, and the client dependencies that would move them.
What does our team need to provide for a stretch build?
Your team supplies the facilities, network and decisions that the build depends on. Confirm each item before scoping:
- Two sites with power, cooling and rack space for the planned hosts
- Inter-site network links with measured latency and bandwidth, plus a separate location for the tiebreaker monitor
- Access to hosts, switches and hypervisor or cloud accounts, and change windows for controlled failover tests
- A named owner who approves design decisions, test results and the go-live decision
Missing or late items are a common cause of schedule slips, so list them as assumptions in the written scope.
Can our in-house team build a stretch cluster without a partner?
An in-house team can build one if it already runs Ceph in production and has rehearsed failover before. Cephadm handles bootstrapping and adding hosts and services, as the cephadm documentation describes, so the deployment mechanics are documented. The harder work is stretch-specific: placement of monitors across three locations, CRUSH layout, pool sizing, and repeatable site-loss runs under realistic client load. Assess the team against those tasks honestly. A fixed-scope design review before the build can cover the gaps without handing over operations. Your team should still own the runbooks and the on-call procedure.
How does Ceph stretch mode compare with RBD mirroring?
They solve different problems. Stretch mode keeps one cluster across two sites with a tiebreaker monitor, so a site loss is handled inside that cluster. RBD mirroring uses rbd-mirror to copy block images to a second, independent cluster, which means operating two clusters and orchestrating failover yourself. Mirroring recovery points depend on whether you use journal or snapshot mode, and it can be one-way or two-way. Stretch mode depends on low-latency links between the sites and a third location for the monitor. Confirm the latency your links actually deliver, the cost of the links, and whether a second independent cluster is a requirement before choosing.
What happens to our data during a site loss in stretch mode?
Stretch mode moves the cluster through degraded and recovery states, and the tiebreaker monitor helps the surviving site keep quorum. Pools use size 4, so each site holds a share of the copies and the surviving site keeps serving data in the degraded state. When the lost site returns, the cluster rebuilds the missing copies, which adds load while clients are still active. Recovery time depends on data volume, network bandwidth and recovery throttling. Ask the partner to demonstrate the states the cluster enters, the commands that move it between them, and what clients see during each transition, using a test run on your release and hardware.
What should a written quote for a stretch cluster include?
A usable quote lists what is delivered, what is bought, and what could change the price. Ask for:
- A scope statement naming the Ceph release, the hosts at each site, the tiebreaker location and the stretch mode scenarios to be tested
- An itemized split of hardware and software you buy versus what the partner supplies
- Design, deployment and test effort by phase, with each failure test named
- Handover items: runbooks, monitoring and alerting setup, and the site-loss procedure
- Stated client dependencies and the change process for scope additions
Also confirm the release you will run. The Ceph active releases page lists Squid and Tentacle as maintained, with estimated end-of-life dates, which affects how long the design stays supported.
What a written scope for a stretch cluster should include
A written scope for a stretch cluster starts with the facts that decide whether two sites can run as one cluster at all. Before DeployOpen can quote, it needs the round-trip latency and sustained bandwidth between the two data centers, the link redundancy in each direction, and the location of the third site that hosts the tiebreaker monitor. Ask your network team for measured figures under load, not link ratings, and confirm which failure modes the inter-site links have already been through. The third site should be independent of both data centers in power, network path and administrative control, so the scope should name it and describe its connectivity.
The scope should also describe what the cluster will carry. List the workload mix (RBD block for virtual machines, CephFS, or RGW object storage), the usable capacity target, the growth expected over the agreed period, and any pools that need erasure coding instead of replication. Because stretch pools use size 4 replication and erasure-coded pools have stretch limits, a workload that depends on erasure coding must be named early so the design can be checked against it. Record the hardware each site will provide, including whether each site has the same OSD media and device classes, since mismatched sites change recovery behavior.
The most useful part of the scope is the list of failure scenarios and acceptance criteria. Name the events the cluster must survive and recover from, such as loss of one data center, loss of the tiebreaker, a severed inter-site link, and a monitor quorum loss, and state how each will be tested, including which stretch mode you expect to use after a site loss and who decides when to switch. Define what passing means: data availability, client I/O behavior, time to return to a healthy state, and the evidence you will keep. Engagements are fixed-scope, so the written scope, deliverables and timelines are agreed in writing before work starts, and no price is estimated on this page. Ask for the quote to list each scenario, the environments tested, and any work excluded.
How DeployOpen scopes, builds and supports stretch clusters
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside infrastructure you control, including Ceph. For a two-site stretch cluster, we start with a written scope that fixes the site layout, tiebreaker placement, pool design and the failure scenarios that must be tested before go-live.
How we scope
- Each engagement is fixed-scope, with deliverables and timelines agreed in writing before work starts.
- The written scope lists both data centers, the tiebreaker monitor location, and the network links between sites.
- We confirm the Ceph release, cephadm version and whether erasure-coded or device-class pools are in or out of scope.
- Site-loss scenarios to be tested, including degraded and recovery stretch mode steps, are named in the scope.
- Ask us to state which items depend on your own network, power and facility decisions, so they are not assumed.
How we build
- We deploy the cluster with cephadm inside your environment, with monitors and managers placed to match the written layout.
- Size 4 pools, CRUSH failure domains and stretch mode settings are configured and reviewed against the agreed design.
- We run failure tests for each site and for the tiebreaker, recording observed behavior against the expected result.
- Network bonding, drive layout and WAL and DB placement are checked against the Ceph hardware recommendations.
- Any change to the agreed scope goes through a written change request before it is built.
How we hand over
- Handover includes architecture documentation covering site layout, monitor placement and pool definitions.
- Runbooks describe degraded and recovery stretch mode steps, including who decides when to switch and how to verify.
- We document the test results, the observed recovery times and any deviations from the original design.
- Your team receives configuration exports and a list of settings that must not change without review.
- Knowledge transfer sessions are scoped in advance so your operators can run failover drills on their own.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Retainers can cover Ceph point-release upgrades, periodic failover drills and review of health warnings.
- Ask for the support hours, response expectations and escalation path to be written into the retainer before signing.
- Upgrades between Ceph releases are planned as separate written scopes with their own test and rollback steps.
- We recommend re-running site-loss tests after any hardware, network or topology change to confirm the design still holds.
