Ceph RBD snapshots and clones for VM templates

Ceph RBD snapshots and clones for VM templates: when teams need them

Teams that rebuild virtual machine images by hand usually run into the same problem: every new VM starts from a different package set, a different patch level, and a different set of manual steps, so provisioning is slow and hard to reproduce. A golden image on Ceph block storage addresses this by capturing one known-good disk state and creating new disks from it. In RBD terms, a snapshot is a read-only, point-in-time view of an image. A protected snapshot is a snapshot that RBD refuses to delete while clones still depend on it. A clone is a new, writable image that uses a protected snapshot as its parent, so it starts almost instantly and stores only the blocks it changes. The pattern fits fleets that provision many similar VMs, lab and test environments that need to be reset quickly, and any workflow where the template is updated on a schedule.

The pattern does not replace configuration management, image pipelines or backups. A clone inherits everything in its parent, including credentials, host keys, machine IDs and any defect in the template, so the build process for the golden image still needs to generalize the OS and be tested. Snapshots also live in the same pool as the images they protect, so they are not an independent copy of your data. Before adopting the pattern, confirm with your storage and virtualization teams which image format, which cephx user and which pool layout each consumer will use, and check the release notes for the Ceph version you plan to run, since the release lifecycle is documented on the Ceph active releases page.

The design decisions that matter most come after the first clone works. Clone chains grow when a clone is itself snapshotted and cloned again, and each level adds a read dependency on its ancestors, which can slow reads and complicates deletion. Flattening a clone copies its parent data into the child so the dependency is removed, at the cost of the storage and I/O that copy requires. Plan a naming and retirement scheme for template versions, decide which snapshots are protected, and test that a snapshot can be restored or exported as a backup source before relying on it. Ask the implementation team to demonstrate deletion of an old template version with live clones attached, because that is where most operational surprises appear.

Which snapshot and clone options fit which template workflow

This table compares the main ways to build VM templates on Ceph RBD, including protected parent snapshots, per-VM clones, flattening and crash-consistent versus application-consistent snapshots. Read each row for when the option fits and what it costs you later, then choose the combination that matches how your templates change and how your backups are taken.

Option	When it fits	Main trade-off	What to confirm
Protected parent snapshot as a read-only golden image	One base image per OS build that many VMs clone from, with changes made only by creating a new snapshot	Clones keep a dependency on the parent snapshot, so the snapshot cannot be removed while clones exist	Confirm the snapshot is protected before cloning and that your tooling blocks deletion of parents in use
Per-VM clone from a protected snapshot (copy-on-write)	Fast provisioning of many similar VMs where each VM writes only its own changes	Each clone reads unchanged blocks from the parent, so the parent's pool and OSDs carry shared read load	Confirm the parent and clones sit in a pool layout that matches your performance and failure domain plan
Deep clone chains (clone of a clone)	Layered templates, for example a base OS image, then a patched image, then a per-team image	Each level adds a dependency, which can slow reads and complicates cleanup and recovery when a middle parent is removed	Confirm how many layers your template process creates and keep the chain shallow where possible
Flattening a clone to remove the parent dependency	A long-lived VM that must be independent of its template, or before retiring a parent snapshot	Flattening copies all inherited blocks into the clone, so it uses extra capacity and I/O during the operation	Ask how flatten progress is monitored and plan it outside peak I/O windows
Crash-consistent snapshot	Quick, frequent snapshots of VMs where the guest can recover from a sudden stop, and fast template captures	Data in guest memory and open application buffers may not be on disk, so database or journal state can need recovery	Confirm the guest filesystem and applications can recover cleanly from a power-loss-style state
Application-consistent snapshot (quiesced guest)	Databases and stateful services where a clean on-disk state matters for restore and for backup sources	Requires guest agent or pre and post hooks, and quiescing can pause writes for a short window	Confirm which applications need flush hooks and test that the quiesce step succeeds before snapshots are trusted
Snapshot as a backup source	Restore points taken from a snapshot and exported or diffed to a backup target on a schedule	A snapshot is not a backup by itself, since it lives in the same cluster and shares its failure domain	Confirm where exports land, how long they are kept, and that restores are tested from the copy, not the snapshot
Rebuilding templates from scratch instead of cloning history	Templates that change often and where a clean rebuild is cheaper to audit than a long chain	Rebuilds take longer than a clone operation and need an automated image pipeline to stay repeatable	Ask how the build pipeline records the image version each VM was cloned from

Why deep clone chains slow VMs and complicate operations

Each RBD clone keeps a reference to its parent snapshot, so a clone of a clone adds another layer that reads and bookkeeping must cross. The risks below show where that layering affects VM latency, deletion, space and recovery, and what to check before you rely on a template chain in production.

Reads traverse every parent

A clone stores only the blocks it has written. For any block it has not written, the read falls through to the parent snapshot, and from there to that parent's parent if the parent is itself a clone. Each extra level adds lookups for cold blocks, which is where VMs booted from deep chains often notice slower first boots and log writes. Measure read latency for a boot workload on your own pool, comparing a one-level clone with your deepest chain, rather than assuming a figure.

Parent snapshots must stay

A snapshot that has children cannot be removed, and a protected snapshot cannot be unprotected while clones depend on it. Teams often find this out when they try to retire an old golden image months after rollout. Before you add a layer, record which snapshot each clone uses, who owns it, and what must happen before the parent can be removed.

Deletion must follow the chain

Deleting a clone is simple, but deleting the parent in the middle of a chain is not. Removing images in the wrong order leaves snapshots blocked or orphaned, and scripts that assume a flat hierarchy can fail partway through. Write deletion procedures that walk children first and test them on a non-production image before automating cleanup.

Space accounting becomes unclear

Each clone uses space only for its own writes, so the capacity a template appears to consume is spread across many images. Shared blocks sit under a parent that may be snapshotted, backed up or deleted later, and that makes per-tenant usage reports misleading. Confirm how your Ceph release reports used space for clones and snapshots before you bill or quota teams by image.

Flatten when chains grow

Flattening copies the parent's data into the clone and severs the link, so reads no longer cross layers. The trade-off is that flattening writes the full image size to the pool and takes time and I/O, so it should run in a planned window. Flatten the clones that serve production VMs first, and keep the golden snapshot in place for images that still benefit from sharing.

Snapshots as backup sources

A snapshot of a deep clone is still only a point in time of that clone, and restoring it depends on the parent chain existing. Backup tools that export or mirror snapshots need to handle the same dependencies, or a restore can fail when a parent has been removed. Confirm with your backup tooling and test a full restore from a chained image before you treat snapshots as recovery points.

Setting up a golden image snapshot and clone workflow

This sequence turns a sealed operating system image into a protected RBD snapshot that every new VM clones from, with a check at each stage so that a clone is only created from a template you have verified. The result is a repeatable template path that keeps the source image immutable and makes each VM's parent relationship visible.

Create a dedicated namespace for templates
The storage administrator creates a namespace in the pool that will hold golden images, using rbd namespace create with the pool and namespace names your design defines. Separate cephx capabilities should let hypervisor clients read template snapshots and write only to their VM namespace, so no VM client can alter the master image. Confirm the namespace exists with rbd namespace ls and test one client's access by attempting a write to the template namespace, which should fail. The step is complete when the namespace is listed and the write attempt is denied.
Build and seal the base image
The platform team installs the guest operating system, adds the agent and packages the template needs, and seals the instance. On Linux, that usually means running cloud-init clean, clearing machine IDs and removing SSH host keys so each clone generates its own identity at first boot. On Windows, sysprep generalizes the image. Shut the source VM down cleanly once sealing is finished. The step is complete when the image is powered off with no host-specific identity left in it, which you confirm by booting a throwaway copy and checking that the new identity is generated.
Import the sealed image into RBD
Write the sealed disk into the template namespace with rbd import, creating a format 2 image with the layering feature enabled, since clones depend on it. The image name should carry the OS and version so that operators can identify it later. Record the source image checksum alongside the import so you can prove the RBD image matches the file you sealed. The step is complete when rbd info shows format 2 and the layering feature, and the checksum of the exported data matches your record.
Create a named snapshot of the template
Create a snapshot of the imported image with rbd snap create, using a name that carries a version, such as a release label and date your change process defines. The snapshot is the immutable point that clones will reference, so the name must be stable once VMs depend on it. Do not modify the template image itself after this point. Check the result with rbd snap ls on the template, which should show exactly one snapshot with the expected name and size.
Protect the snapshot before cloning
Mark the snapshot as protected with rbd snap protect so that it cannot be deleted while clones still depend on it. Protection is the control that prevents an accidental rbd snap rm from breaking every VM built on this template. Confirm the behavior for your release in the Ceph documentation for your version, because the protection workflow has changed across releases. The step is complete when rbd snap ls marks the snapshot as protected and a test removal attempt is refused.
Clone the snapshot for each VM
The hypervisor or provisioning tool runs rbd clone from the protected snapshot into the VM's namespace, producing a copy-on-write image that initially stores no unique data. Clone each disk as a format 2 image so it inherits the layering feature. Attach the clone to the VM and let cloud-init or sysprep complete first-boot configuration. The step is complete when rbd info on the clone shows the template snapshot as its parent, and rbd children on the snapshot lists the new clone.
Flatten clones that must leave the chain
Flatten a clone with rbd flatten when it needs to stop depending on the template, for example before the template snapshot is retired or when a long clone chain adds read overhead. Flattening copies all referenced data into the clone, which uses capacity and I/O, so schedule it for periods your change process approves. Ask the operations team to estimate the impact on your cluster before running it at scale. The step is complete when rbd info shows no parent for the flattened image and the snapshot's children list no longer includes it.

Using RBD snapshots as backup sources safely

Exporting a snapshot with rbd export or rbd export-diff writes the image state at the snapshot point to a file or stream that lives outside the cluster. A full export gives a self-contained copy, while export-diff between two snapshots produces an incremental stream that a restore must apply in order onto a matching base. The snapshot guarantees a point-in-time view of the image, but it does not quiesce the guest filesystem or application, so crash consistency is what you get unless the guest flushes or freezes I/O first.

Consistency limit: the export captures the block state at the snapshot, so databases and filesystems need an application-aware freeze or a guest agent step before the snapshot is taken.
Chain integrity: each export-diff depends on the snapshot it starts from, so deleting or rotating an intermediate snapshot breaks later increments.
Retention ownership: confirm whether the backup tool or the platform team deletes the RBD snapshot after export, because removing it too early can break the next incremental.
Tooling checks: confirm that the backup tool records the snapshot name, image name, pool and export format, and that restores are tested into a scratch image.
Target storage: confirm the export destination is outside the failure domain of the source pool and that it is encrypted and access-controlled.

Before relying on this path, restore a sample image from an incremental chain and boot it, because a successful export does not prove a usable restore.

Access control and operational safeguards for snapshots

A golden image is only as trustworthy as the credentials that can change it. The cards below cover how cephx capabilities scope snapshot and clone actions, how a compromised hypervisor could reach a template, and the monitoring and retention rules that keep snapshot sprawl from becoming a storage or recovery problem.

Scope cephx caps to template operations

Cephx capabilities are granted per client and can be restricted by pool and by RBD image name prefix. Clients that clone from templates need read access to the parent image and write access only to the pool where clones land, while template administration should use a separate identity. Check that clone-consuming hypervisor keys cannot run snapshot creation, protection or removal commands. Confirm the exact cap syntax for your release in the RBD and cephx documentation before you write the grants, and test each identity with a throwaway image.

Separate template authors from consumers

Treat the identity that creates and protects a golden image as an administrative role, distinct from the identities that boot VMs from it. Keep the administrative key on a small set of management hosts rather than on every hypervisor. This separation means a single compromised hypervisor cannot rewrite the image that all of its neighbors clone from. Document which person or pipeline is allowed to promote a new template, and record each promotion in your change system.

Protect against a compromised hypervisor

A hypervisor with broad cephx access can delete protected snapshots if its key allows it, and a clone's parent snapshot cannot be removed while clones depend on it. Ask whether your hypervisors hold only the caps needed to attach and clone, and whether any key has pool-wide write access. Run a tabletop exercise where one hypervisor is assumed compromised and confirm which templates it could alter. Flag the answer as a design risk until a test proves it.

Monitor snapshot count and space

Snapshots consume space as data diverges from the parent, and a large number of snapshots per image slows operations and complicates cleanup. Track snapshot counts per image, the space used by snapshots and the number of clones that depend on each protected snapshot. Use the rbd du and rbd snap ls commands in your monitoring job, and set alerts on growth rather than on a fixed threshold you have not validated. Confirm the output fields and their accuracy for your Ceph release.

Govern retention and deletion

Define who may approve deleting an old template snapshot, how long superseded templates remain available for rebuilds, and what happens to VMs cloned from a version scheduled for removal. Because a parent snapshot with children cannot be deleted until its clones are flattened or removed, retention decisions affect clone chains as well as storage. Write the policy as a rule with an owner, a review date and an exception process. Ask your compliance lead which retention periods your audit scope requires before you set them.

Questions teams ask before building a template pipeline on RBD

How much effort is building a template pipeline on RBD in-house?

The core RBD steps are short: create a golden image, protect a snapshot, clone it, and test that the clone boots. The effort sits in the work around those steps. Your team has to automate image builds, scope cephx users to the right pool, set naming and retention rules, schedule flattening, verify snapshots and document recovery. Each of these is simple alone, but together they need an owner and regular testing. No duration is supplied for this work, so ask any partner for a written estimate based on your image count, hypervisor estate and existing automation, and compare it with the staff time your own team can commit.

What do we need to provide before a partner starts the work?

A partner needs the inputs that define the scope before any build begins. Plan to provide:

Scoped access to the Ceph cluster, using dedicated cephx users for template creation and for the hypervisors that clone from them, not admin keys
The current image build pipeline or its source, such as the scripts, kickstart or cloud-init files and the list of OS variants you support
A named backup owner who accepts the snapshot and export policy and can approve restore tests
A test hypervisor host and a change window for pool and image changes

DeployOpen plans and documents the work against these inputs, so gaps found early are cheaper to close than gaps found during cutover.

Can we flatten template clones without taking VMs offline?

Usually yes, because flattening runs against the child image while it stays attached to its hypervisor. Flattening copies the parent data into the clone, so it uses extra capacity and I/O, and client performance can drop while it runs. Treat it as a planned background task rather than a guaranteed no-impact operation. Confirm the behavior for your Ceph release and client library versions in a lab that mirrors production, then run flattens in batches with I/O monitored. Keep a rollback path, such as keeping the parent snapshot until the flattened clone passes boot and application checks.

How do we detect and fix deep clone chains in RBD?

A deep chain is a clone of a clone of a clone, where each image depends on a parent snapshot further up the tree. Find them by checking each image with rbd info, which shows its parent, and by listing the children of each protected snapshot with rbd children. Keep a written depth limit for your templates. To fix a chain, flatten the affected clones or rebuild them from a current golden image, then retire the old parent once no children depend on it. Add the depth check to your pipeline so chains are caught when they form, not during an incident.

Can snapshots of golden images serve as backup sources?

Snapshots can feed a backup process, but they do not protect you on their own. A snapshot lives in the same pool and cluster as the data it protects, so losing the cluster loses both. Use snapshots as the consistent point-in-time source, then copy them elsewhere with an export or diff-based process, or with RBD mirroring to a second cluster if that fits your design. Also make sure a protected snapshot used as a clone parent is never deleted while children exist. Agree the recovery point and recovery time targets with the backup owner before choosing the method.

What drives the cost of this work up or down?

Cost follows the number of decisions and the amount of testing. It rises with more image families and OS variants, more hypervisor hosts that must boot clones, an air-gapped or heavily restricted network, weak or missing build automation, and new compliance evidence that must be produced. It falls when the image pipeline already exists, the cluster is accessible, the backup tooling is in place and the scope is limited to a few well-defined templates. DeployOpen works to a fixed scope agreed in writing, so ask each quote to list which of these factors it assumes and what would change the price.

Who pays for what in a template pipeline project?

Three parties carry costs. Your organization pays for the hardware, the storage capacity, the hypervisor hosts and the staff time needed to own the pipeline after handover. Ceph's source is licensed under LGPL, and the project itself offers no paid edition, so the software itself has no license fee to plan for. A partner charges for the agreed scope, normally as a fixed-scope implementation, and may offer a retainer for ongoing support and testing. One-time costs cover the build and documentation. Recurring costs cover operations, restore tests and upgrades, so ask each quote which items recur and how often they are billed.

Can we build this ourselves using only the Ceph documentation?

Yes, for a lab or a small estate, because the RBD snapshot, protect, clone and flatten commands are documented by the Ceph project. The gap is operating the workflow under production load: sequencing flattens, enforcing depth limits, rotating cephx keys, and proving that a restore works. Ceph's documentation describes the features but cannot decide your retention, your recovery targets or your change control. Most teams that build it alone should first run a restore test on a copy of a production template, then decide which parts they will maintain themselves and which they want a partner to design or review.

How DeployOpen scopes and delivers RBD template workflows

DeployOpen plans and implements Ceph RBD snapshot and clone workflows for VM templates inside your own infrastructure. We start from a written scope that names the pools, images, clone policy and backup sources, then build, test, document and hand over the workflow so your team can run it.

How we scope

Inventory the golden images, pools and hypervisor clients that will consume snapshots and clones
Agree in writing which snapshots are protected, how long they are kept and who approves changes
Define when clones stay shallow and when they are flattened, with expected chain depth limits
Engagements are fixed-scope, so deliverables and timelines are agreed before any build work starts

How we build

Configure the template image lifecycle: create, protect, clone, flatten and retire steps
Set cephx client permissions so hypervisors can clone without broad pool access
Test clone creation, boot from clones and flatten operations against a non-production pool first
Validate snapshot-based backup export and restore on a copy of real template data

How we hand over

Runbooks for promoting a new golden image, rolling it out and retiring old snapshots
Documented commands and configuration for snapshot protection, clone chains and flatten jobs
Verification records showing which restore and clone tests were run and their results
Knowledge transfer sessions with the engineers who will operate the workflow

How we support

Ongoing support, upgrades and continuous testing are available as retainers
Review clone chain depth and snapshot growth as new templates are added
Retest the workflow when Ceph or the hypervisor client is upgraded
Confirm supported release and client versions with your Ceph vendor or documentation before each change

Ceph RBD snapshots and clones for VM templates