Kubernetes backups to Ceph RGW with Velero

Velero can use Ceph RGW as an S3-compatible backup location, but the bucket, credentials, addressing and restore testing each need deliberate design. This page explains how to set it up and where it fails.

On this page

How Velero writes Kubernetes backups to Ceph RGW

Velero can write Kubernetes backups to a Ceph RGW bucket, so the backup copy lives on your own Ceph cluster rather than on the cluster being protected. Velero, formerly Heptio Ark, is open source under the Apache-2.0 license. It runs as a server on the cluster and is driven by a command-line client on your workstation. The server backs up Kubernetes resources such as namespaces, deployments and secrets, and it backs up persistent volume data. The backup destination is an object storage location, and for Ceph that means the Ceph Object Gateway (RGW) exposing an S3-compatible API. Ceph itself is licensed under LGPL-2.1 or LGPL-3.0, and the Ceph project does not sell a paid edition, so the storage layer carries no license fee of its own.

Velero's supported providers page lists Ceph RADOS v12.2.7 among S3-compatible providers that users report working. The same page notes that these providers are not regularly tested by the Velero team. Treat that listing as a starting point, not as a support commitment. Before you rely on the setup, confirm the Velero and Ceph versions you plan to run, then complete a full backup and restore in a non-production namespace. Ask your partner or internal team to record which Velero release, which Ceph release and which S3 settings were tested, so the same combination can be repeated during upgrades. The Velero supported providers page is the right place to check the current wording before you commit to a version.

What you need before Velero can reach RGW

Velero needs four things lined up before the first backup runs: a Ceph cluster with the Object Gateway, a supported Kubernetes cluster, a current Velero release, and a network path from both Velero components to RGW. Each card below is a decision or check that would otherwise surface as a failed backup location during setup.

A Ceph release under support

Confirm the Ceph release in use is one that is actively maintained. The Ceph releases page lists Tentacle (20.2.x, latest 20.2.4, estimated end of life June 1, 2027) and Squid (19.2.x, latest 19.2.7, estimated end of life October 31, 2026) as the current active releases. A cluster on Squid should have a planned upgrade before its estimated end of life, since backups that depend on RGW inherit that schedule.

Object Gateway running and reachable

Velero writes to an S3-compatible bucket, so RGW must be deployed and serving the S3 API before any Velero configuration. Confirm the RGW service is healthy under the cluster's orchestrator and that a test bucket can be created and listed with an S3 client. Ask who owns the RGW service, since Velero's backup schedule depends on its availability.

Kubernetes version within support

Check the Kubernetes minor version of the cluster being protected against the release list. Kubernetes maintains the three most recent minor releases, so an older cluster may fall outside the versions Velero's maintainers and your distribution vendor test against. Plan the cluster upgrade before the backup design is finalized, because restore targets must also run a version you can support.

A Velero release and its plugin

Use a current Velero release. The latest supplied is v1.18.4, published September 28, 2026, after v1.18.3 on September 21, 2026. Velero's supported providers page lists Ceph RADOS v12.2.7 as a provider that users report working, but that page also notes the Velero team does not regularly test these providers. Confirm the exact object storage plugin for AWS-compatible storage in the current Velero documentation before you install.

Network paths from both components

Velero runs a server on the cluster and a node-agent on each node, and the node-agent is the component that uploads File System Backup data. Both need a route to the RGW endpoint on its HTTPS port. Test from inside the cluster network with a pod, not from a workstation, because node firewalls and egress policies often block the path that matters.

CA certificate for HTTPS

If RGW serves HTTPS with a certificate issued by an internal certificate authority, Velero must trust that CA. Obtain the root or intermediate certificate in PEM form, confirm its validity dates and chain, and plan how it will be mounted into the Velero deployment. Skipping certificate verification to get past a trust error weakens the protection of the backup path and should not be the fix.

Configuring the RGW bucket, credentials and Velero backup location

This sequence creates a dedicated RGW identity that can only reach the backup bucket, connects Velero to that bucket through the RGW endpoint, and confirms the backup location is available before any production cluster depends on it.

  1. Create a dedicated RGW backup user

    The Ceph administrator creates a separate RGW user for Velero, not a shared administrative account, and generates its S3 access key and secret key. Keep this user limited to the backup purpose so that a leaked key exposes only the backup bucket. Record the user ID and key pair in your secrets manager, not in a shared document. Done when the user appears in the RGW user list and the access key can be displayed once for handoff to the platform team.

  2. Create the backup bucket

    Create one bucket for Velero using an S3 client that can reach the RGW endpoint, and name it according to your naming standard. Decide whether the cluster gets its own bucket or shares one with a key prefix, since that choice affects how you scope access later. Apply the access policy so that only the Velero user can write to and read from this bucket. Done when a test object uploaded with the new key lands in the bucket and a second user without rights is refused.

  3. Restrict the user to the backup bucket

    Attach a bucket policy or user policy that grants the Velero user object and list actions on the backup bucket only. Confirm the exact action set against the Ceph Object Gateway documentation for your release, since policy support depends on the version you run. Test the restriction by attempting an operation on a different bucket with the same key. Done when the other-bucket request is denied and the backup bucket operations succeed.

  4. Install Velero with the RGW backup location

    Install the Velero server into the cluster being protected, or into a separate management cluster if you choose that layout, using the AWS object storage plugin for S3-compatible targets. Set the backup location to the RGW endpoint URL, the bucket name and a region value your Ceph setup accepts, and enable path-style addressing so that requests use the endpoint host and bucket in the path. Confirm the plugin version and install flags against the Velero documentation for your release. Done when the Velero deployment reports ready and the node-agent pods are running if you plan to use File System Backup.

  5. Store the credentials in a cluster secret

    Create a Kubernetes secret in the Velero namespace that holds the access key and secret key in the credentials format Velero expects, and reference it from the install configuration. Restrict read access to this secret with RBAC so that workloads in other namespaces cannot list it. Avoid placing the key in Helm values files committed to source control. Done when the secret exists, the Velero pod starts without credential errors, and a describe of the secret does not reveal the key value in plain text to users without rights.

  6. Confirm the backup location is available

    Run the Velero status command for backup storage locations and check that the RGW location reports an available phase. If it reports unavailable, check the endpoint URL, TLS trust for your internal certificate authority, the path-style setting and the credentials secret before retrying. Confirm the bucket contents show Velero's own metadata objects after the first check. Done when the location is listed as available and the bucket contains the expected Velero prefix.

  7. Run a test backup and verify the objects

    Create a small test namespace with a sample workload and a persistent volume, then run a manual backup scoped to that namespace. Wait for the backup to finish and check its phase and any warnings in the Velero output. List the bucket to confirm that the backup prefix contains the expected objects, including the resource metadata and, if you use File System Backup, the data files. Done when the backup completes without errors and the objects appear under the backup name in the RGW bucket.

Choosing snapshots or File System Backup for each volume

Velero can protect persistent volume data in two ways. The first is a volume snapshot, taken through the storage platform or a CSI driver that supports snapshots. The snapshot itself stays with the storage system, while Velero records the resource metadata and backup information in the object store. The second is File System Backup, where Velero reads the data from inside the volume and copies it to the backup location using modules from the open-source Kopia backup tool. With Ceph RGW, the second path is the one that writes volume data into your bucket, so the bucket and its credentials carry the content of the workload.

  • Snapshot path: depends on a snapshot-capable block storage driver or CSI driver in the cluster, and the snapshot is only as portable as the storage behind it.
  • File System Backup path: copies file data to S3-compatible object storage, so the RGW bucket becomes the store for application data.
  • Per-volume choice: Velero's backup configuration lets you decide the method for each volume, so a cluster can use both paths.
  • Restore implications: confirm that your restore test covers each method separately, since they fail in different ways.
  • Capacity planning: File System Backup adds data transfer and storage growth to the RGW cluster that snapshots do not.

Confirm with your storage vendor and CSI driver documentation which volumes support snapshots, and record the decision for each volume before you size the bucket.

How Velero concepts map to RGW buckets, users and keys

This table pairs each Velero setting with the Ceph RGW object it depends on and the component that reads it. Use it to check that every credential, bucket and endpoint is created on the Ceph side before the backup location is registered. The main takeaway is that the bucket, the RGW user and the Kopia repository path must be planned together, because a shared bucket or key lets one cluster read or overwrite another cluster's backups.

Velero conceptCeph RGW objectConsumed by and what to confirm
Backup storage locationBucket plus optional prefix on one RGW endpointVelero server. Confirm the bucket exists and the prefix is unique to this cluster before creating the location.
BucketRGW bucket owned by a dedicated backup userVelero server and node-agent. Create it with the RGW user that holds only this bucket's permissions.
PrefixObject key path inside the bucket, such as one folder per clusterVelero server. Use a distinct prefix per cluster so backup names and object paths do not collide.
Access key and secret keyRGW user access key pair in the cloud credentials secretVelero server and node-agent. Store the pair in a Kubernetes Secret and confirm the user cannot reach other buckets.
Endpoint URLRGW service address, usually the load balancer or RGW host on HTTPSVelero server and node-agent. Confirm the endpoint resolves from every node that runs the node-agent.
Region settingValue the RGW zonegroup accepts for S3 requestsVelero server. Confirm the value with the RGW configuration, since Ceph does not require a real AWS region name.
Path-style addressingBucket name in the URL path rather than a subdomainVelero server. Confirm the setting is enabled, since virtual-host addressing needs wildcard DNS for RGW.
CA bundleCertificate chain issued for the RGW endpointVelero server and node-agent. Mount the bundle when RGW uses an internal certificate authority.
Kopia repository pathRepository data stored under the backup prefix in the same bucketNode-agent for File System Backup. Confirm the repository path is separate for each cluster and that the encryption key is protected.

Why backups must live outside the cluster they protect

A backup bucket on the same cluster, or in a namespace the cluster can reach freely, is not an independent copy. Anyone who can compromise the workloads or the cluster control plane can usually reach whatever credentials and network paths the cluster holds. If the Velero server can read, write and delete objects in the bucket, and that bucket sits on storage the same cluster administers, an attacker who gains cluster-admin rights can overwrite or remove the backups along with the data they were meant to protect. Ceph RGW should run on a separate failure and administrative domain where practical, with its own administrators, so that losing the Kubernetes cluster does not also mean losing the place the restore comes from.

The Velero credentials deserve the same scrutiny. The Velero server needs to write backups and read them back for restores, and it needs to delete only what retention and expiry logic require. Give the RGW user a policy scoped to the one backup bucket and prefix, avoid administrative or bucket-owner rights, and confirm that the credentials stored in the cluster as a Secret cannot change bucket configuration. Restrict who can read that Secret, since a copy of the access key held by a compromised workload is enough to act within the policy.

Object Lock is the control that limits deletion, but only if it is configured and behaves as you expect. With a lock in place, backup objects cannot be removed or overwritten until their retention period ends, even by a holder of valid credentials. Your team must confirm Object Lock support, its retention modes and how bucket versioning interacts with it on the Ceph release it runs, in its own environment, by testing a delete attempt against a locked object before relying on it. Confirm the release against the Ceph active releases page and record the result as part of the restore test plan.

Securing the RGW credentials and backup data

A Velero backup location on Ceph RGW concentrates cluster data and the keys to read it in one place. The cards below cover the controls that protect that location: scoped RGW users, encrypted transport, Kubernetes secret handling, pod privileges, Kopia encryption and object immutability.

Scope RGW users to one bucket

Create a dedicated RGW user for Velero with access limited to the backup bucket it writes to. Do not reuse an administrative or storage-wide key, because a leaked credential would then expose every bucket on the gateway. Keep separate users for separate clusters so one cluster's key can be rotated or revoked without touching the others. Confirm the exact capabilities and bucket policy syntax for your Ceph release in the RGW documentation before you apply them.

Encrypt traffic to the RGW endpoint

Velero sends backup data and credentials to the S3 endpoint on every backup and restore, so serve RGW over HTTPS with a certificate your cluster trusts. If the endpoint uses an internal certificate authority, supply that CA to Velero rather than disabling verification. Confirm how your Velero version expects a custom CA to be configured. Ceph's messenger v2 secure mode protects traffic inside the cluster, which is a separate control from the client-facing TLS.

Store credentials in a restricted Secret

Velero reads the RGW access key through a Kubernetes Secret in its own namespace. Limit get, list and watch on that Secret with RBAC to the Velero service account and to the people who administer backups, because list and watch expose the values to any principal that holds them. Kubernetes stores Secrets unencrypted in etcd by default, so enable encryption at rest for the API server and include the etcd snapshots that hold those Secrets in your backup and access controls.

Limit node-agent privileges

File System Backup runs node-agent pods that need root and sometimes privileged access to read volumes on each node. Apply Pod Security Standards to the Velero namespace deliberately, and document the exemption that the node-agent needs instead of loosening the policy for the whole cluster. Review which workloads share the namespace, because anything else scheduled there inherits the same allowance. Confirm the required security context with the Velero release you deploy.

Treat the Kopia key as a secret

Velero's documentation warns that File System Backup uses a static, common encryption key, so anyone who can read the backup storage can decrypt the backup data. Treat the backup bucket as sensitive, restrict who can list and fetch its objects, and avoid relying on that encryption as the only protection for regulated data. Where snapshots are available for your volumes, weigh them against File System Backup for sensitive workloads. Ask your Velero contact how the key is generated and stored in your version.

Enable Object Lock for immutability

Object Lock lets RGW hold backup objects for a retention period so they cannot be deleted or overwritten by ordinary credentials, including a compromised Velero key. Confirm with the Ceph documentation for your release whether the bucket supports Object Lock, which retention modes it offers, and how legal holds and default retention interact with Velero's own object expiration. Test that a delete attempt fails before you rely on the setting. Plan how the retention period works with your backup schedule and storage capacity.

Testing restores before you need them

A backup is only proven when a restore has succeeded, so the restore test plan matters as much as the schedule. Run restores into a separate test cluster or, if that is not available, into a dedicated namespace that cannot collide with production names, Services or Ingress hosts. Restore a representative set: a stateful application with a persistent volume, a namespace with Secrets and ConfigMaps, and at least one workload that depends on a cluster-scoped resource. Confirm the restore target can reach the Ceph RGW endpoint and the bucket that holds the backup, because a restore that works only from the production network proves less than it appears to.

For each test, check three things and record them. First, verify persistent volume contents against known data, such as checksums or row counts written before the backup, rather than trusting that the pod started. Second, confirm the application starts and passes its own health checks, including connections to external dependencies that the test namespace may not reach. Third, record the elapsed time and every gap you found, such as a missing StorageClass, an image pull failure or a Secret that was excluded by a selector. The recorded time is your realistic recovery estimate, and the gap list is the work for the next cycle. Because File System Backup and snapshot backups behave differently, run both paths if your design uses both.

Repeat the test after each Velero, Kubernetes or Ceph upgrade, since any of them can change how resources are captured or how volumes are mounted. Pay particular attention to unmounted PVCs and hostPath volumes. Velero File System Backup only works on volumes mounted by a running pod, so an unmounted PVC needs a staging pod during backup, and hostPath volumes are not supported by that method at all. Treat these volumes as exceptions in the plan, document how each is protected, and test them separately. Confirm the exact behavior with the Velero documentation for your release, which is linked from the file system backup section.

Common failures when Velero writes to Ceph RGW

This table lists the failures teams most often hit when Velero writes backups to a Ceph RGW bucket, with the likely cause and a check to confirm the fix. Work through the rows in order during setup, since addressing, TLS and credentials usually fail before any backup runs. The main takeaway is that most failures show up as an unavailable backup location, so verify the location status before testing any restore.

FailureLikely causeFix and how to verify
Bucket DNS lookup failsPath-style addressing is not set, so the client expects bucket names in the hostnameSet the S3 URL to the RGW endpoint with path-style addressing enabled, then confirm the backup location reports Available
Signature or region mismatchThe region in the backup location does not match what RGW expects, or the signing version differs from the clientUse the region setting your RGW deployment is configured for and confirm it with your Ceph administrator, then run a test backup and read the Velero server logs for signature errors
TLS certificate not trustedRGW serves a certificate issued by an internal CA that the Velero pod does not trustMount the CA bundle into the Velero server and configure the backup location for it, then check that the location status changes to Available
Bucket policy or user rights missingThe RGW user lacks read, write, list or delete permissions on the backup bucketGrant the access key the bucket permissions Velero needs, then run a backup and confirm the object count in the bucket increases
Node-agent pods blockedPod Security Standards at the restricted or baseline level reject the root or privileged settings that File System Backup needsLabel the Velero namespace for a level that permits node-agent requirements, then confirm node-agent pods are running on every node
hostPath volumes silently skippedFile System Backup does not support hostPath volumes, so they are not capturedInventory hostPath use before go-live, then cover those workloads with another method and verify with a restore test
Unmounted PVCs not capturedFile System Backup only reads volumes mounted by a running podAttach a staging pod to the claim for the backup window or use snapshots where supported, then check the backup contents list the volume
Backup location shown unavailableEndpoint, credentials, TLS or bucket name is wrong, or the bucket does not existRun the Velero backup location status command, check the server logs for the first error, and fix the reported field before retrying

Questions teams ask before running Velero on Ceph RGW

How much effort and time does a Velero on Ceph RGW setup take?

No duration is published for this work, so the honest answer depends on the inputs: the number of clusters, whether volumes are covered by CSI snapshots or File System Backup, whether Object Lock is required, and how far restore testing goes. Each cluster needs its own Velero install, credentials and schedules, and each added volume type or retention rule adds design and test work. Ask for a written scope that lists these items so the effort can be compared like for like. A fixed-scope agreement fixes the deliverables and timeline in writing before work starts.

Can our own team set this up without a partner?

Yes, if your team already runs Kubernetes and has someone who administers the Ceph cluster. The work involves creating an RGW bucket and a dedicated user, configuring Velero with path-style addressing, deciding on snapshot or File System Backup use, and proving restores on a separate cluster. The areas most often underestimated are restore testing and the Object Lock decision, because both require coordination with whoever owns the storage. A partner is most useful when the team lacks capacity or wants an independent restore test before relying on the backups.

What do we need to provide before the work starts?

Plan to provide four things:

  • Ceph administrator access to create the bucket, user and any Object Lock settings on the RGW endpoint
  • Kubernetes RBAC rights to install Velero and its node-agent components on the clusters in scope
  • A separate test cluster or namespace where restores can be run without touching production
  • Written retention decisions covering how long backups are kept, how many restore points exist, and who may delete them

Missing retention decisions are the most common reason a backup design has to be reworked.

Is Ceph RGW supported as a Velero backup location?

Velero supports any S3-compatible object store, and Ceph RGW is listed on Velero's supported providers page among S3-compatible providers that users report working. The same page notes that these providers are not regularly tested by the Velero team, so you carry the responsibility for validating your Velero and Ceph versions together. Confirm the exact Velero version, Ceph release and features such as Object Lock against your own test restore before you rely on the setup in production.

Does File System Backup change what we must protect and test?

Yes. File System Backup moves data with Kopia modules and needs root and sometimes privileged node-agent pods, does not support hostPath volumes, and reads the live file system, which is less consistent than snapshots. Velero's documentation also warns that it uses a static, common encryption key, so anyone with access to the backup storage can decrypt the data. Those points should shape the bucket access policy, the restore tests and the security review. Unmounted PVCs also need a staging pod before they can be backed up.

What should a quote for this work include?

A quote should follow from a written scope and list the clusters in scope, the volume types and whether snapshots or File System Backup are used, the Ceph and Kubernetes versions, the Object Lock requirement, the retention model, and the restore test scope with the number of restore scenarios. It should separate one-time work such as design, installation and the first restore test from any recurring work such as version upgrades, schedule reviews or periodic restore drills. Ask who performs each task and which items you must supply. DeployOpen does not publish a price list, so a quote is prepared from that written scope.

What recurring work keeps Velero backups trustworthy on Ceph RGW?

Backups need ongoing attention because Velero, Kubernetes and Ceph each release updates. Velero publishes regular patch releases, for example v1.18.4 on September 28, 2026, and Kubernetes versions move on a roughly 14-week release cadence with the three most recent minors maintained. Recurring tasks include testing Velero upgrades against your Ceph release, reviewing schedules and retention, confirming bucket credentials are rotated, and running restore tests on a set cadence. Ask any provider to name which of these are included in a retainer and which are billed separately.

How DeployOpen scopes and delivers this integration

DeployOpen agrees the scope, deliverables and timeline in writing before any work starts. For a Velero backup location on Ceph RGW, the written scope usually names the Kubernetes clusters being protected, the Ceph cluster that will hold the backups, the bucket layout, the credential model, the retention rules and the restore tests that will count as acceptance. DeployOpen is a Chainscore Labs company and implements this work inside your own cloud accounts, data centers or air-gapped networks, so the cluster, the Ceph cluster and the backup bucket all remain under your control. Because the backup target sits outside the cluster being protected, the design decisions that matter most are made in the scope, not during configuration: which Ceph failure domains hold the copies, who owns the RGW user, and how long each backup class is kept.

The delivery work covers the bucket and RGW user, the Velero install with its BackupStorageLocation and credentials, the choice between volume snapshots and File System Backup, and the Object Lock or retention settings you decide to use. It also covers the restore tests, run into a separate namespace or cluster, with the results recorded as evidence. Ongoing work such as Velero and Kubernetes upgrades, periodic restore testing and continuous security testing can be handled under an optional retainer. DeployOpen prepares systems, configuration records and test evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. Those come from independent auditors and certification bodies, and you should confirm early which of your controls they will expect to see as evidence.

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.