Migrating HDFS data to Ceph RGW with S3A

What migrating HDFS to Ceph RGW involves and how much effort it takes

The move from HDFS to Ceph RGW is mainly a semantics and job-inventory exercise, with the data copy as one step among several. HDFS is a file system with directories, atomic renames and permissions, while RGW is an S3-compatible object store where buckets and keys have different behavior. Before any data moves, you need to know which jobs, pipelines and downstream consumers depend on HDFS behavior, because those are the places where the migration can fail quietly. Effort is driven far more by how many jobs assume HDFS semantics than by the number of terabytes to copy.

Start by inventorying every Spark, MapReduce, Hive, Oozie and scheduler job that reads or writes HDFS paths, including the ones that run rarely. For each job, record the path patterns it uses, whether it renames or moves directories to commit output, whether it relies on hdfs dfs commands or the Hadoop FileSystem API directly, and which committer it uses. Jobs that commit output by renaming a temporary directory need a different commit strategy on an object store. Confirm the S3A connector version and committer behavior with your Hadoop and Spark distributions before you classify any job as ready.

DeployOpen plans this work in writing against your inventory. The written scope lists the jobs in each category, the changes each one needs, the data sets to copy and the verification steps, and it is agreed before work starts. Ask any partner to show how they will classify jobs, which items they consider out of scope, and what they will test in a staging environment. The effort estimate should follow from that inventory, so ask for the quote to be broken down by job category rather than by data volume alone.

Which HDFS capabilities map to RGW and which need redesign

This table compares the HDFS capabilities that Hadoop jobs rely on with the nearest Ceph Object Gateway or S3A equivalent, and notes where a migration needs job or design changes. Rows marked with a check mark map cleanly, while rows marked with a cross or a dash need changes to code, schedules or data layout. Confirm each RGW feature and S3A option against current Ceph Object Gateway and Hadoop documentation before you publish or commit to a design.

Capability	HDFS behavior	RGW or S3A equivalent and migration note
Object read and write	Files are read and written through the HDFS client as byte streams	Maps cleanly to S3 GET and PUT through S3A. Confirm S3A endpoint, path-style access and credential settings for your RGW version.
Bucket listing with prefixes	Directory listing returns children of a path	Maps to prefix-based listing in RGW. Listing is not a directory operation, so confirm pagination behavior and how jobs expand large prefixes.
Server-side encryption	Transparent encryption zones protect data at rest in HDFS	Partly maps. Check which server-side encryption options RGW documents for your release and how keys are managed, then confirm S3A settings for them.
Hierarchical directories	Real directories exist as inodes with their own metadata	Needs design change. Object stores have no true directories, so folder markers and prefix conventions must be defined and tested by every consuming job.
Atomic rename	Rename of a file or directory is a single metadata operation that commit protocols depend on	Needs job change. S3 has no atomic rename, so S3A copies and deletes objects. Redesign commit paths, for example with a committer built for object stores, and confirm the option for your Hadoop version.
Append	Files can be reopened and extended	Needs redesign. Objects are written whole, so jobs must write new objects or partitions instead of appending to existing ones.
Permissions and ACLs	POSIX-style owner, group and mode bits, plus extended ACLs	Needs design change. Map access to bucket policies, RGW user accounts or identity-provider groups, and confirm which policy features your RGW release supports.
Snapshots	Point-in-time directory snapshots are available to jobs and recovery	Needs redesign. Use bucket versioning, lifecycle rules or copy-based backups, and confirm which of these your RGW configuration supports.
Replication factor and block locality	Replication is set per file, and schedulers use block locations for data-local tasks	Needs redesign. Durability moves to the Ceph pool and CRUSH rules, and locality is lost, so plan for network throughput and test job runtimes.

Documented copy paths and client configuration for S3A on RGW

Three documented routes connect HDFS data to an RGW bucket: a Hadoop filesystem copy pointed at an s3a:// destination, S3A connector settings aimed at the RGW endpoint, and direct use of the RGW S3 API as the target. Each route moves bytes differently and preserves a different subset of HDFS metadata, so choose them by the job and data profile in front of you.

Hadoop filesystem copy to s3a

Hadoop's distributed copy tooling can read from HDFS and write to an s3a:// destination, which suits bulk transfers of large directory trees run as a cluster job. It copies file contents and directory structure but does not carry HDFS ownership, permissions, extended attributes or snapshots onto object keys, so plan a separate record of those attributes. Confirm the exact tool options and flags against the Hadoop release you run, since they change between versions.

S3A connector endpoint settings

The S3A connector reads its endpoint from Hadoop configuration, so the RGW URL, path-style access and credential provider are set in core-site.xml or per job. Name every configuration key from the Hadoop S3A documentation for your Hadoop version, and test each one with a small bucket before a full run. Misconfigured endpoint or path-style settings are a common cause of failed jobs, so verify with a listing and a read-back.

RGW S3 API as the target

Writing directly through the RGW S3-compatible API suits applications that already speak S3 and need object-level control. The Ceph Object Gateway offers S3-compatible and Swift-compatible APIs, and the target buckets can use its documented compression and multi-factor authentication features. Confirm which S3 operations your clients use against the RGW API compatibility documentation before cutover.

What copies do not preserve

Object stores have no true directories, so HDFS directory semantics, atomic renames and in-place appends do not map cleanly onto S3 writes. Copy tools also do not replay HDFS permissions, quotas or snapshots. Document each dropped property and decide how jobs will recreate it, because a copied tree can look complete while losing the metadata that downstream jobs depend on.

Validation after each copy

Compare object counts, total bytes and checksums for a sample of files between HDFS and the target bucket after every copy pass. Listing results should match the source tree, and the S3A client should read back a sample without errors. Record the results per path so the team can rerun only the failed partitions rather than the full migration.

Why atomic renames break first and how to find the jobs that depend on them

S3 has no native directory rename. On HDFS, renaming a directory is a metadata operation that moves a whole tree at once, and many Hadoop and Spark commit protocols rely on that behavior to publish output only when a job succeeds. On RGW, the S3A connector has to emulate the rename by copying each object to its new key and then deleting the original. That copy-and-delete sequence is per object, not per directory, so a job can fail partway through and leave the destination with some files published and others missing. Any committer that assumes a rename either fully happens or does not happen will behave differently on RGW, and the difference often shows up only under failure or on large outputs.

Start the search by listing every job that writes to HDFS and reading the code paths that produce its output. Look for calls to rename on FileSystem or FileContext, for the FileOutputCommitter algorithm version in job configuration, for mapreduce.fileoutputcommitter.algorithm.version, and for any custom committer. Then check the job's wrapper scripts, shell steps and orchestration definitions for hdfs dfs -mv, -mv inside pipelines, and any step that moves a staging directory into a final location. Include jobs owned by other teams, scheduled reports, and one-off repair scripts, because these are the places where a directory move is most often written as a quick shell command and forgotten in the inventory. Confirm the S3A and committer behavior against the Hadoop version you run, since committer options change between releases.

The fix belongs in the committer or the job design, not in a storage setting on the RGW side. Choose a committer built for object stores, or write output to a unique location and publish it with a single manifest or marker that readers check, then test each job's output under failure. Kill tasks and the driver at several points, rerun the job, and check for partial files, duplicated records and files visible to readers before the job reports success. Record the results per job, because a job that passes a clean run can still fail when a task retries. Treat any job that cannot be tested this way as a migration blocker until its publish step is redesigned.

How Spark, MapReduce and direct clients each change on RGW

Spark applications usually change less at the storage layer than they change in design. Spark reads and writes through Hadoop filesystem connectors, so s3a:// paths and S3A settings carry over, but Spark does not decide where tables live or who may read them. Spark's cluster overview lists three cluster managers: its standalone manager, Hadoop YARN and Kubernetes. The manager you use determines how endpoint and credential settings reach each driver and executor.

Cluster manager settings. Standalone, YARN and Kubernetes each distribute configuration differently. Confirm where fs.s3a.endpoint and credentials are set for every submission path.
Catalog and table format. Spark has no catalog or access control layer of its own. Choose the table format and catalog in the target design before you move any table.
Access model. Decide how permissions that were enforced by HDFS paths will be enforced on buckets, prefixes and catalog objects.
Version lines. Check your target against the Spark versioning policy. Spark 4.2.0 was released on July 14, 2026, 4.1.x and 4.0.x receive maintenance releases, and the 3.5.x extended LTS ends in November 2027 with security fixes only.
Scala 2.13 for Spark 4. Spark 4 is pre-built with Scala 2.13 and no longer supports Scala 2.12, so application jars built for Scala 2.12 must be rebuilt before cutover.

Confirm the Spark and Scala versions of every application before you plan the bucket layout, because the catalog and table format decisions depend on them.

MapReduce and Hive jobs depend on HDFS in ways that are easy to miss. Job configurations, Hive table locations and scheduler parameters often contain hdfs:// URIs that were written years earlier. The data copy can succeed while jobs still point at the old cluster, so inventory these references before you copy anything. Confirm the exact Hadoop and Hive versions in use, because S3A behavior and configuration keys depend on the release.

Hard-coded paths. Search job definitions, Hive metastore locations and wrapper scripts for hdfs:// and relative HDFS paths, and record each one with its owner.
S3A configuration. Set the endpoint, path-style access and credentials through the Hadoop configuration your version supports. Check each key against the S3A documentation for that release.
Output committers. Committers that publish results by renaming temporary directories assume cheap, atomic renames. On S3A, a rename copies and then deletes objects, so review committer choice and run representative jobs before production cutover.
Metastore cutover. Repoint table locations only after copied data passes row counts and checksums. If jobs cannot pause, plan a period where both paths are readable.
Orchestration parameters. Check whether YARN queues, Oozie workflows or Airflow DAGs pass HDFS paths as arguments that need updating.

Run one representative job per table family against the target before you schedule the full move, and record the result for each committer and path.

Sequence for inventory, pilot, parallel run and cutover

This sequence moves HDFS-backed data and jobs to Ceph RGW in gated stages, so each dataset and job is proven on the new path before the old one is retired. Every stage ends with a written exit criterion that the client approves.

Inventory paths, jobs and owners
DeployOpen works with your data platform team to list every HDFS path, the Spark, Hive, MapReduce or other jobs that read or write each path, the schedules, the downstream consumers and a named owner for each. The inventory records approximate data volume, file counts and small-file patterns, since these shape both the copy plan and the RGW bucket layout. The exit criterion is a signed inventory that the data owners have reviewed and that accounts for every path referenced by production jobs.
Classify jobs by HDFS dependency
Each job is tagged by how much it depends on HDFS semantics. Jobs that rely on atomic directory renames for commit, on HDFS-specific permissions or extended attributes, on append, or on locality-based scheduling are flagged for redesign before any data moves. Jobs that only read and write whole objects are candidates for early migration. The output is a classification table with a proposed wave for each job, which the owners and the platform team approve before pilot work begins.
Confirm RGW cluster readiness
The Ceph release is chosen from the actively maintained releases on the Ceph releases page, and the hardware is checked against the Ceph hardware recommendations, including a dedicated drive per OSD, shared SSDs for WAL and DB on HDD-based OSDs, enterprise-class SSDs for metadata, bonded network links across two switches, and memory sized for peak use. Benchmarks run before purchase or handover. The exit criterion is a readiness report showing measured throughput, a tested RGW endpoint with TLS, and an agreed authentication model for the Spark clusters.
Migrate a pilot dataset and one job
One low-risk dataset is copied to an RGW bucket, and one job that the classification marked as object-friendly is reconfigured to use S3A against that bucket. The team records copy checksums, object counts, job runtime and any S3A errors, and compares the results against the HDFS run. Problems found here, such as commit behavior or listing performance, feed back into the job configuration and the bucket design. The exit criterion is a successful end-to-end run that the job owner has validated against agreed output checks.
Run both paths in parallel
Selected jobs run on HDFS and on RGW with the same input, and their outputs are compared record by record or by agreed checksums and row counts. Differences are logged and traced to a cause, whether that is file ordering, timestamp handling, partial-commit behavior or a configuration mismatch. The parallel run continues across enough business cycles for the owners to trust the results. The exit criterion is a comparison report with every difference explained and accepted in writing by the data owners.
Cut over by dataset and by job
Each dataset and its dependent jobs move to RGW as a unit, following the waves from classification. Producers and consumers switch their endpoints, HDFS paths are set to read-only for the cut-over dataset, and monitoring watches job failures and latency. Rollback steps are written down for each wave before it starts. The exit criterion for each wave is a sign-off from the job owner confirming that outputs match and that the rollback path has been rehearsed.
Decommission HDFS after agreed retention
Once all waves are signed off, HDFS data is kept read-only for a retention period that the client and DeployOpen agree in writing, so that late discoveries can still be checked against the original data. The retention period, the audit evidence of final comparisons and the deletion record are documented. Cluster hardware is released only after that period ends. The exit criterion is a written decommission approval from the data owner and the security or compliance lead.

What drives the effort and cost of the migration

This table lists the factors that most change how much work the HDFS to Ceph RGW move involves, how each one affects effort, and what to request in a written quote. Use it to see which answers your team must gather before scoping. The main takeaway is that job dependencies on HDFS semantics and the length of the parallel run usually matter more than raw data volume.

Driver	Effect on effort	What to ask for in a quote
Jobs with rename or append dependencies	Each job that relies on atomic directory renames, appends or HDFS-specific commit behavior needs redesign or testing. This is often the largest variable in the work.	A list of jobs classified by HDFS operations used, with the proposed change or committer approach for each
Volume and file count	Total bytes set the transfer window. Large numbers of small files raise copy overhead, listing time and request counts against RGW, even when total volume is modest.	Inventory output with total bytes, file count, and a size distribution per dataset, plus the copy tool and parallelism assumed
Number of consuming applications and owners	Each application team needs endpoint, credential and path changes, testing and sign-off. Effort grows with owners, not only with applications.	A named owner per application, the cutover approval steps, and who performs regression testing
Required parallel run length	Running HDFS and RGW side by side means validating outputs twice and keeping sync or reconciliation jobs active. Longer runs add recurring operational effort.	The parallel run window the client requires, the reconciliation method, and who monitors discrepancies during it
Security integration with RGW	Connecting RGW to Keystone or LDAP authentication, and mapping HDFS Kerberos identities to S3 credentials, adds design and testing work beyond the storage copy itself.	The chosen authentication backend, credential issuance process, and how Kerberos-based principals will map to S3 access
Network capacity between clusters	Bandwidth and latency between the Hadoop and Ceph clusters set copy throughput. Constrained links lengthen the transfer and may require scheduling outside business hours.	Measured throughput between the two networks, the planned transfer window, and any bandwidth limits or change windows
Number of Spark versions in use	Each Spark line may need a different S3A setup, JAR set or committer configuration. Mixed versions multiply test cases.	The Spark versions in production, the S3A configuration per version, and the test matrix to be covered

Failure modes that usually appear and how to prevent them

Most HDFS-to-RGW migrations fail in predictable places: output that was never committed, copies that were interrupted, listings that changed underneath a job, and configuration that drifted between environments. Each card below names one failure, the check that catches it, and the decision to make before the copy starts.

Partial output after a failed job

HDFS jobs often rely on a rename at the end of a task to publish output atomically. Object storage has no native directory rename, so a committer that writes directly to final keys can leave half-written output visible after a failure. Decide the output commit strategy before the first run, and validate row counts or record checksums at the job level before downstream jobs read the results. Test a deliberately killed job in a non-production bucket.

Duplicate or missing files after interruption

An interrupted bulk copy can leave some objects missing and others re-copied on retry. Without a manifest, operators cannot tell which side is complete. Generate a file list with sizes and checksums from HDFS before copying, compare the destination manifest after each pass, and keep the copy tool's retry and overwrite behavior explicit in the runbook so that a rerun does not silently replace good data.

Directory listings that differ between runs

Jobs that enumerate a prefix to decide what to process can see different results than they did on HDFS, especially when new objects arrive during a run or when a prefix is treated as a directory. Confirm the listing behavior of the RGW version and S3A settings you plan to use with a test job, and make input sets explicit (a manifest or a date-partitioned list) rather than relying on whatever a listing returns at start time.

Credentials and endpoints that differ by environment

S3A jobs point at an endpoint, path style setting and access keys that often differ between development, test and production clusters. A job that works in one environment can read from the wrong bucket or fail with authentication errors in another. Keep endpoint and credential settings in one versioned configuration per environment, use separate RGW users per application, and confirm each environment's settings with a read and write smoke test before any job runs.

Undersized network or OSD capacity

Copy throughput is often limited by the network path, the number of OSDs serving the bucket data pool, or the write pattern of many small files, and these limits appear only under load. Benchmark the cluster with a representative object size and concurrency before buying hardware or setting a cutover date. The Ceph hardware recommendations advise benchmarking before buying because no two clusters are alike, and bonded links across two switches reduce the risk of a single network failure stopping the copy.

HDFS data still in use after cutover

When some jobs, users or scheduled tasks still write to HDFS after the copy, the RGW bucket and the HDFS copy diverge and the cutover cannot be reversed cleanly. Agree a freeze window with the owners of every job and pipeline, disable writes to the affected HDFS paths, run a final delta copy with checksum comparison, and keep HDFS read-only for a defined period so that rollback remains possible.

Questions teams ask before moving HDFS data to Ceph RGW

What does moving HDFS data to Ceph RGW cost?

The cost is driven by the work involved, not by a fixed figure. The main drivers are the number of jobs that read or write HDFS, how many of them depend on rename, append or directory semantics, the volume of data to copy and how long the parallel run lasts. Ceph itself is open source under LGPL-2.1 or LGPL-3, so the spend is on hardware, on staff time and on any external services. One-time costs include the job inventory, code changes, test runs, the copy and the cutover. Recurring costs include the Ceph cluster, its capacity growth and the operating team. When you request a quote, ask for the scope split by job inventory, code changes, data copy, cutover and handover, and ask which items are fixed and which depend on findings from the inventory.

How long does an HDFS to RGW migration take?

The duration depends on the job inventory and on how long the parallel run needs to be. A small set of jobs that all tolerate object storage semantics can move quickly. A large estate with many rename-dependent pipelines, multiple downstream consumers and strict reconciliation rules takes longer, mostly in testing and in agreeing cutover windows. Data volume and network throughput between the Hadoop and Ceph sites set the copy time. Ask any partner for a timeline that is tied to a written inventory, a defined parallel run period and named job owners, rather than a single headline duration.

Can our own team migrate HDFS data to Ceph without outside help?

Yes, if your team already has operators who run Hadoop and operators who run Ceph, and if there is time for a parallel run in which HDFS and RGW both serve the jobs under test. The work then rests on inventory, S3A configuration, copy tooling, validation and cutover. Outside help is most useful for the parts your team has not done before: checking rename and append dependencies across the estate, designing the reconciliation checks, and running cutover rehearsals. An external partner can also run a fixed-scope inventory or pilot first, so you can decide on the rest with real findings. DeployOpen implements, hardens, documents and supports self-hosted platforms in your own infrastructure, but it does not issue audit reports or certifications.

What does the migration need from our side?

Four things need to be in place before the work starts:

Job owners who can explain what each pipeline reads, writes and expects from directory operations, and who can sign off on test results.
A test window in which jobs can run against both HDFS and the RGW target without affecting production.
A Ceph cluster ready for the target, with capacity, networking, authentication and buckets planned for the data that will land there. Hardware guidance for Ceph is in the project documentation.
An agreed cutover policy that states who decides go or no-go, how writes are frozen and how the team rolls back if reconciliation fails.

Without these, the parallel run cannot produce evidence, and the cutover date becomes a guess.

Do our Spark jobs need to be rewritten for object storage?

Usually only the jobs that depend on HDFS semantics need changes. Most read and write jobs work through the Hadoop filesystem interface once S3A points at RGW. The jobs that need attention are those that rely on atomic directory renames to commit output, on append operations, on in-place file modification or on directory listings that assume HDFS consistency. Those patterns are found in the job inventory, which should list the commit mechanism and any rename or append calls for each job. Confirm the behavior of your Spark, Hadoop and S3A versions in testing rather than assuming it, because committer choice and listing behavior affect results.

Does Ceph RGW support the S3 features our jobs use?

RGW offers an S3-compatible API, but support for individual features varies by feature and release, so confirm each one against the RGW documentation for the exact Ceph release you plan to run. Build a list from the job inventory and the S3A configuration: bucket operations, multipart upload, listing behavior, object versioning if used, server-side encryption, lifecycle rules and any authentication method. For each item, record the documented behavior, the test that proves it and the owner who accepts the result. Features that are not covered should be handled in the job design before cutover, not discovered during it.

How DeployOpen scopes, delivers and supports an HDFS to RGW migration

DeployOpen approaches a move from HDFS to Ceph RGW as a fixed-scope engagement that starts with an inventory of the jobs, their HDFS dependencies and the data that must move. We build and test the S3A configuration, copy and verify data, and hand over documented runbooks inside infrastructure you control.

How we scope

We inventory Spark, Hadoop and other jobs that read or write HDFS paths and flag any reliance on atomic renames.
We agree in writing which datasets, buckets, credentials and S3A settings are in scope for the migration.
We record the Ceph cluster version, RGW endpoints and features to disable so the scope matches your environment.

How we build

We configure S3A connection settings against your RGW endpoints and test them with representative read, write and list jobs.
We design the bucket layout, naming and copy method, then run a pilot copy and verify object counts and checksums.
We adjust job code or commit logic where HDFS rename semantics are required, and test those changes before cutover.

How we hand over

We deliver the written scope, configuration files, copy scripts, verification reports and a cutover runbook.
We document rollback steps for each job group and the criteria your team uses to confirm a successful cutover.
We walk your platform and data engineering teams through operations so they can repeat or extend the copy process.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after the handover.
A retainer can cover Ceph RGW and Spark S3A configuration changes when your Spark or Ceph versions move.
Ask in your quote which post-cutover checks, reviews and response arrangements are included and which are optional.

Migrating Hadoop HDFS data to Ceph RGW with S3A