Hadoop and Spark access to Ceph RGW through S3A

Hadoop and Spark access to Ceph RGW through S3A for data jobs on your own storage

Data teams that need S3 buckets on a self-hosted Ceph Object Gateway (RGW) instead of a public cloud store should use the Hadoop S3A connector, configured against the RGW endpoint. Spark does not have its own path to object storage. It reads and writes through the same Hadoop filesystem layer, so the S3A settings you make for Hadoop apply to Spark jobs too. This section covers the endpoint, path-style access, the region setting, V4 request signing, the Hadoop features to disable for a third-party store, and cleaning up incomplete multipart uploads.

Start by confirming three things on the RGW side before touching any job configuration: the endpoint URL that your Spark and Hadoop clients can reach, the access key and secret for a dedicated user with access only to the buckets the jobs need, and that the gateway accepts AWS Signature Version 4 requests. Hadoop's third-party store guidance requires a store that supports V4 signing, and it requires a non-empty region value even though RGW is not an AWS region. The region string is a label your team chooses and must match in every job, so record it alongside the endpoint in your platform documentation.

Expect to set fs.s3a.path.style.access to true for RGW deployments that do not provide virtual-host style bucket addressing in your environment, and confirm this with your gateway's DNS design before rollout. The S3A documentation also notes that some features may need disabling on third-party stores, including bulk delete, encryption, versioning, storage classes, conditional creation and multipart uploads, so test each job against a non-production bucket first. Incomplete multipart uploads are the usual source of hidden capacity loss. Where no lifecycle rule exists to expire them, plan a scheduled purge job and confirm with your storage team which buckets it may touch. Check the Apache Hadoop third-party S3 stores guide against the Hadoop release your clusters run, since option names and defaults change between versions.

What you need in place before configuring S3A against RGW

S3A failures usually trace back to a prerequisite rather than to a Hadoop setting. Confirm the Ceph release, the RGW user and bucket, the network and trust chain, and the Hadoop and Spark builds before you touch the connector configuration. Each card below is a check to complete first.

Target Ceph release line

Decide between Tentacle (20.2.x) and Squid (19.2.x) before testing, since both are listed as actively maintained on the Ceph releases page and receive periodic backports and security fixes. Pick the line you will run in production, then confirm with the Ceph documentation how RGW behaves on your exact point release, because S3A test results from one minor version do not transfer automatically to another.

RGW user with access keys

Create a dedicated RGW user for Spark and Hadoop jobs instead of reusing an administrator identity. Scope the access key to the buckets the jobs need, and plan how the key is delivered to executors and drivers through a secrets store rather than plain-text job configuration. Ask who rotates the key and how a rotation reaches running jobs.

Bucket and layout decisions

Create the bucket before the first job runs and decide its layout up front: one bucket or several, the prefix scheme for job inputs, outputs and temporary data, and whether the workload needs versioning. Confirm that the bucket permissions let the job user list, read, write and delete, since Spark commit behavior depends on delete and list operations working as expected.

Reachable endpoints and CA trust

Confirm that every Spark driver, executor and Hadoop client node can reach the RGW endpoints over the network path you intend to use, including load balancers in front of RGW. If your certificate authority is internal, install its root certificate in the Java truststore used by Hadoop and Spark on every node, since a trust failure on a single executor often looks like an intermittent job error.

Hadoop and hadoop-aws module match

The hadoop-aws module must match the Hadoop version in your build, and its AWS SDK dependency must match what that module expects. Mixing a Hadoop client from one release with a hadoop-aws jar from another is a common source of classpath errors. Record the exact versions in your build manifest so that upgrades are tested as a unit.

Spark build and V4 signing check

Choose a Spark release from the current downloads page and confirm it is built against the Scala version your jobs use, since Spark 4 is pre-built with Scala 2.13 and no longer supports Scala 2.12. Then verify that RGW accepts AWS V4 signed requests from your Hadoop build, because S3A on third-party stores requires V4 signing and a store that only accepts older signatures will fail at first request.

How to configure S3A for a Ceph RGW endpoint, step by step

This sequence takes a Ceph RGW bucket from creation to a verified read and write path from Hadoop and Spark through the S3A connector. Each step ends with a check you can run before moving to the next one.

Create the RGW user and bucket
The Ceph administrator creates a dedicated RGW user for the analytics workload, generates its S3 access key and secret, and creates a bucket for the job data. Keep the user separate from administrative accounts so that the S3A credentials can be scoped and rotated without affecting other clients. Confirm the bucket exists and that the key pair authenticates by running a listing with an S3 client such as the AWS CLI pointed at the RGW URL. Done means the listing returns the bucket and no authentication error. Ask your RGW administrator which tenancy and quota settings apply to this user before the bucket is shared with jobs.
Set the endpoint and addressing in core-site.xml
On each Hadoop and Spark node that runs jobs, set fs.s3a.endpoint to the RGW URL, including the scheme and port that your gateway serves. If your RGW endpoint does not support virtual-host style bucket addressing, set fs.s3a.path.style.access to true. Hadoop's third-party store documentation lists path-style access as required for stores without virtual-host support, so confirm this with your RGW administrator's DNS and load balancer design. Done means the configuration files load without errors when you run a simple Hadoop command, and the endpoint value matches the URL you tested in the previous step.
Set a non-empty region for the endpoint
Set fs.s3a.endpoint.region to a non-empty value. The Hadoop third-party store guidance requires a region setting for the S3A client, and the value is used in request signing, so it must match what your RGW deployment expects. Agree the exact string with the Ceph team and record it with the endpoint in your configuration standard. Done means the property is present in the same configuration file as the endpoint, and a listing from the Hadoop shell succeeds without a region-related signing error.
Confirm V4 signing and credentials
Set the S3A signing algorithm to AWS Signature Version 4 and supply the access key and secret for the RGW user through your approved credential mechanism rather than plain text in shared files. Hadoop's third-party store guidance states that the store must support V4 signing, so confirm this on the RGW side before you troubleshoot anything else. Done means a Hadoop shell listing of the bucket succeeds with the V4 setting in effect, and a request with a wrong secret fails, which proves the signature is being checked.
Disable features that third-party stores do not support
The S3A documentation notes that some features may need disabling on third-party stores, including bulk delete, encryption, versioning, storage classes, conditional creation and multipart uploads. Turn off each feature your RGW version does not support, starting with the ones your job code triggers. Test each change against the RGW version you run, since support depends on the release and configuration, and confirm the exact behavior with your Ceph vendor or release notes. Done means a Spark job that writes Parquet files to the bucket completes without an unsupported-operation error on the write path.
Configure lifecycle rules or purge incomplete uploads
Multipart uploads that fail midway leave parts in the bucket and continue to consume capacity. The S3A documentation says incomplete uploads can be purged where lifecycle rules are missing, so add an RGW lifecycle rule that expires incomplete multipart uploads after a window your team sets, or run a scheduled purge job from the Hadoop shell if your RGW version does not support the rule. Done means the rule appears in the bucket's lifecycle configuration when you query it, and a test upload interrupted on purpose shows up in the list of incomplete uploads until the purge runs.
Run a write and read round trip from Spark
From a Spark job, write a small DataFrame to an s3a:// path in the bucket, read it back, and compare row counts and a checksum of a column with the source data. Run the job with the same configuration your production jobs will use, so that any setting you missed appears here first. Include a Hadoop shell listing of the output prefix to confirm the objects are where you expect. Done means the counts and checksums match and the listing shows the expected part files, which means the endpoint, addressing, region, signing and feature settings all work for both writes and reads.

How S3A talks to RGW and where Spark fits in

Spark does not talk to Ceph RGW directly. Spark's jobs read and write data through Hadoop filesystem classes, and the S3A connector translates those file operations into calls to the S3 API. The RGW endpoint is therefore reached by every node that runs part of the job: the driver, which plans the work and coordinates it, and the executors, which read and write the objects. Those nodes need network routes to the RGW endpoint, and the endpoint must be the one you configured in fs.s3a.endpoint. A job that works from a laptop can still fail on the cluster if worker subnets cannot reach the gateway, so test connectivity from the cluster nodes themselves before debugging Spark settings.

Spark's cluster overview lists three cluster managers: Spark's own standalone manager, Hadoop YARN and Kubernetes. The choice of manager changes how executors are launched and scheduled, but it does not change how S3A reaches RGW. The same fs.s3a.* settings apply in each case, so you should set them once in the configuration the job uses on every node, not in a per-manager workaround. Confirm with your Spark distribution how these Hadoop properties are passed to the driver and executors, since the mechanism differs between submission methods. Credentials need the same care: keys that are visible in job submission arguments or logs are a common exposure point.

Spark Connect is a client-server option that changes where the storage credentials sit. Its architecture sends unresolved logical plans over gRPC to a server, and the server returns results as Apache Arrow batches. The client needs no storage credentials, because the server runs the S3A calls against RGW. PySpark clients are supported since Spark 3.4 and Scala clients since Spark 3.5. Before choosing it, confirm that the workloads you need run through the Connect server, since not every API and deployment mode behaves the same way, and plan who operates that server and how its access is controlled.

Which S3A settings map to which RGW and Hadoop values

This table maps each S3A property to the value the Ceph RGW side must provide, the team that owns the setting, and the check that confirms it works. Read it as a checklist: fix the RGW side first, then set the Hadoop properties, then verify each one with a small read and write before running production jobs.

S3A setting	RGW value or behavior	Owner	Check that confirms it
fs.s3a.endpoint	Hostname and port of the RGW endpoint, reachable from every Spark executor and Hadoop node	Platform team with Hadoop admins	List a known bucket from an executor node, not only from the driver
fs.s3a.path.style.access	Set to true when the RGW endpoint is an IP address or lacks wildcard DNS for bucket subdomains	Hadoop admins	Confirm reads succeed when the bucket name is not in the hostname
fs.s3a.endpoint.region	A non-empty region string that matches what the gateway expects; confirm the exact value with your RGW configuration	Hadoop admins	Run a signed request and confirm no signature or region error appears in logs
Access and secret keys	An RGW user with keys scoped to the buckets the job needs; store them in a credential provider, not in plain XML	RGW admin issues, Hadoop admins reference	Confirm the job reads its provider path and that keys do not appear in job configuration dumps
fs.s3a.connection.ssl.enabled	true when the gateway serves HTTPS; the certificate chain must be trusted by every JVM	Security or PKI team	Test a TLS handshake from each node type with the JVM trust store in use
Bulk delete	Often needs disabling on third-party stores; confirm that the gateway handles multi-object delete requests	Hadoop admins	Delete a test prefix containing many objects and check the job reports no partial failures
Server-side encryption	Enable only if RGW encryption is configured for the bucket; mismatched settings fail writes	Platform team	Write one object and verify its encryption status with an S3 client
Versioning and storage class	Disable where RGW does not implement these features for the bucket; check bucket policy first	RGW admin with Hadoop admins	Confirm a write and read succeed and that no unsupported-feature error is logged
Multipart upload cleanup	Configure lifecycle rules on the bucket to expire incomplete uploads, since abandoned uploads consume space	RGW admin	List incomplete multipart uploads on a test bucket after a failed job and confirm the rule removes them

Security settings for S3A access to RGW

S3A jobs read and write production data through a single endpoint, so the security decisions sit at the edges: how the endpoint is encrypted, which identity each job uses, how secrets reach the job, and which network paths are open. The cards below cover these decisions in the order most teams need to settle them.

TLS with a trusted chain

Serve the RGW endpoint over HTTPS with a certificate issued by an authority that every Hadoop and Spark node already trusts, including the full intermediate chain. Hadoop clients validate the certificate on each request, so a self-signed or partially chained certificate tends to fail in the middle of a job rather than at setup. Test the endpoint from a worker node with the same JVM truststore the executors use, and confirm that certificate rotation does not require restarting every job. Avoid disabling certificate validation to get a test running.

Least-privilege RGW users per job

Create a separate RGW user for each team or recurring job rather than sharing one administrative key across all pipelines. Grant each user access only to the buckets and prefixes that job reads or writes, and separate read-only consumers from pipelines that write and delete. When a job is retired or a credential leaks, a per-job user lets you revoke access without breaking unrelated workloads. Ask your Ceph administrator which RGW capabilities and bucket policies your release supports before you design the role split.

Bucket policies for prefixes

Use bucket policies to narrow what each identity can do inside a shared bucket, for example limiting a Spark job to a curated/ prefix and denying deletes on raw landing zones. Explicit deny statements protect against a misconfigured job that lists or removes data outside its scope. Note that S3A also calls operations such as bulk delete and multipart listing, so test each policy with the exact operations your connector version issues. Keep policies in version control so reviews can compare them with the job design.

Keeping keys out of job files

Do not place fs.s3a.access.key and fs.s3a.secret.key in core-site.xml, spark-defaults.conf or job scripts that live in source control or shared home directories. Instead, pass credentials through a credential provider such as a Hadoop credential store or the environment of the scheduler, and reference only the provider class in configuration. Rotate keys on a schedule and after staff changes, and confirm that Spark driver logs and the Spark UI do not echo configuration values. Verify the exact provider options with your Hadoop version.

Network path between nodes and RGW

Decide which network carries S3 traffic from Hadoop and Spark workers to the RGW endpoint, and keep it separate from the Ceph cluster and public networks where your design allows. Limit RGW ingress to the subnets of worker nodes and the load balancer that fronts them, and confirm the path from every executor subnet, not only from the edge nodes. Ceph's messenger v2 secure mode encrypts traffic inside the Ceph cluster on port 3300, but it does not encrypt S3 requests from Hadoop to RGW, so the HTTPS endpoint must still be configured separately.

Ceph internal transport

Enable msgr2 secure mode if your security policy requires encryption between Ceph daemons and clients, and confirm it with your Ceph administrator as a cluster setting. This protects traffic between monitors, OSDs and RGW gateways once a request reaches the cluster. It is an internal transport control, so it does not replace HTTPS on the S3 endpoint or change how S3A authenticates. Record it as a separate control in your review so auditors do not read it as encryption of S3 payloads.

Which S3A features to turn off on a third-party store

The Hadoop documentation for third-party S3 stores warns that several S3A features may need to be disabled, including bulk delete, encryption, versioning, storage classes, conditional creation and multipart uploads. Do not assume any of them work against Ceph RGW just because the S3 API responds. Each feature depends on how the gateway implements the corresponding S3 call, and RGW is not the same as AWS S3 for every operation. The practical approach is to treat the feature list as a set of hypotheses and confirm each one against your own cluster before a production job depends on it.

Run a small, representative job for each feature you are unsure about. For example, write and read a few objects with a Spark job that exercises multipart upload, then run a bulk delete against a scratch prefix and confirm the objects are gone. Check the Hadoop and Spark logs for warnings or retries, and check the bucket contents directly with an S3 client. Record which settings you changed, such as the fs.s3a keys you set in core-site.xml or on the Spark submit command, so the configuration can be reproduced and reviewed. Make the same checks again after any Hadoop, Spark or Ceph upgrade, because a change on either side can alter which features behave correctly.

The exact settings depend on the Hadoop and Ceph versions in use, and on the RGW features enabled in your environment. Confirm with your Hadoop distribution's documentation which S3A options it supports, and check the Ceph Object Gateway documentation for the S3 operations your RGW release implements. Incomplete multipart uploads are a separate concern: where lifecycle rules are missing, they can accumulate and consume capacity, so decide early who owns the cleanup policy and how it is verified. Treat any setting you cannot confirm for your versions as unsupported until a test proves otherwise.

Common S3A failures against RGW and how to fix them

This table pairs each common S3A failure against Ceph RGW with its likely cause and the setting that fixes it. Use it when a job fails at first contact with the endpoint, or when storage fills up after jobs have run. Most failures trace back to the endpoint, path-style, region, signing or multipart settings covered in the setup section.

Failure	Likely cause	Fix
Signature mismatch errors	The client is not signing requests with AWS V4, or the store does not accept the signing method in use	Confirm RGW accepts V4 signing and that the Hadoop S3A client is configured for it, then retest a single list or read call before running jobs
403 or path resolution errors	Virtual-host addressing is used, so the bucket name is placed in the hostname, which the RGW endpoint does not resolve	Set fs.s3a.path.style.access to true so requests use the bucket in the URL path
Empty region errors	fs.s3a.endpoint.region is blank, so S3A cannot build a valid signing scope	Set fs.s3a.endpoint.region to a non-empty value and keep it consistent across all Spark and Hadoop clients
Certificate trust failures	The RGW endpoint uses an internal certificate authority that the JVM trust store does not include	Add the internal CA to the Java trust store used by Spark drivers and executors, and confirm the endpoint URL uses the name on the certificate
Bucket not found	Path-style access is misconfigured, or the endpoint points at a different RGW zone or service than the one holding the bucket	Check fs.s3a.path.style.access and fs.s3a.endpoint together, then list the bucket with an independent S3 client to confirm the name and endpoint
Storage filling with incomplete multipart uploads	Failed or cancelled writes leave parts behind, and no lifecycle rule removes unfinished uploads	Purge stale uploads with the Hadoop S3A cleanup tooling, and add a bucket lifecycle rule that expires incomplete multipart uploads after a period your team sets
Unsupported feature errors on writes	Bulk delete, encryption, versioning, storage classes or conditional creation are enabled but not supported by the store	Disable the listed features in the S3A settings that the third-party store documentation names, then confirm which ones your RGW release supports

How to clean up incomplete multipart uploads on RGW

This sequence keeps failed Spark writes from leaving orphaned multipart parts on RGW buckets. It confirms that lifecycle cleanup is supported, sets an abort rule on the bucket, proves the rule works with a deliberately failed job, and sets a recurring check on storage usage.

Understand why failed writes leave parts behind
When a Spark or Hadoop task fails after starting a multipart upload but before completing or aborting it, the uploaded parts stay in the bucket and continue to consume storage. Stores without lifecycle rules for incomplete uploads have no automatic clean-up, so these parts accumulate across job retries and reruns. The data platform team should review recent failed jobs and the bucket's current usage to confirm whether the problem already exists. The output is a short list of affected buckets and prefixes.
Confirm lifecycle support in your RGW version
Ask the storage team to confirm that the Ceph release running your Object Gateway supports bucket lifecycle configuration and the abort-incomplete-multipart action, and that the S3 client used by your jobs can apply lifecycle rules. Check the Ceph Object Gateway documentation for your release and test the behavior in a non-production cluster where possible. The step is complete when the team has written confirmation of supported behavior for the exact release in use.
Add an abort-incomplete-multipart rule
Apply a bucket lifecycle rule that aborts multipart uploads not completed within a set number of days, scoped to the prefixes Spark writes to. The storage administrator or platform engineer sets the rule with the S3 API or an administrative tool, using the retention value the data owners agree on. Confirm the stored configuration by reading it back from the bucket. The step is complete when the bucket returns the rule with the intended prefix and age.
Run a deliberately failing test write
Start a small Spark job against a test prefix on the bucket and stop it partway through the write, for example by killing an executor after some parts have uploaded. Record the upload ID that the job leaves behind by listing in-progress multipart uploads on the bucket. This confirms that the job actually created incomplete uploads that the rule should target. The step is complete when the team has the leftover upload ID and its part count.
Verify the rule purges the incomplete upload
After the configured age has passed, list in-progress multipart uploads again and confirm the test upload no longer appears and its parts no longer count toward bucket usage. Because timing depends on the rule's age setting, the team should wait for the period it configured rather than assume a fixed interval. If the upload remains, review the rule's prefix, the RGW lifecycle processing status, and the S3 client settings with the storage team. The step is complete when the test upload has been removed by the rule.
Clean up existing orphaned uploads
Rules added today do not necessarily remove uploads that started before the rule existed, depending on the release and the tooling available, so list the current in-progress uploads on affected buckets and decide with the data owners which to abort. Abort only uploads that are confirmed to belong to failed jobs. Record the count and bytes reclaimed for the change log. The step is complete when the backlog listed in the first step has been reviewed and the chosen uploads are aborted or documented as retained.
Schedule periodic review of storage usage
Set a recurring review of bucket usage and in-progress multipart uploads, with an owner named for each Spark bucket. The review compares usage against the expected size of stored datasets and flags growth that does not match completed jobs. Include the check in the operations runbook and in monitoring alerts where the storage platform supports them. The step is complete when the review has an owner, a documented cadence agreed with the data platform team, and a first recorded result.

Questions teams ask before connecting Hadoop or Spark to Ceph RGW

How much effort does connecting Hadoop or Spark to Ceph RGW take?

The configuration itself is small, but the effort grows with the number of jobs and the features they rely on. Each job needs an endpoint, a region value, and path-style access where the gateway lacks virtual-host support, so a single cluster with a few jobs is a bounded task. The larger drivers are the Hadoop and Spark versions in use, because S3A behavior differs between releases, and the number of features your jobs touch. Bulk delete, encryption, versioning, storage classes, conditional writes and multipart uploads may need disabling or testing on a third-party store. TLS certificates for the gateway also add work when internal certificate authorities are involved. Ask for a job inventory before estimating anything.

Can an internal team connect Hadoop or Spark to Ceph RGW alone?

Yes, if the team has someone comfortable with Hadoop configuration, Spark job submission and basic S3 administration. The S3A third-party store guidance is documented and the required settings are few. The harder parts are verifying behavior on your own jobs, choosing which optional features to disable, and setting up cleanup of incomplete multipart uploads so storage does not fill quietly. Teams without a Ceph administrator on hand often find the RGW side (users, keys, buckets, lifecycle rules) is where help is most useful. A partner can cover the parts you do not want to learn under deadline, and your team keeps the knowledge afterward.

What does the work need from our side before it starts?

The work depends on access and people more than on hardware. Typically you need administrative access to the RGW side or a named Ceph administrator who can create users, keys and buckets. You also need dedicated test buckets that are separate from production data, so feature tests and cleanup runs do not touch live objects. Network access from the Hadoop or Spark nodes to the RGW endpoints must be confirmed, including ports, proxies and any load balancer in front of the gateway. Finally, name the job owners who can run representative workloads and confirm results. Without job owners, validation tends to stop at a successful connection test.

How long does the integration take?

Duration depends on scope, and no fixed figure applies across environments. A narrow scope, such as one Spark cluster reading and writing a single bucket with known features, moves much faster than a multi-cluster Hadoop estate with many jobs, several Spark versions and certificate work. Timing also depends on how quickly access, test buckets and job owners are provided. Ask for a written scope that lists the clusters, jobs, Hadoop and Spark versions and features in use. That list is the basis for a schedule, and it is the first thing to review when a timeline slips.

Do Spark release dates change how we should plan this?

Yes, they affect when work is scheduled and which version you target. Spark's downloads page lists recent releases, and its versioning policy says feature releases follow a quarterly cadence starting with 4.3.0. Each major and feature branch is maintained for six months, while the last release in each major line is an LTS maintained for 18 months. The 3.5.x extended LTS ends in November 2027 with security fixes only. If your jobs run on a line that is near the end of its support, plan the upgrade before the S3A work, or validate S3A on the version you intend to keep. Confirm your own support commitments before choosing a target.

What should a fixed-scope quote for this work include?

A fixed-scope quote should list the work in writing before it starts. Expect the scope to name the Hadoop and Spark clusters covered, the job inventory, the Hadoop and Spark versions, and the features to be tested or disabled. It should state which Ceph RGW tasks are included, such as user and bucket setup, and which remain with your team. Deliverables should be named, for example configuration applied to each cluster, a test report, and documented cleanup of incomplete multipart uploads. The quote should also describe the access and test buckets you must provide, the acceptance criteria, and what is excluded, such as application code changes. Ask whether ongoing support is priced separately.

What are the most common failures when S3A talks to RGW?

The most frequent problems are endpoint and addressing errors. Teams often leave path-style access off for a store that needs it, or they omit a non-empty region value, and the job fails before any data moves. Signing errors appear when the store does not accept AWS V4 signatures as configured. Some jobs fail on bulk delete or multipart behavior that the store does not support, so the fix is to disable that feature and retest. Incomplete multipart uploads accumulate over time when no lifecycle rule cleans them, which consumes capacity without visible errors. Test each failure mode on a non-production bucket and record the settings that worked.

Does Spark include the storage connector and access control itself?

Spark provides the processing engine and cluster managers, but the S3A connector comes from Hadoop's hadoop-aws module, and Spark does not include a catalog, job scheduler UI or access control layer of its own. Your platform therefore relies on your chosen components for job scheduling and user access. Confirm which Hadoop libraries your Spark build uses and that they match the S3A behavior you tested. Access to buckets is controlled by RGW users and policies, so decide early which jobs get which credentials and how those keys are rotated.

How to verify the S3A connection before production jobs run

Treat verification as a gate: no production job should run against a Ceph RGW bucket until a scripted test plan passes end to end from the same network path, credentials and Hadoop build that the jobs will use. Start with listing and a bucket-level check that confirms the endpoint, fs.s3a.path.style.access and the region setting are being honored, since a misconfigured endpoint often fails late and with unclear errors. Then write a small object and a large object, and a multipart write above the configured part size, so that both the single-PUT and multipart code paths are exercised. Confirm that V4 signing succeeds on each request type, and record the exact Hadoop and AWS SDK versions in the test log.

Next, run a Spark job that writes a Parquet dataset to the bucket, reads it back, and compares row counts and a checksum of a key column against the source. This catches problems that simple file operations miss, such as commit behavior, rename assumptions and partition listing. Include deletes, both single-object and bulk, and check whether the bulk delete feature listed in the Hadoop S3A third-party stores guide needs to be disabled for your RGW version. Deliberately fail a multipart write mid-job, then confirm the incomplete upload is purged or that a lifecycle rule removes it, because orphaned parts consume capacity silently.

Finally, measure throughput against the expected load of your heaviest job, not a synthetic best case. Confirm the target workload size and concurrency with the team that owns the jobs, then run the same tests on hardware that matches production, including network paths, RGW instances and load balancers. The Ceph hardware recommendations advise benchmarking before buying because no two clusters are alike, and the same logic applies to a shared RGW tier. Record latency and failure rates alongside throughput, and repeat the run during a period of background recovery or scrubbing, since that is when slow requests appear. Ask your Ceph operators to confirm the RGW version and any known S3A-related limitations before sign-off.

How DeployOpen scopes and delivers Hadoop and Spark access to Ceph RGW

DeployOpen starts with the Hadoop and Spark jobs, the Ceph RGW buckets they read and write, and the network path between them. We write the S3A configuration, test it against your own RGW endpoint, and hand over a runbook that covers cleanup of incomplete multipart uploads.

How we scope

Inventory the Hadoop or Spark versions, job types and buckets that will use S3A against your RGW endpoint.
Confirm with your Ceph team whether RGW supports the V4 signing, path-style access and region behavior your clients need.
Agree in writing which S3A features will be disabled on your store and why, before any job is changed.

How we build

Write and review the fs.s3a endpoint, path-style access and region settings for each Hadoop or Spark client configuration.
Test read, write, listing and delete paths with representative jobs in a non-production bucket first.
Define a scheduled cleanup for incomplete multipart uploads, with lifecycle rules where your RGW version supports them.

How we hand over

Deliver the configuration files, credential handling notes and a tested set of job submission examples.
Document the disabled features, the reasons for each and the checks that confirm they stay disabled after upgrades.
Provide a runbook for diagnosing signing errors, endpoint failures and growing storage from abandoned uploads.

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, upgrades and continuous testing are available as a retainer, including retests after Spark or Hadoop upgrades.
Ask any partner for a written quote that lists the S3A test cases, environments and handover documents included.