How Drupal file storage on S3 works for multi-server sites
A Drupal site whose public and private files sit on one web server's disk cannot simply run across several servers or containers, because each node would hold its own copy of the files and users would see different results depending on which node answered. The supported approach is to move both public and private files into an S3-compatible bucket using the contributed S3 File System module, which writes Drupal's file operations to object storage instead of local disk. Confirm the module's current status, supported Drupal versions and configuration options on its Drupal.org project page before planning the build, since the integration method is the one to validate against your own environment.
The bucket can be Amazon S3 or a self-hosted service such as Ceph Object Gateway, which exposes an S3-compatible API. That choice shapes what the reader must configure: credentials and their scope, the endpoint URL, the region setting, whether path-style or virtual-host addressing is used, and how public and private paths are separated. A self-hosted endpoint needs its own TLS certificate trust chain inside the PHP container image, and the bucket policy must keep private files unreachable from the open web. Ask the module's maintainers and your storage team to confirm which S3 features the module relies on so that the chosen backend supports them.
Before writing configuration, decide where private files will live, who rotates the access keys, and how a failed bucket write surfaces to editors. Test uploads, image styles, file deletion and cron-driven cleanup against a staging bucket with the same endpoint type as production. Confirm that generated image derivatives and temporary files behave as expected, because these multiply object requests and can affect both cost and latency on a metered service.
What you need before connecting Drupal to object storage
Most object storage problems show up before any file moves, in versions, credentials and network paths. Confirm these prerequisites first, because each one changes how the S3 File System module is configured and tested. The cards below cover the Drupal runtime, the database, the bucket and its access key, the endpoint, and the Ceph cluster if you run your own.
Drupal core and PHP version
Drupal 10 requires PHP 8.1 or later and Drupal 11 requires PHP 8.3 or later, so confirm the PHP version on every web node before you add the module. Then check the S3 File System release that matches your Drupal core version on its project page, because module branches follow core releases. Mismatched versions are a common cause of failed installs in container builds, so pin both in your Composer lock file.
Supported web server and database
Drupal runs on Apache, Nginx and other web servers, with MariaDB, MySQL, PostgreSQL or SQLite as the database. Confirm your combination is on the supported list before you change storage, since a storage move is also a good time to review the rest of the stack. Record the versions in your runbook so that a later upgrade can be checked against the same list.
A bucket with a scoped access key
Create a dedicated bucket for Drupal files, separate from backups and logs, and issue an access key that can read and write only that bucket. Public and private files usually need different prefixes or buckets, because private files should never be reachable through public URLs. Ask who rotates the key, where it is stored, and how it reaches the web nodes without being written into the codebase.
A reachable endpoint with a valid certificate
Every web node and cron host must reach the S3 endpoint over HTTPS, and the certificate must be trusted by the PHP runtime, including for internal certificate authorities. Test name resolution, firewall rules and path style versus virtual-host style addressing from inside the same network zone as Drupal. A certificate error often looks like a Drupal fault, so verify the endpoint with a plain client before you configure the module.
A supported Ceph release for self-hosted storage
If you run Ceph, plan on a release that is still actively maintained, such as Tentacle or Squid, and check its listed end-of-life date before you commit. Deploy the Object Gateway (RGW) with a load balancer or ingress in front of it, since Drupal sees only the endpoint. Confirm with your storage team who patches the cluster and how upgrades are scheduled.
How to configure S3 File System step by step
This sequence moves a Drupal site's public and private files onto S3-compatible object storage in an order that keeps existing files reachable at every stage, so each web node reads and writes the same store before the cutover is final.
Confirm the target stack and compatibility
Before any change, confirm the Drupal core version, PHP version and the S3 File System module release that supports it on Drupal.org. Record the storage target, whether that is Amazon S3 or a self-hosted service such as Ceph, along with its endpoint, region setting and path-style or virtual-host addressing requirements. For Ceph, confirm the release line you run is one that receives backports, since that affects how long the cluster stays supported. The step is done when the team has a written compatibility note and a dated backup of the database and the current files directory.
Install the module with Composer
A developer adds the S3 File System module and its required libraries to the project with Composer, so the dependency is versioned in composer.json and lockfile rather than copied by hand onto each server. The change goes through the normal branch, code review and staging deploy used for the rest of the site. The check is that composer install on a clean build pulls the same module version on every web node, and the module appears in the extension list on staging.
Create the bucket and a least-privilege user
An infrastructure or storage administrator creates a dedicated bucket for the site, with separate prefixes for public and private files if the design calls for it, and a service identity that can only read, write, list and delete objects within those prefixes. Versioning, encryption at rest and access logging are decided here and written down. The check is that the service identity can complete a put, get and delete on a test object inside the bucket and is refused when it attempts the same call on another bucket.
Set the endpoint, region and credentials in settings
A developer places the bucket name, region, endpoint URL and credentials in environment variables or a secrets store, then references them from settings.php or the module's configuration so no keys sit in version control. For self-hosted S3-compatible storage, the endpoint points to the internal gateway address. The check is that Drupal's status report shows the module configured, and a small test script run from one web node can list the bucket contents without errors.
Move the public file path first
The team sets the public stream wrapper to the S3 bucket, leaving the private path on local disk, and runs the module's copy process for the public files directory with a dry run first. Public files are mostly images and assets that anonymous visitors request, so they are the lower-risk half of the move. The check is that a file count and total size from the source directory match the object count and size in the bucket, and that a sample of image URLs loads on the site.
Move the private file path and verify the copy
Once public files are verified, the team repeats the process for the private files path, which typically holds uploads that should only be served through Drupal's access checks. The module serves these through Drupal rather than as public object URLs, so access rules must be tested with a user who lacks permission. The check is a checksum comparison for a sample of private files, plus a confirmation that a restricted file returns a denial to an unauthorized account.
Switch the file system and test from each web node
During a short agreed window, the team freezes content edits, runs a final incremental copy, changes the default file system to the S3 wrapper in configuration, and clears caches. Then a tester uploads a new image and a new private document through the Drupal interface on each web node, and reads each one back from a different server to confirm the object is visible across the cluster. The step closes when every node returns the same file, and the rollback path to the local files directory has been rehearsed and documented.
Why multi-server Drupal needs shared file storage
Drupal keeps uploaded files on a single file system path, so every web node must see the same files. Drupal manages files through two stream wrappers. The public stream (public://) serves files directly from the web root and is intended for assets anyone can request, such as images and downloads. The private stream (private://) lives outside the web root and is served by Drupal after an access check, which is why it suits documents that need permissions. Both streams write to a path on disk by default. When a site runs behind a load balancer, a file uploaded through one node is missing on the others, so editors see broken images and users see errors. The same problem appears in containers: a container's filesystem is discarded when it is replaced, so uploads written inside it disappear on the next deployment. Moving the file streams to object storage gives every node one shared store, and the decision to make is which stream moves, to which bucket, and whether the bucket is public or private.
Generated derivatives are where teams most often get surprised. Image styles create resized copies of uploaded images on first request, and those copies are written to the same file stream as the original. On a multi-node site, a derivative built on one node must be visible to the others, or each node regenerates it and the store fills with duplicates. Private files raise a different question, because Drupal must read the file to serve it, which means the application, not the browser, talks to the bucket. Cron and queued tasks add a further dependency: a scheduled job on one node may delete temporary files or process uploads that other nodes wrote, so the reader should confirm which node runs cron and how temporary files are cleaned up. Check how the chosen module handles derivatives, cache headers and deletion before go-live, and test an upload followed by an image style request from a second node.
The architecture also changes what your team must operate. Object storage becomes a runtime dependency of the site, so an outage or misconfigured credential affects page rendering for images and downloads, not only uploads. Access keys belong in environment or secret configuration rather than in exported configuration files, and the bucket policy must match the public or private split decided for each stream. Whether Amazon S3 or a self-hosted service such as Ceph provides the bucket, the team should confirm endpoint, region and path-style settings with the module documentation, and should rehearse restoring files alongside the database, since neither is useful without the other.
Settings and file paths mapped between Drupal and the bucket
This table maps each Drupal file setting and bucket element to what it controls, so you can check that public files, private files, endpoints and caching behave as intended. Confirm every configuration key name against the release of the S3 File System module you install, because keys and their locations change between releases.
| Setting or element | What it controls | What to confirm before go-live |
|---|---|---|
| Public file scheme (public://) | Where Drupal writes publicly served files such as images and theme-attached media, and which URL prefix the browser requests | Confirm the scheme is mapped to the S3 stream wrapper and that the public URL resolves to the intended bucket path |
| Private file scheme (private://) | Where Drupal stores files that must pass through access checks, such as protected downloads and some form uploads | Confirm the private scheme points to a bucket prefix or a separate bucket that is not publicly readable |
| Bucket name and prefix | The bucket that holds the files and the folder path inside it, which separates Drupal files from other applications using the same bucket | Confirm prefixes for public and private files differ and that other workloads cannot write into them |
| Endpoint and region | The API address Drupal calls, which for self-hosted Ceph is the Ceph Object Gateway address, and the region value used in request signing | For Amazon S3, confirm the region matches the bucket; for Ceph, confirm the endpoint is reachable from every web container |
| Access key scope | The credentials Drupal uses to read, write and delete objects in the configured bucket and prefixes | Confirm the key has only the object actions Drupal needs on its own bucket or prefix, and that keys are stored outside the codebase |
| Cache and public URL settings | How long browsers and reverse proxies cache served files and whether URLs point to the bucket or to a CDN in front of it | Confirm which URL form is generated, and whether purging occurs when a file is replaced or deleted |
| Path-style or virtual-host addressing | Whether requests place the bucket name in the URL path or in the hostname, which some self-hosted gateways require | Confirm the addressing style your gateway supports and test an upload and a public fetch with it |
Securing private files and bucket access
Private Drupal files are only as safe as the bucket policy, the URLs that reach them and the credentials that Drupal uses to read them. The cards below cover the controls that connect the Drupal side of the setup with the Ceph side, from bucket permissions to cluster traffic.
Least-privilege bucket policies
Give the Drupal service account only the actions it needs on the buckets it uses. Public and private files usually belong in separate buckets, so a policy for private files can deny anonymous reads outright. Check that the policy allows object writes and reads for the file paths Drupal uses, and nothing like bucket administration. Ask the Ceph administrator to confirm how the Object Gateway evaluates bucket policy alongside user policy in your version before you rely on either alone.
Keeping private files off public URLs
Private files should never be reachable through a bucket URL that anyone with the link can open. Confirm in the S3 File System settings that private files are served through Drupal's access checks, and that the public and private schemes map to buckets with different access rules. Test by requesting a known private object URL from an unauthenticated client and confirming it is refused. Pay attention to CDN and reverse proxy rules, which can cache responses that should never leave Drupal.
TLS to the storage endpoint
Every request from Drupal to the S3 endpoint should use HTTPS, including traffic inside a private network. Terminate TLS at the Object Gateway or at a load balancer in front of it, and decide who holds the certificate and how renewal works. Confirm that the endpoint setting in Drupal uses an https URL and that the PHP runtime trusts the internal certificate authority. Plain HTTP between web containers and storage is a common gap in container deployments.
Secrets stored outside the codebase
Access keys for the bucket user should not appear in settings.php, exported configuration or the Git repository. Inject them through environment variables, an orchestrator secret store or a dedicated secrets manager, and read them from Drupal's settings in a way that does not write them to exported config. Plan rotation: a key change should require a restart of the web tier and no code change. Ask who can read the secret in each environment, and remove developer copies of production keys.
Messenger v2 secure mode for cluster traffic
Ceph's internal traffic between monitors, OSDs and other daemons carries cluster data. The messenger v2 protocol, which listens on port 3300, offers a secure mode that encrypts all post-authentication traffic with AES-GCM and adds integrity checks. Confirm with the storage team that secure mode is enabled on the cluster and that the network placement of these daemons matches your design. This protects the storage backend, not the Drupal application, so it complements the controls above.
Object Gateway authentication as documented
The Object Gateway documentation lists supported authentication integrations, including Keystone and LDAP, along with multi-factor authentication and Open Policy Agent integration. Choose the option your environment already operates and confirm its behavior in the Ceph release you run. Do not assume a particular integration works with the S3 File System client without testing. Dashboard single sign-on is a separate administrator concern and does not control how Drupal reads or writes files.
Common failures when Drupal writes files to S3
This table lists the failures teams most often meet when Drupal public and private files move to S3-compatible storage, with the fix for each and the logs or checks that confirm the cause. Read it as a triage guide: match the symptom you see, apply the fix, then run the confirming check before changing anything else. Several failures look alike from the browser, so the checks matter more than the symptom.
| Failure | Likely cause and fix | Logs or checks that confirm it |
|---|---|---|
| Uploads fail or resolve to the wrong host | Path-style or virtual-host addressing does not match the endpoint. Self-hosted Ceph RGW usually needs path-style addressing with a custom endpoint. Set the bucket addressing style in the S3 File System settings to match what your endpoint supports, and confirm with your storage team. | Look for DNS or 404 errors naming the bucket in the Drupal watchdog log. Test a single PUT and GET with the AWS CLI against the same endpoint and addressing style. |
| Signature mismatch or request rejected as expired | Clock skew between the Drupal web or cron containers and the storage cluster. Signed requests fail when the time difference exceeds the server's tolerance. Synchronize all nodes with NTP or chrony and check container hosts, not only the storage nodes. | Check storage gateway logs for signature or request-time errors. Compare date -u output on the web, cron and storage hosts. Skew is confirmed when errors clear after time sync. |
| Public files return 403 or do not load in the browser | The bucket or object lacks the read policy or public-read setting that public files need, or the bucket has no CORS rule for the site origin. Apply a bucket policy scoped to the public files prefix only, and add a CORS rule that allows the site's origin for the methods your front end uses. | Request the file URL directly and inspect the response code and headers. Check browser console for CORS errors. Confirm the policy with the storage admin API or CLI. |
| Private files reachable through a direct object URL | Private files were written to a bucket or prefix readable by anonymous users, or a public URL style was used for private stream wrappers. Keep private files in a non-public bucket or prefix, serve them through Drupal's access checks, and use short-lived signed URLs only where the design requires them. | Request a known private file URL without a session and expect denial. Review bucket policy and ACLs. Audit the Drupal file table for private URIs and confirm none map to public paths. |
| Image styles and derivatives missing after cron runs | Derivatives generated by cron or queue workers were written to local disk, or the worker runs in a container without the S3 credentials. Confirm the file system is set for the derivative paths and that cron containers share the same configuration and secrets as web containers. | Check the cron job output and the Drupal log for upload exceptions tagged to the S3 stream wrapper. Compare object counts under the styles prefix after a cron run. |
| Stale or outdated files served after an edit | A CDN, reverse proxy or browser cache keeps the old object because the URL did not change or purge was not triggered. Use versioned or hashed filenames where possible, and configure cache tag purging or CDN invalidation for changed files. | Compare response headers such as age and cache status from the CDN with the object's last-modified time in storage. Re-fetch after a purge to confirm the new object is served. |
| Files written under the wrong prefix or environment | Staging and production share a bucket, or the configured root path differs between containers after a deploy. Give each environment its own bucket or root prefix and set it through environment variables, not hard-coded settings. | List the bucket with a prefix query per environment. Check the rendered settings file in each running container to confirm the root path. |
| Intermittent upload timeouts on large files | Network path or endpoint limits on multipart uploads, or PHP memory and time limits in the upload handler. Confirm the endpoint's multipart settings with your storage team and review PHP limits for the web tier. | Look for timeouts in the web server and PHP-FPM logs during upload. Retry a large test file directly against the endpoint to separate network issues from Drupal. |
Questions teams ask before moving Drupal files to object storage
What drives the cost of moving Drupal files to object storage?
The cost is driven mostly by the number of file entities and the total volume of public and private files, the number of environments that must read the same storage, and how much custom code the site already has. Each component is paid for separately: the planning and configuration work, the module setup and testing, the migration of existing files and their database references, and the storage and transfer charges that recur after launch if you use Amazon S3, or the hardware and operations time if you self-host Ceph. Ask any quote to separate one-time work from recurring costs, to state the file count and volume it assumes, and to list which environments are covered. Storage bills and egress fees are set by your provider, so confirm them directly with that provider.
Can our internal team move Drupal files to S3 without outside help?
A team that already runs Drupal in production and has someone comfortable with Composer, Drush and cloud identity can usually configure the S3 File System module on a test copy. The harder parts are less visible: deciding which files stay private, rewriting or verifying stored file paths, controlling access to private files through Drupal rather than direct bucket URLs, and testing image styles, cron, backups and failover across every server. If those skills or the test time are not available internally, a fixed-scope engagement can cover the design, build and cutover while your team keeps ownership of the result. Either way, agree in writing who signs off on the production cutover.
How long does a Drupal file storage migration take?
Duration depends mainly on how many files you hold, how large they are, and which migration method you choose. Copying files with a sync tool, then pointing Drupal at the bucket, takes a different amount of time from migrating files through Drupal's own file entities, which also has to update database records and run in batches. Other factors include bandwidth between your current storage and the new bucket, how long you can run both systems in parallel, and how much verification you require before cutover. No duration is fixed here because the inputs vary from site to site. Ask for a timeline that lists the file count, the volume, the chosen method and the rollback window, and test the method on a copy of production first.
What do we need to provide before a file storage project starts?
Before work begins, the team should prepare a short inventory and a few access decisions. Typically that includes:
- A count and size of public and private files, and a list of any file fields or media types that hold large assets
- The current Drupal version, PHP version, hosting layout and the list of environments that share files
- A named owner for the bucket or Ceph cluster, with the account, network and key-management rules that apply to it
- Access to a non-production copy of the site with a recent database dump and file set
- Decisions on backup, retention and who may read private files
The quality of this input sets the accuracy of the quote. Gaps usually appear later as extra scope, so resolve them before the statement of work is signed.
Should we use Amazon S3 or self-host Ceph for Drupal files?
Amazon S3 suits teams that want a managed service and already run workloads in AWS, where the operating burden of storage sits with the provider. Self-hosted Ceph, which exposes an S3-compatible API through the Ceph Object Gateway, suits organizations that must keep data in their own data centers or air-gapped networks, or that want to run storage as a shared platform. Ceph is licensed under LGPL-2.1 or LGPL-3, and the project's current maintained releases include Tentacle and Squid, but running it well requires dedicated drives per OSD, network redundancy and capacity planning. Choose on data residency, staffing and the cost of operating a cluster, and check the current release lifecycle on the Ceph releases page before you commit.
How do Drupal and S3 File System module upgrades affect our storage setup?
Upgrades touch three layers: Drupal core, the contributed S3 File System module and the storage backend. Drupal core and contributed modules follow their own release schedules, and the module's configuration, stream wrapper and dependency versions must remain compatible with each core release. Plan a staging upgrade that reads and writes real files, confirms image derivatives regenerate, and checks private file downloads and cron. On the storage side, changes to Amazon S3 SDK behavior or to Ceph release versions need their own test. Bucket policies, credentials and lifecycle rules should be recorded as code so they survive a rebuild. Ask for an upgrade routine that names which checks run before each production change.
Does moving files to object storage affect security and compliance?
It can, because files move out of the web server and into a bucket or cluster that has its own access policy. Private files must stay behind Drupal's access checks, so bucket URLs should not be public for them, and credentials should be stored as secrets rather than in code. Encryption in transit and at rest, access logging, and retention settings should be decided up front. If your organization is preparing for an assessment, DeployOpen can prepare the systems, controls and evidence, but independent auditors issue any audit report or attestation. Confirm the specific control requirements with your assessor before design starts.
What operations look like after files move to object storage
Moving Drupal files to object storage does not remove work from your team. It moves the work to the bucket, the credentials and the storage cluster, and most of it recurs. Plan an owner for each item before cutover. The first is bucket lifecycle and backups. Decide which prefixes hold public files, which hold private files, and whether private files need versioning or a separate backup target. Object storage does not back up the Drupal database, so the file_managed records and the bucket contents must be restored to matching points in time. Confirm with your storage team how the S3 File System module's cache and metadata are rebuilt after a restore, and test that restore on a copy of the environment before you rely on it.
The second recurring item is credential rotation. The access key that Drupal uses to write to the bucket should be scoped to that bucket, stored in your secrets tooling rather than in settings.php, and rotated on a schedule your security team approves. Rotation touches every web node and cron runner at once, so rehearse it. Monitor storage errors separately from Drupal errors. Failed uploads, signature errors and throttling responses from the S3 endpoint often appear first as broken images or missing private downloads, which makes them easy to misdiagnose as a theme problem.
The third item is platform currency. For a self-hosted cluster, check the Ceph releases page on a regular schedule and plan upgrades against the active release list, which currently names Tentacle and Squid, each with an estimated end-of-life date. Confirm the end-of-life dates against that page when you plan, because they change. For Drupal, the core release schedule sets bugfix releases on the first Wednesday of each month and security releases on the third Wednesday. Build a standing window for those dates, test each update on staging with the storage module enabled, and track Drupal 10 end of life on December 9, 2026 as a planning milestone for the wider site.
How DeployOpen scopes and delivers this integration
DeployOpen plans and builds Drupal file storage on S3-compatible object storage inside your own cloud accounts, data centers or air-gapped networks. Work starts from a written scope that names the buckets, the public and private file paths, the bucket policies and the cutover approach before any configuration changes.
How we scope
- Inventory existing public and private file paths, image styles, and any code that writes to the filesystem directly.
- Confirm whether the target is Amazon S3 or a self-hosted S3-compatible service such as Ceph, and who operates it.
- Agree in writing on bucket layout, credential model, CDN or proxy role, and the cutover and rollback steps for existing files.
How we build
- Configure the S3 File System module for public and private streams, keeping private files behind Drupal access checks.
- Test uploads, image derivatives, cron cleanup and file deletion against a staging copy of the site and a representative file set.
- Run the file migration in rehearsals, then verify file counts, checksums and sample URLs before the production change.
How we hand over
- Document the settings, environment variables, bucket policies and secret locations so your team can rebuild the configuration.
- Provide a runbook covering credential rotation, adding a second site or server, and restoring files from object storage backups.
- Walk your operations and platform engineers through the deployment, including what to check after a Drupal or module update.
How we support
- Support is available as a retainer for module updates, security releases, and storage endpoint changes after launch.
- Confirm with your storage operator the backup, versioning and replication settings that apply to your bucket.
- Agree on response expectations and escalation paths in the written scope rather than assuming them.
