Drupal cron and queue workers at scale

Why Drupal cron and queue workers need a plan at scale

A single web node can run Drupal cron on a schedule and nobody thinks about it. Once a second or third web node serves the same site, the same cron job can fire on every node at once, so the scheduler, not Drupal, becomes the source of duplicate runs, overlapping imports and locked tables. The fix is a plan that decides which host runs cron, how long a run may take, and where heavy work waits. Drupal core cron, the Queue API and Ultimate Cron each cover a different part of that plan.

Core cron runs the hook_cron() implementations of enabled modules, and Drupal.org documents that it can be triggered by a web request or by a command line run through Drush. Core does not know that another node is already running it, so on a multi-server setup the trigger must be restricted to one place, such as a single scheduled job that calls Drush cron. The Queue API is a separate mechanism: modules add items to a named queue and a worker processes them later, which keeps slow work such as fetching remote data or regenerating derivatives out of the page request. Ultimate Cron is a contributed module that adds per-job schedules, logging and a way to stop a job from starting again while a previous run is still active. Confirm its current release and supported Drupal versions on its Drupal.org project page before you rely on any of its features.

A single host stops being enough when cron runs longer than its schedule interval, when queues grow faster than one worker drains them, or when editors notice saves and imports slowing while background tasks run on the web nodes. At that point the question is no longer how to call cron, but where each job is allowed to run and what happens if a run fails halfway. Ask your team to list every scheduled and queued task on the site, note its typical duration and whether it can safely run twice, and confirm which of them write to shared storage. That inventory is the starting point for the decisions covered on the rest of this page.

Which cron and queue approach fits your topology

This table compares four ways to run Drupal cron and queue work, from one host to several web nodes with heavy background jobs. Read each row against your server count and how much visibility your team needs into scheduled jobs. The main takeaway is that adding servers without a single scheduling owner is the most common source of duplicate or overlapping runs.

Approach	When it fits	What the team needs	Main trade-off
System cron on one host	A single web server or a small site where one machine runs all Drupal cron work	Shell access, a cron entry that calls Drush or the cron URL, and log rotation for the output	Simple and cheap, but it fails silently if the host is down and does not scale past one node
Dedicated scheduler or container job	Several web nodes or Kubernetes, where cron should run once per site regardless of pod count	A separate job runner (CronJob, scheduler service or CI-managed task), a concurrency policy and alerting on missed runs	Runs once by design, but the team must own job definitions, retries and alerts outside Drupal
Ultimate Cron module	Sites with many jobs that need different schedules, timeouts and per-job logs in the Drupal admin	Module configuration, a named owner for each job, and a plan for how the module is kept current on each Drupal upgrade	Good visibility and per-job control, but a poorly scheduled job can still overlap and the module adds code to maintain
Separate queue worker processes	Heavy tasks such as imports, image derivatives, search indexing or external API calls that should not run during page requests or cron	Supervised worker processes (for example systemd or container restarts), memory limits and a monitor on queue depth	Keeps cron fast and isolates load, but workers need recycling, and queue items can run twice if a worker stops mid-task

How to set up cron and queue workers across servers

This sequence moves Drupal's scheduled and heavy background work out of web requests and onto one controlled scheduler, so each job runs once, on time, and leaves a record that can be checked. It is written for teams running more than one web server who need predictable background processing.

Turn off cron on web requests
Start by stopping Drupal from running cron whenever a visitor loads a page. In the site settings, set the automated cron option to disabled so that no anonymous or logged-in request triggers background jobs. Confirm the change in the Drupal administration interface under the cron settings, then load several pages on each web server and check the logs to confirm that no cron run was triggered by traffic. This step is done when public requests no longer start cron and the scheduler you choose in the next step becomes the only source of runs.
Choose one scheduler for every server
Pick a single component to trigger cron, such as a system crontab on one designated host, a Kubernetes CronJob, or a scheduler in your container platform. Running the same job on every web node is the most common cause of duplicate runs, so the schedule should exist in one place. Ask your platform team to confirm which scheduler they operate and how it records failed runs. Document the chosen trigger, the host or namespace that owns it, and the command it executes. You know this step is complete when the schedule is written down and there is exactly one active definition for it.
Run cron with Drush from the scheduler
Configure the scheduler to call Drush inside the Drupal codebase, using the same PHP version and user account as the web application so file permissions match. A typical call uses the drush cron command, run from the Drupal root or with the site alias supplied. Exact syntax, flags and the handling of multisite installations depend on your Drush and Drupal versions, so confirm the command against the Drush documentation for your release before you deploy it. Send output to a log file or your log platform. The step closes when a manual run from the scheduler host completes without error and the log shows the expected start and end lines.
Move heavy work into queue workers
Identify tasks that take long enough to slow cron, such as imports, search indexing, image derivative generation, or mail sending, and move them into Drupal's queue system. Cron's job is to add items to a queue. Separate worker processes then consume those items, using drush queue:run with the named queue, or a long-running worker managed by your process supervisor. Start with one queue and one worker, watch the item counts and failures, and only then add workers. Confirm the worker command and any supervisor settings against your Drupal version. This step is complete when a queued item is created by cron, processed by a worker, and removed from the queue without error.
Protect against overlapping runs
Two cron runs that start before the first one finishes can process the same items twice or corrupt state. Drupal core uses a lock API to prevent this for cron itself, so confirm that the lock backend works across all web and worker containers. If you need per-job schedules, retries, or a visible run history, evaluate the contributed Ultimate Cron module, which adds job scheduling and lock behavior on top of core cron. Test overlap deliberately by starting a long job twice and checking that the second attempt is skipped. Where you use a shared cache or lock backend such as Redis, confirm that its availability is part of the design. The step is complete when a deliberate duplicate start is refused and logged.
Set per-job schedules and timeouts
Decide how often each job should run and how long it may take before it is considered stuck. Ultimate Cron lets you set separate schedules for individual jobs, so a fast cleanup task does not wait behind a slow indexing job. Record each job, its frequency, its expected duration range, and the owner who responds when it fails. Ask your team to agree on these values with the business owners of the content and data they affect. Confirm the configuration options available in the module version you install. This step is done when every job has a documented schedule and owner.
Verify each job with logs and last-run checks
Check every job after it runs, not only when a user reports a problem. Review the Drupal recent log messages and the Ultimate Cron run history where installed, and confirm that each job recorded a last successful run within its expected window. Set up an alert in your monitoring system for jobs that have not run on time or that report errors, and test the alert by pausing the scheduler for one cycle. Queue item counts should return to low levels after each run. The step is done when you can show a recent successful run for every job, and when the alert fires during the test and clears once cron resumes.
Hand over the runbook and the alerts
Write a short runbook that names the scheduler, the Drush commands, the queues and workers, the lock backend, the job schedules, and the alerts. Include the steps to pause a job, clear a stuck lock, and restart a worker, and state who may perform each action. Confirm the commands against your exact Drupal and Drush versions before publishing the runbook. Run one tabletop review with the operations team in which they follow the runbook for a failed job. The step closes when the team can diagnose and recover a failed cron run from the runbook alone, without relying on the original builders.

How the parts fit together during a cron run

A cron run on a multi-server Drupal site is a short chain of hand-offs, and most failures happen at one of the joins. The scheduler (system cron, a container job or a Kubernetes CronJob) fires an HTTP request at /cron/{key} or runs drush cron on one host. Drupal then checks whether it may start: the cron last-run timestamp and a lock in shared storage decide whether this invocation proceeds or exits. Every web or worker node that can receive the trigger must see the same answer, so the lock has to live where all of them can read it, which is the shared database by default and Redis or another shared store when you move locks there.

Once cron starts, Drupal runs the hook_cron implementations of core and contributed modules, and modules that need heavy work push items onto a queue rather than doing it inline. Queue storage is also shared state. With the database backend, items sit in a queue table that worker processes claim with a lease, so a worker that dies mid-item releases its claim after the lease expires. Workers consume these items through drush queue:run or a long-running process, and they do not need the web request that enqueued the work to still be open. Which queues exist, and how many items each holds, is the first thing to check in a new environment.

Before you tune anything, confirm three facts in your own environment: which host or container runs the trigger, whether cron is also reachable through the web front end (and if so, whether the key is protected), and where the lock and queue data are stored. Those answers decide whether a slow run blocks the next one, whether a queue backlog grows silently, and whether moving storage to Redis changes locking behavior. Cache backend configuration for Redis or Memcached is covered on the sibling pages, so this section only names where locks and queues sit in the request path.

Security and operational risks in background processing

Background work runs without a user watching it, so a weak control or a silent failure can go unnoticed for days. The cards below cover the access, credential, monitoring, locking and logging risks that tend to appear once cron and queue workers run across several servers.

Unauthenticated cron URL

Drupal's cron can be triggered over HTTP through a URL that carries a key. If that key is missing, guessable or shared in a ticket, anyone can start a run, which can hammer the database or re-trigger heavy jobs. Use a long random cron key, restrict the cron path at the web server to trusted sources where possible, and prefer system cron or a container job so the web entry point is not needed at all. Verify that an unauthenticated request to the cron path is refused.

Over-privileged worker credentials

Workers often reuse the web application's database user and API tokens, so a compromised job can read or change far more than it needs. Create a dedicated database account with rights limited to the tables the jobs touch, give each integration its own scoped API credential, and keep those secrets in the platform's secret store rather than in settings files. Verify by reviewing the grants on the worker account and checking that a failed job cannot reach unrelated systems.

Failing jobs with no alerts

A queue item that errors repeatedly can sit in the queue or silently stop a scheduled job, and editors may not notice until content is stale or an invoice never sends. Send job failures and growing queue backlogs to your monitoring system, record the last successful run time for critical jobs, and set an owner for each alert. Verify by forcing a controlled failure in staging and confirming that an alert reaches a person.

Stuck locks after crashes

When a worker dies mid-run, its lock can remain set, so later runs skip the job or two workers wait on each other. Set a lock lifetime that matches the longest expected run, use a lock backend that expires entries automatically, and confirm that a killed worker releases or expires its lock. Document how an operator clears a stale lock safely. Verify this by stopping a worker during a long job and checking that the next scheduled run proceeds.

Overlapping runs across servers

With several web nodes, each may start the same scheduled job, which can duplicate emails, imports or API calls. Pick one scheduler per job, such as a single cron host or a single container job, and make jobs idempotent so a repeated run does not double the effect. Verify by starting the job on two nodes at once in staging and confirming that only one run processes the data.

Sensitive fields in job logs

Queue payloads and job logs often capture entity values, email addresses, tokens or personal data when something fails. Log job IDs, outcomes and counts rather than full payloads, mask fields that hold credentials or personal data, and set a retention period on log storage that your data protection policy allows. Verify by triggering a failure with test personal data and checking what appears in the log sink and who can read it.

Cron patterns for VM, container and Kubernetes hosting

On traditional virtual machines, cron usually means the operating system's crontab running drush cron or a Drupal cron URL on a schedule. This works well for a single web node, but once a second web server exists, every node with a crontab entry will fire the same job. The usual fix is to run cron from exactly one designated host, or to use a wrapper that takes a lock before starting, so overlapping runs exit early instead of colliding.

Pick one scheduler host: Keep the crontab entry on a single node or a small utility host, and leave web nodes without cron entries.
Use file locks: Wrap commands with flock so a new run exits if the previous one still holds the lock.
Set a timeout: Use timeout on Drush commands so a hung process cannot block the next schedule indefinitely.
Scale workers as separate services: Run queue workers through drush queue:run under systemd units, with the number of units set per queue load.
Log per run: Send output to a dedicated log file so failed runs are visible to the operations team.

Before relying on this pattern, confirm with your hosting team how their scheduler, time synchronization and log rotation behave on your specific VM images.

On container platforms, cron is often handled by a scheduler outside the application container. The container image should contain the Drupal code, while a separate scheduled task starts a short-lived container that runs cron and then exits. Because the scheduler and the web service are separate, you control the schedule from one place, but you must still prevent two scheduled tasks from starting for the same site at the same time.

Run cron as a one-off task: Start a short-lived container that runs drush cron and exits, rather than running cron inside the web container.
Control concurrency at the scheduler: Set the scheduler's concurrency policy to forbid overlapping runs for the same job.
Separate worker containers: Run queue workers as their own service with a replica count that you change independently of web capacity.
Share a common database and cache: Ensure all containers point to the same database and, where used, the same Redis or Memcache backend, so locks and queue state are shared.
Set resource limits: Give cron containers memory and CPU limits so a heavy job cannot starve the web service.

Confirm with your container platform team how scheduled tasks are retried, how failed runs are reported, and whether the platform guarantees at-most-once or at-least-once execution.

On Kubernetes, the usual pattern is a CronJob for periodic cron work and a Deployment for long-running queue workers. A CronJob with concurrencyPolicy: Forbid prevents a new run from starting while the previous one is still active. Note that Kubernetes CronJobs can still create more than one Job in unusual failure conditions, so the Drupal side should still tolerate duplicate execution through locks or idempotent jobs.

Set concurrencyPolicy to Forbid: Use this on the CronJob to skip a run when the previous Job is still active.
Set deadlines: Use startingDeadlineSeconds and activeDeadlineSeconds so stale or stuck Jobs do not pile up.
Keep history limits sensible: Configure successfulJobsHistoryLimit and failedJobsHistoryLimit so failures stay visible without clutter.
Scale workers with a Deployment: Run queue workers as a Deployment, and use a Horizontal Pod Autoscaler where queue depth or CPU is a reliable signal.
Plan for drain and restart: Configure graceful termination so a worker finishes its current item before the pod stops.

Confirm with your Kubernetes platform team which cluster version, controller behavior and autoscaling metrics they support in your environment before setting these values.

Limits to confirm in your environment

Cron and queue design depends on facts about your Drupal version, PHP runtime, database and hosting platform. Confirm these before choosing a scheduling model, because each one can rule out an option or add work.

Drupal core and module versions

Ask which Drupal core release and which contributed versions of Ultimate Cron, Advanced Queue or any queue backend module you plan to use are installed or targeted. Check the module's supported core range on its Drupal.org project page. Confirm the upcoming core timeline, since Drupal 10 reaches end of life on December 9, 2026, and a cron setup built on a version you will soon replace needs a migration plan too.

PHP version support

Confirm the PHP runtime your release requires and whether your worker images or hosts run it. Drupal 10 requires PHP 8.1 or later, Drupal 11 requires PHP 8.3 or later and Drupal 12 requires PHP 8.5 or later. Ask whether the CLI PHP used by cron and Drush matches the web PHP, since mismatched versions are a common cause of silent job failures.

Database locking behavior

Ask which database engine you run (MariaDB, MySQL, PostgreSQL or SQLite) and how its locking behaves under concurrent cron and queue workers. Confirm whether your cron backend relies on database locks, how long they are held, and whether your database connection limits allow several workers at once. Test overlap on a staging copy before production.

Queue storage capacity

Ask where queue items live, how many items a heavy job can create in one run, and how much storage and I/O that uses. Confirm whether queue tables share the database with user traffic, and who owns the growth limit and the cleanup routine for completed or failed items. Without these answers, a backlog can slow the whole site.

Long-running worker permission

Ask whether your hosting provider allows persistent background processes, such as a Drush queue worker managed by a supervisor, and how long a single process may run before the platform kills it. Check whether managed platforms limit scheduled tasks or require their own job runner. A process that the host stops mid-job can leave partial work behind.

Scheduler and container behavior

Confirm whether your environment runs one cron trigger per site or several, and whether containers can run a scheduler alongside the web tier. Ask who maintains the trigger when a pod restarts or a host is replaced, and whether the trigger can fire on more than one node at the same time. This decides how much locking your jobs need.

Questions teams ask before running cron at scale

How much effort does a cron and queue setup take?

Effort depends mostly on how many servers run Drupal, how many jobs need their own schedule, and how many heavy tasks should move off the request cycle. A single-server site with core cron and a few light jobs is a small change. A multi-server estate needs a single trigger source, lock behavior that prevents overlapping runs, a queue worker deployment, and monitoring for failed or stalled jobs. Ask any partner to split the estimate into job inventory, configuration, worker deployment, testing under load and handover documentation, so you can see where the time goes.

Can our in-house team set up cron and queues without a partner?

Yes, if your team already runs Drupal in production and has someone comfortable with Drush, container or system scheduling and the Drupal queue API. The hard parts are less about the first configuration and more about proving that jobs do not overlap across servers, that failed items are retried or surfaced, and that the schedule survives deploys and node replacements. Teams without that experience usually benefit from a short review of their current jobs before they change anything, whether they build internally or with DeployOpen.

What drives the cost of this work up or down?

The main drivers are the number of Drupal sites and environments, the number of jobs and their frequency, whether you need per-job schedules through Ultimate Cron, how many custom or contributed modules add queue items, and the hosting model (virtual machines, Kubernetes or managed platforms). Costs also rise when logs are hard to reach or monitoring does not exist yet. A fixed-scope quote should list each job, each server role, each environment and each monitoring integration, and state which items are one-time and which need ongoing support.

How long does the work take for a typical scope?

Duration follows scope. A small site with one server and a handful of jobs can be configured and tested within a short engagement. A multi-server platform with several sites, custom queue workers and load testing takes longer because each change needs staging verification and a controlled rollout. DeployOpen agrees timelines in writing as part of a fixed-scope plan, so ask for the milestone list and the testing steps that gate each one rather than a single headline figure.

What must our team provide before the work starts?

Plan to provide access to the hosting environment with enough rights to read and change scheduling, worker and service configuration in staging and production. You will also need access to Drupal logs, web server and database logs, a current inventory of cron jobs and queue-producing modules, and a contact who can approve maintenance windows. Where Drupal runs in a container platform, confirm access to the cluster manifests or deployment pipeline. Ask your partner to list these items in the scope document so gaps surface before the first sprint.

When does a retainer make more sense than a one-time build?

A one-time build fits when the job inventory is stable and your team can operate the result. A retainer fits when jobs change often, when new modules add queue work, when Drupal core or contributed modules need regular updates, or when you want someone to review failed-job alerts and respond to incidents. DeployOpen offers ongoing support and upgrades as retainers, and the right choice depends on how often your schedule changes and how much operational load your team can carry. Compare the recurring hours against the cost of a missed or overlapping job.

How do we know overlapping cron runs are actually prevented?

Test it rather than assume it. Start by checking whether each job uses a lock, whether the scheduler runs on one designated node, and how long a job may run before the next trigger fires. Then simulate a long-running job on staging with two or more servers triggering it and confirm from logs that only one run proceeds. Ask for the test script, the results and the alerting rules as part of the deliverables.

Keeping cron healthy after launch

Cron health after launch is a recurring operational task, not a one-time setup. The first recurring check is last-run time for every scheduled job, because a job that silently stops is the most common failure on a multi-server Drupal site. Ultimate Cron records the start and finish of each run and shows when a job last completed, so review that report on a fixed schedule and alert when a job misses its expected window. Confirm with your monitoring team which signals they can pick up, for example a heartbeat from the scheduler or a check on the Ultimate Cron log table. Pair this with a periodic review of failed queue items, since items that repeatedly fail usually point to a bad dependency, an expired credential or a remote API change rather than a transient error.

Patch cadence is the second recurring workload. Drupal core bugfix releases arrive on the first Wednesday of each month and security releases on the third Wednesday, according to the Drupal core release schedule, so build a standing window that covers both. Ultimate Cron and any other contributed modules that run your schedules need the same treatment as core, and a security release for a contributed module can land in the same window as a core update. Test each update on staging against the same cron and queue configuration used in production, and confirm the expected run times of long jobs still fit between scheduled starts once the new code is in place.

The third recurring task is rehearsing a restart after a node failure. On a cluster, confirm which node currently owns scheduled work, what happens to jobs that were running when that node died, and whether a second node will take over without a manual step. Run this drill at least once per release cycle, record the time to recovery and the number of jobs that needed manual intervention, and feed the results back into your runbook. Ask your operations partner or internal team to document who is on call for scheduler failures and where the procedure lives, because a runbook that nobody has rehearsed will not hold up during an incident.

How DeployOpen works on this

DeployOpen plans, implements, hardens, documents and supports self-hosted Drupal inside your own infrastructure, including cron scheduling and queue workers for multi-server sites. Engagements start with a written, fixed scope, and ongoing support is available as a retainer.

How we scope

Inventory every cron job, queue and batch task, with its runtime, frequency and the modules that register it.
Define which servers may run cron, which run queue workers and how overlapping runs are prevented.
Agree in writing the Drupal version, hosting layout, monitoring targets and acceptance tests before build starts.

How we build

Configure system cron or container jobs with a single scheduler, locking and timeouts for each Drupal site.
Set up Ultimate Cron schedules and queue worker processes, with settings for concurrency and retry behavior.
Test long-running jobs under realistic content volumes and document failures, logs and alert thresholds.

How we hand over

Deliver runbooks covering schedules, worker commands, lock recovery and how to pause or drain queues safely.
Walk your operations team through monitoring dashboards, alerts and the steps for a stuck or failed job.
Provide configuration as code in your repositories, so changes go through your normal review and release process.

How we support

Review cron and queue health after Drupal core and contributed module updates, which follow the core release schedule.
Offer ongoing support, upgrades and monitoring review as a retainer, scoped separately from the build.
Confirm with your team the support hours, escalation path and change window before the retainer starts.