Why ERPNext background jobs get stuck and how to tell
When an import, repost or document submission seems to hang or finish hours late in a running ERPNext, the cause is usually in the background layer, not the page you are looking at. Frappe runs this work as jobs placed into named queues, and workers pick them up from those queues. Most stalls trace to one of four things: a job sent to the wrong queue, too few workers for the load on that queue, a job that exceeds its timeout, or a job that failed and is waiting to be read. The scheduler is a separate process that enqueues recurring work such as auto repeat and scheduled emails at set intervals, so a scheduler problem looks different from a worker problem even though both show up as "nothing happened."
A team needs this diagnosis when work that normally completes in minutes starts to queue up, when a bulk data load or stock repost sits at the same progress indicator for a long time, or when users begin resubmitting documents because they assume the first attempt was lost. Before changing any setting, confirm which site and bench you are looking at, which queues the job was sent to, and whether the work ran in the foreground request or in a worker. Ask the people who operate the bench to confirm the worker processes and their logs on that host, because the same symptom can come from a stopped worker, a full Redis connection, or a job that is still running and simply slow.
Treat a stuck job as a question with a short list of answers, not a reason to restart services. Restarting workers can discard in-flight work, leave a document half-processed, and hide the failed job record you need. Record the job name, its queue, when it was enqueued, and its last status before you touch anything. Those details tell you whether the fix is a queue assignment, a capacity change, a timeout adjustment, or a rerun of one failed job, and they make any later vendor or partner review far faster.
Which queue and worker settings fit which symptom
This table matches common background job symptoms in ERPNext to the queue, worker, timeout or scheduler setting that usually fits, and the change to test first. Read each row as a starting hypothesis to confirm in the worker logs and the Background Jobs view before changing any production setting.
| Symptom | Likely cause | Change to test first |
|---|---|---|
| Short queue jobs (emails, notifications, light hooks) wait for minutes | Short workers are busy or stuck behind a long job, or too few are running for the load | Confirm the short worker count and whether it runs only short jobs, then add capacity before moving any work |
| Default queue jobs fail intermittently with timeouts | Jobs run longer than the configured timeout for their queue, often during peak hours | Review the timeout for the default queue and the job's own work, then raise the timeout only after checking the job's query cost |
| Long queue jobs such as imports, reposts or bulk updates never finish | The job exceeds the long queue timeout, or it shares a worker that is restarted during deployments | Check the long queue timeout and worker restarts in the logs, then split the import into smaller batches as the Data Import guidance advises |
| Many jobs queue up in one queue while others sit idle | Workers are subscribed to the wrong queues or the queue names in the site configuration do not match the workers | Compare the queues each worker process listens to with the queues the jobs are enqueued on, and correct the mismatch first |
| Scheduled tasks such as auto repeat or backups run late or not at all | The scheduler is disabled or paused on the site, or scheduler jobs wait behind a busy default queue | Check scheduler status for each site before adding workers, since the scheduler enqueues work and does not process it |
| Worker memory or CPU climbs and the host swaps | Too many worker processes for the available memory, or a job that loads large datasets into memory | Lower the worker process count on the busiest queue and watch memory per process before increasing concurrency again |
| Jobs run correctly but the database becomes the bottleneck | Adding worker processes multiplies concurrent queries against MariaDB and Redis | Hold the worker count steady, profile the slowest jobs, and fix their queries before adding more parallel work |
Diagnosing stuck jobs, slow jobs and failed jobs
A job that sits in a queue and never begins usually means the worker that should pick it up is absent, busy with other work, or listening on a different queue than the one the job was sent to. Start by confirming that the background workers are running for the bench and that they are consuming the queue the job targets. The short, default and long queues are separate lists, so a job placed on the long queue will wait forever if only short-queue workers exist.
- Worker presence: Confirm the worker processes are up for each site's bench, using the process supervisor your deployment uses (for example, the supervisor status output on a bench install or the container status on a Docker install).
- Queue contents: Check the pending jobs for the site in the desk and note which queue each one sits in, then compare with the queues your workers are configured to consume.
- Worker counts: Look at how many workers exist per queue. If all of them are occupied by long jobs, short jobs queue behind them.
- Scheduler status: Recurring work that never enqueues often means the scheduler is not running, which is a separate fault from a stuck worker.
- Redis connectivity: Queued jobs live in Redis, so confirm the workers can reach the Redis service configured for the bench.
Record the queue name, the worker count per queue and the time the job was enqueued before changing any setting, so you can judge whether the backlog is growing or draining.
A job that runs much longer than its peers is either doing more work than expected, waiting on a lock or external call, or hitting a timeout that the worker then enforces. The worker logs and the job's own status are the two sources that separate these causes. Do not raise timeouts first. A longer timeout keeps a bad job alive longer and can hide a query that needs an index or a loop that needs batching.
- Worker logs: Read the worker error logs for the time window of the job. Repeated timeout messages or a traceback at the end of a run point to the stage that stalled.
- Job identity: Match the running job to its document or report in the desk, so you know whether it is a reposting, a bulk update or a report build.
- Database activity: Check for long-running queries or lock waits on the database while the job runs, because a blocked transaction often looks like a slow worker.
- Import size: The Data Import documentation advises importing only a few thousand records at a time, so a very large import in one pass is a common cause of long runs.
- Timeout settings: Confirm the worker and queue timeout values in the site and bench configuration before editing them, and change one value at a time.
Compare the run time against a known good run of the same job type on the same data size before deciding that it is slow.
A failed job leaves a traceback and usually an entry in the Error Log in the desk. The error text tells you whether the fault is in data, in a custom script or app, or in the environment. Fix the cause before retrying, because a retry of a job that fails on the same record will fail again and may repeat partial writes.
- Error Log: Open the Error Log in the desk and filter by the time of the failure. The traceback identifies the function and the document involved.
- Failed job entries: Review the failed jobs listed for the site, then compare the exception types across several failures to see whether they share one cause.
- Custom code: If the traceback passes through a Server Script or a custom app, check whether that code changed recently. Server Scripts are disabled by default from version 15, and the Frappe documentation requires the server_script_enabled setting to be turned on, so a script that depends on that setting may fail after an upgrade.
- Data problems: Missing link values, invalid dates or duplicate names in an import file produce validation errors. Check the source file rows named in the error.
- Worker logs: Confirm the same error in the worker log, since the desk log may not show every low-level failure.
Only retry after you have changed the cause, and record which jobs were retried so partial results can be reconciled.
How to work through a background job investigation
This sequence moves from the reported symptom to a verified fix, so the team changes one thing at a time and knows which job, queue and site the change affected.
Capture the symptom and the affected site
Record what stopped or slowed, which site and bench it belongs to, when it started, and whether the trigger was a user action, a scheduled event or a bulk import or repost. Ask the operator who reports the issue to name the document, report or job involved. Note the exact time window and whether other sites on the same bench are affected. The step is done when you have a written statement of the symptom, the site name, and a start time you can match against logs.
Confirm workers and scheduler are running
Check that the worker processes and the scheduler are alive on the host, using the supervisor or process manager that starts them in your deployment. In a Docker-based setup, check the worker and scheduler containers instead. Confirm that the worker and scheduler services are running the same ERPNext and Frappe versions as the bench, since a mismatch can stop jobs without an obvious error. The step is done when every expected worker and the scheduler report as running, and you have recorded any that have restarted recently.
Inspect the queue backlog by queue
Look at the pending and started jobs for each queue (short, default and long) for the affected site. A growing short queue points to blocked or slow jobs that hold the worker, while a long queue with a backlog usually means long-running imports or reposts are occupying their slots. Compare the count of jobs queued with the count of workers listening to each queue. The step is done when you can say which queue is backed up, how many jobs are waiting, and whether the backlog is growing, flat or shrinking.
Check the job timeout and worker logs
Identify the job at the head of the stuck queue, then find its configured timeout and the error or traceback it wrote to the worker logs and the Error Log in the desk. Look for timeout errors, database lock waits, memory exhaustion and repeated failures of the same job method. Check whether the job is being retried or has been marked as failed. The step is done when you have the job's method name, its timeout setting, and the most recent error or the confirmed absence of one.
Reproduce the problem on a copy of the site
Restore a recent backup of the site to a separate, non-production bench that matches the production versions and apps, then trigger the same job or import with the same input. Keep the test isolated from live email, integrations and scheduled jobs so that the copy does not send messages or post entries to other systems. Confirm the copy shows the same timeout, error or slowdown. The step is done when the failure reproduces on the copy, or you have recorded that it does not, which points to data volume, concurrency or another worker in production.
Apply the smallest change that addresses the cause
Make one change that matches what the evidence points to, such as raising the timeout for a known long import, splitting a large import into smaller files, restarting a stuck worker, or adjusting worker counts for the affected queue. Record the change, the reason, and the previous value so it can be reverted. Avoid bundling several configuration changes together. The step is done when the change is applied in the test environment first and then in production, with a written rollback note.
Verify the queue and document the result
Watch the queue after the change until the backlog clears or stabilizes, then rerun the original job on production with a small input where possible. Confirm that no new errors appear in the worker logs and the Error Log, and that the Scheduled Job Log shows the expected runs. Write a short record covering the symptom, the root cause, the change, and the verification. The step is done when the queue is clear, the job completes, and the record is saved where the operations team can find it.
Handling long imports and stock reposts without blocking the site
The fastest way to stop a long import or stock repost from blocking the site is to keep it off the short and default queues and to size each batch so that no single job holds a worker for hours. The ERPNext Data Import documentation advises importing only a few thousand records at a time, and that advice is the starting point. Each row in a CSV or Excel file creates or updates a document, runs its validations and may trigger child table logic, stock ledger entries or general ledger postings, so the cost per row is much higher than the row count suggests. Before you run a large file, confirm with your own test site how long a few hundred rows take, then plan the full file as a series of files that finish within a window your users accept.
Long-running work belongs on the long queue. Reposting stock and rebuilding valuation are the typical heavy jobs, and Frappe assigns work to the queue named in the code path that started it, so check which queue a repost lands on in your version before you assume it runs in the background at all. Check worker counts for the long queue with the process manager you use, and confirm the job timeout for that queue in your bench configuration. If a repost starts during business hours, schedule the largest reposts and imports for an off-peak window, and avoid running a bulk import at the same time as a month-end close or a stock closing entry, because both compete for the same stock and accounting tables. Splitting a file by date range or by warehouse also makes a failed batch easier to restart without repeating work.
This section covers only the batching and queue placement for large operations. Jobs that stop running on schedule, reports that time out in the browser and slow pages are covered on the sibling pages on scheduled jobs and slow reports, so use those when the symptom points there. A useful check before you approve a large import is to ask whether anyone has a written rollback plan, because a partial import or repost is harder to reverse than a single document. Record the file name, the batch number and the start and end time for each batch in your own change log so that a failed batch can be identified and rerun without duplicating records.
Security and operational risks when changing worker settings
Changing worker counts, queue assignments or timeouts touches more than throughput. The same settings decide what gets logged, which server features a site can use and how quickly a failed job is noticed. Review these four risks before you change anything on a live ERPNext site.
Worker logs can expose business data
Background job logs and error traces often include document names, customer identifiers, amounts and payload fragments from the job that failed. When you raise log verbosity to diagnose a stuck queue, that data lands in server log files that may be readable by more people than the ERPNext desk permits. Before you enable verbose logging, confirm who can read the log directory, how long files are kept and whether the worker logs are shipped to a central system. Restrict access to the same people who can see the underlying documents.
Server Scripts are off by default
From version 15, Server Scripts are disabled by default and must be enabled with the server_script_enabled setting. Fixes that depend on a Server Script, such as a document event that retries or reroutes a job, will silently do nothing until that setting is turned on. On Frappe Cloud public shared benches, Server Scripts are not allowed at all, so a plan that relies on them needs a dedicated bench or a different approach. Check the setting and the hosting type before you design the fix, not after the first failed test.
Changing settings without a backup
Adjusting worker counts, queue timeouts or scheduler state on a live site changes how the queued work is processed, and some jobs may run twice or be dropped when workers restart. A database backup taken before the change gives you a known state to return to if a long-running repost or import goes wrong. Confirm that the backup restores, record the current settings so you can revert them, and make changes during a window when no imports or reposts are in progress. Test the change on a copy of the site first where possible.
Alert gaps for failed jobs
A failed background job is recorded in the desk, but nothing pages anyone unless you have built a check for it. A queue can stop processing for hours while users see no visible error, which is most damaging for scheduled emails, auto repeat documents and backups. Define who reviews failed job records each day, set an alert on growing queue backlog or repeated failures, and confirm the alert reaches a person on call. Test the alert by forcing a failure on a non-production site so you know it fires before you need it.
Questions teams ask before hiring outside help
What drives the cost and effort of diagnosing ERPNext background jobs?
The main driver is how many sites, benches and worker types are involved and how much history you can give us. A single site on one bench with clear logs is a small diagnosis, while several sites sharing Redis queues, long reposts and an unclear hosting layout take more time. Costs come from the engineer's time for log review, reproduction in a non-production copy, and fix recommendations. Ask any provider for a quote that lists the sites in scope, the queues to review, whether a staging copy is included, and whether fixes are in the same fixed scope or a separate change.
How long does a background job diagnosis usually take?
Duration depends on scope rather than on a single fixed number. A diagnosis limited to one stuck queue on one site is quicker than one that covers worker counts, timeouts, failed jobs and long imports across several sites and environments. The biggest time sinks are reproducing a failure that only appears under load and waiting for a copy of production data. No duration figure is supplied for this work, so ask for a timeline in the written scope and confirm which steps depend on your access and your change windows.
What access and logs does the work need from our team?
At minimum we need read access to the bench and its logs, the site configuration for each site in scope, and the worker and scheduler logs covering the incident window. Useful inputs include the Error Log and Scheduled Job Log entries from the desk, the failed job records, and the deployment layout (bench, Docker Compose or other). Where the work touches a live site, ask for a named contact who can approve restarts and any change to worker counts. Confirm how credentials are shared and whether database access is limited to read-only queries.
Can our internal team diagnose stuck ERPNext workers without outside help?
Yes, many teams can, especially when they have someone comfortable with Linux services, Redis and the bench command line. The work is usually manageable when the failure is tied to one queue and the logs are clear. Outside help is more useful when failures repeat across sites, when the cause sits between the application, the database and the hosting layer, or when your team cannot test changes without disrupting production. Decide early which skills are missing internally and whether the fix or the knowledge transfer matters more.
What does a fixed-scope engagement cover for this topic?
A fixed-scope engagement sets out, in writing before work begins, which sites, queues and worker types are examined, what evidence is delivered, and what changes, if any, are made. For background jobs, a typical written scope lists the logs and configuration reviewed, the reproduction steps, the recommended worker and timeout settings, and the runbook for the team. Ongoing monitoring, upgrades and continuous support are separate retainers. Ask what happens if the root cause sits in a custom app or an integration that falls outside the agreed scope.
Does a diagnosis guarantee that our jobs will stop failing?
A diagnosis identifies likely causes, tests them where access allows, and recommends or applies fixes within the agreed scope. It cannot promise that every failure disappears, because some causes depend on your data volume, hosting capacity or third-party services that change over time. Ask for a clear statement of what is verified in staging, what is confirmed only in production, and what monitoring the team should keep to catch a recurrence.
Which ERPNext version and support status should we confirm before we start?
Confirm the exact ERPNext and Frappe Framework versions in production, because the diagnosis and the recommended settings depend on them. Frappe's supported versions page lists which versions are supported and when support ends, and the ERPNext releases page shows the minor releases published for each supported major version. If you run a version past its listed end of life, that changes both the risk and the options for a fix. Also confirm whether you hold a Frappe Cloud support plan, since Frappe's supported versions policy ties warranty coverage to a listed version and a matching plan.
What a fixed-scope background job review covers
A fixed-scope background job review starts from a written list of what will be examined and what will be handed over. For this topic, the scope usually covers the Redis-backed queues that ERPNext uses (short, default and long), the worker processes serving each queue, the scheduler and its job records, and the failed job and error history for each site on the bench. Before any work begins, the client and DeployOpen agree on which sites, which benches and which environments are in scope, whether the review includes a read-only pass on production or a copy of it, and who on the client side can grant access to the server, the database and the desk. The reviewer needs read access to worker and scheduler logs, the Error Log and Scheduled Job Log in the desk, and the site configuration, and the client should confirm these access paths before the review is priced.
The deliverables are the part a CIO or platform lead can hold the work to. A typical written output names each queue and worker configuration observed, the jobs and job types that failed or ran past their timeouts, the import or repost jobs that were running long, and a ranked set of changes with the reason for each. Each change should state what it touches (for example, workers or timeout settings in the bench configuration, or the code path that enqueues a long job), what it risks, and how to verify it after the change. Ask for the list of verification steps in writing, and ask whether any configuration changes will be applied by DeployOpen or only recommended for your team to apply.
A retainer covers what a one-time review cannot: watching queue depth, failed job counts and worker health after the fixes land, and reacting when a new import or a large scheduled report changes the pattern. Ask which signals the retainer monitors, how alerts are routed, who responds out of hours, and how upgrades to ERPNext versions that alter job behavior are handled. Confirm these terms against the written scope rather than assuming them, because the monitoring depth, response expectations and access model vary from one engagement to the next.
How DeployOpen scopes, delivers and supports ERPNext operations work
DeployOpen approaches background job and worker problems as a bounded piece of operations work on your own ERPNext infrastructure. We start from the queues, sites and workloads you run, agree a written scope, make changes in controlled steps, and leave your team with documented settings, checks and runbooks.
How we scope
- Written scope covers the affected sites, benches, queues and job types named in your incident or review.
- We collect worker counts, queue configuration, timeout settings and recent failed job records before proposing changes.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
How we build
- We reproduce stuck or slow jobs in a non-production copy of your environment before changing live settings.
- Each change to worker counts, timeouts or queue routing is tested, recorded and reversible.
- Long imports and reposts are split, sequenced and monitored so they do not block short queue work.
How we hand over
- You receive a written record of root causes, configuration changes and the evidence behind each one.
- Runbooks cover checking worker health, clearing stuck jobs safely and reading the Error Log and Scheduled Job Log.
- Your team walks through the handover and runs the checks once without DeployOpen to confirm they work.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after the initial work closes.
- Retainer scope, response expectations and review cadence are confirmed in writing, not assumed.
- Support covers your self-hosted infrastructure, and we confirm any dependency on ERPNext or Frappe version changes before acting on them.
