When Trino fault-tolerant execution is worth switching on
Standard execution is Trino's default, and it fails a query when a worker fails. Fault-tolerant execution is off by default and is enabled with a QUERY or TASK retry policy. The TASK policy is the one Trino's documentation recommends for large batch queries, because it retries individual tasks rather than restarting the whole query. A platform team running Trino for dashboards, ad hoc SQL and batch ETL on one cluster should not change the default for all of them at once. Start by listing which workloads run longer than a worker failure is likely to interrupt, and which are short queries where a rerun costs little. Confirm that mix from your own query history before deciding, because the answer depends on query duration, how often workers restart in your environment and how much rework a failed batch job causes downstream.
Standard execution fits interactive dashboards and short ad hoc queries, where a failed query is cheap to resubmit and fault-tolerant execution would add overhead without a clear return. QUERY retry suits teams that want automatic reruns of failed queries without adding storage, but it restarts the whole query, so it helps less as queries grow. The TASK policy fits long batch ETL on Trino, where losing hours of work to one lost worker is the real cost. Its extra setup is the exchange manager, which spools intermediate data to S3, Azure Blob Storage, Google Cloud Storage or HDFS. That spooling store is a new dependency to size, secure, monitor and clean up, and you should confirm its throughput and retention behavior in your environment before committing batch pipelines to it.
Before switching, run a representative batch query on a test cluster with the exchange manager configured, and compare runtime and failure behavior against standard execution. Ask the team who will own the spooling bucket, its lifecycle rules and its access policy, since those are the controls that keep the extra setup from becoming a long-term burden. If most of your workload is interactive, the honest answer may be to keep standard execution and apply the TASK policy only to a separate batch cluster or catalog. Document the decision and the test results so the next platform review can revisit it when the workload mix changes.
Trino fault-tolerant execution versus standard execution, factor by factor
This table compares standard execution with the QUERY and TASK retry policies of Trino's fault-tolerant execution across failure behavior, query length, spooling, configuration and operator workload. Read each row as a question to answer for your own queries: the TASK policy is the one that changes the storage and operating picture most.
| Factor | Standard execution | QUERY retry policy | TASK retry policy |
|---|---|---|---|
| Failure behavior | A query fails if a worker it depends on fails, and the client must resubmit it | The whole query is retried after a failure, from the start | Individual tasks are retried after a failure, so completed work can be reused |
| Query length fit | Suited to short and interactive queries where a rerun is cheap | Suited to queries that are long enough to hurt on failure but whose full rerun is acceptable | Documented as recommended for large batch queries that run for a long time |
| Spooling storage | No exchange spooling is required | Requires an exchange manager, so confirm the spooling target in the fault-tolerant execution docs | Requires an exchange manager spooling to S3, Azure Blob Storage, Google Cloud Storage or HDFS |
| Configuration | Default mode, with no fault-tolerant settings to enable | Off by default; enabled with the QUERY retry policy and an exchange manager | Off by default; enabled with the TASK retry policy and an exchange manager, which adds configuration to review |
| Workload types suited | Dashboards, ad hoc exploration and short queries where resubmission is acceptable | Mixed workloads where long queries should not be lost but spool storage is not yet justified | Batch ETL and large scheduled queries that run long enough that reusing finished tasks matters |
| What changes for the operator | Minimal extra setup, but users must handle failed queries and retries | Adds an exchange manager and its storage to operate and monitor | Adds an exchange manager, object storage or HDFS access, and retention and cleanup of spooled data to operate |
| Cost components to confirm | Cluster compute only, from your own infrastructure | Cluster compute plus spool storage and its I/O, which recur | Cluster compute plus spool storage and its I/O, which recur and grow with query size |
| Questions to ask before enabling | Are failed queries rare enough that resubmission is acceptable? | Which queries are long enough to justify a full rerun protection? | Can your team operate the chosen spool storage, and do your batch queries fit the TASK policy's requirements? |
How standard execution handles a worker failure today
In standard execution, which is the default, Trino does not retry work that was running on a worker when that worker disappears. The coordinator plans the query, splits it into stages and tasks, and streams intermediate results between workers as the query runs. If a worker holding tasks for an in-flight query is lost, that query fails, and the work already done on every other node for that query is discarded. Nothing is checkpointed to durable storage in this mode, so the cluster has no partial result to resume from. The practical effect is that a failure ends the query, and the user or the calling tool must submit it again from the beginning.
The loss is proportional to how long the query has been running. A dashboard query that scans a few partitions and returns in seconds has little invested at risk, and rerunning it is cheap and often invisible to the user. A long batch query that joins large Iceberg or Hive tables, aggregates for an hour or more, and writes a table at the end can lose most of its compute when a single worker fails near the finish line. The longer the query and the more workers it spans, the higher the chance that some worker fails before it completes. This is why batch ETL is the workload where the failure behavior matters most, and why the decision to change it should start with a list of your longest-running queries rather than with a general preference for resilience.
Your deployment topology sets the context for this behavior. Trino's deployment model is one coordinator plus workers, and the Trino deployment documentation recommends a dedicated coordinator for larger clusters. Before deciding anything, confirm your worker count, how workers are provisioned (for example, on Kubernetes with Helm or on virtual machines), and whether autoscaling or spot capacity can remove workers mid-query. Also check your own historical logs for queries that failed due to worker loss, and measure their typical runtimes. Those numbers, not a generic recovery estimate, should drive whether the extra setup for fault-tolerant execution is justified.
Choosing QUERY or TASK retry policy by workload
Dashboards and ad hoc SQL usually run in seconds or a few minutes, and analysts can simply rerun a query that fails. For this workload, standard execution is often the simpler fit, because fault-tolerant execution adds coordination and storage that a short query rarely repays. If a team still wants retries here, the QUERY policy restarts the whole query on another set of workers, so the rerun cost is the full query and no exchange storage is needed beyond what the cluster already uses.
- What gets rerun: The entire query, from the beginning, after any worker failure.
- Extra storage: The QUERY policy does not require an exchange manager, so no spooling bucket or HDFS path is needed for this policy alone.
- Latency sensitivity: Dashboard users notice a restart more than a batch owner does, so confirm the acceptable wait with the people who use the dashboards.
- Concurrency: Many small queries on a shared cluster rarely benefit from retries individually, so measure failure frequency before enabling anything.
- Confirm: Ask whether your engineers can see failed queries clearly and whether analysts already rerun them without escalation.
Before enabling any retry policy for this group, confirm how often queries fail in your environment and whether the restart cost matters to the people waiting on results.
Medium queries run long enough that a failure is annoying but short enough that restarting the whole query is tolerable. This is the group where the QUERY policy is most often a reasonable middle ground. Teams weigh the occasional full rerun against the operating work of adding an exchange manager and its storage. If failures are rare and the rerun is cheap, the extra setup of the TASK policy may not be justified for this group.
- What gets rerun: The whole query, so the wasted work grows with how far the query had progressed.
- Extra storage: The QUERY policy avoids a spooling store, which keeps the setup closer to standard execution.
- Failure rate: Check how often these queries fail on your workers and whether those failures come from node loss or from query errors that retries will not fix.
- Scheduling: If these queries feed scheduled reports, a restart may still miss a deadline, even though the query eventually succeeds.
- Confirm: Ask your platform team for failure history over a representative period and whether the rerun wait is acceptable to the report owners.
If the failures in this group are mostly node loss and the restarts cause missed reports, the next step is to test the TASK policy against a sample of these queries.
Large batch queries, such as multi-hour ETL or table rebuilds over lakehouse tables, are where the documentation recommends the TASK policy. With TASK, a failed task can be retried on another worker using intermediate results that were spooled to external storage, so the work already completed is not lost. The trade-off is that the exchange manager must write intermediate data to S3, Azure Blob Storage, Google Cloud Storage or HDFS, which adds storage cost, network traffic and an extra system to operate and secure.
- What gets rerun: Only the failed tasks, with their inputs read from the spooled exchange data.
- Extra storage: An exchange manager spooling to S3, Azure Blob Storage, Google Cloud Storage or HDFS, sized for the largest intermediate results you expect.
- Access control: The spool location holds intermediate query data, so confirm its encryption, retention and who can read it.
- Lifecycle: Confirm how spooled data is cleaned up after queries finish or fail, and who owns that process.
- Confirm: Verify the storage option you can use under your cloud or data center policies and test a representative batch job before rolling it out.
Before committing to TASK, confirm the spool storage, its retention and its cost with your cloud or infrastructure team, and measure how much rerun time it would have saved on your real batch jobs.
What the exchange manager needs from your storage
The TASK retry policy writes intermediate exchange data to external storage, so the exchange manager becomes a dependency you must provision and operate. The four storage options below cover the choices Trino's fault-tolerant execution documentation names, followed by the checks to complete with your storage owners before you enable it.
Amazon S3 as the spool
Point the exchange manager at an S3 bucket dedicated to Trino spooling. The team must create the bucket, grant the Trino nodes credentials that can write, read and delete objects in it, and confirm that the network path from every coordinator and worker to the S3 endpoint is open, including any VPC endpoint or proxy rules. Ask whether lifecycle rules will remove spooled objects after a query ends, because leftover data accumulates quietly. Confirm the bucket and any encryption settings with the cloud account owner before you enable TASK.
Azure Blob Storage container
Use an Azure Blob Storage container as the spool location when your lakehouse runs in Azure. The team must create the container, decide how Trino authenticates to the storage account, and confirm that worker and coordinator subnets can reach the account through private endpoints or firewall rules. Check that the identity used has write and delete rights limited to the spool container. Agree with the Azure owners how long spooled blobs may persist, since cleanup depends on how the policy is set up.
Google Cloud Storage bucket
Provide a Google Cloud Storage bucket for the exchange manager when your Trino cluster runs on Google Cloud. The team must create the bucket, assign a service identity with object create and delete permissions scoped to that bucket, and confirm that the cluster can reach the storage endpoints under your VPC controls. Ask the platform team whether bucket-level retention or lifecycle rules will clean up spooled objects and whether any object versioning will multiply storage use. Verify these settings in your own project before enabling TASK.
HDFS path for on-premises clusters
On-premises clusters can spool to HDFS instead of object storage. The team must create a dedicated directory, grant the Trino service account read and write access, and make sure every node can reach the NameNode and DataNodes across any firewall segments. Confirm with the Hadoop administrators how Kerberos or other authentication applies to the Trino identity and whether quotas on the directory could block spooling during large batch runs. Plan for cleanup of exchange files after queries complete.
Storage owner sign-off before TASK
Before you set the TASK retry policy, meet with the storage, cloud and security owners and agree on the spool location, the identity Trino uses, the network paths, the cleanup mechanism and who is paged when spooled data fills a quota or a write fails. Ask them to confirm encryption, access logging and data residency rules for intermediate query data, since spooled output can contain sensitive rows. Record each decision in your runbook so the fault-tolerant setup can be audited and rolled back.
Operating effort for each mode: upgrades, spool storage, monitoring and skills
This table compares the recurring work of running Trino in standard mode against running it with fault-tolerant execution, covering upgrades, spool storage, configuration, failure testing, monitoring and skills. Read each row as the question your platform team will have to answer after go-live, not only at setup. The main takeaway is that fault-tolerant execution adds a storage and monitoring workload that standard mode does not carry, so confirm capacity for that workload before enabling it.
| Area | Standard execution | Fault-tolerant execution (QUERY or TASK) |
|---|---|---|
| Trino release upkeep | Track the current release line (release 484, October 15, 2026, per the Trino release notes) and test each upgrade against your own queries and connectors. | Same upgrade cycle, plus retest the retry policy and exchange manager after each release, since both are part of the cluster's behavior under failure. |
| Java runtime | Run a 64-bit Java 25 runtime, minimum 25.0.1, per Trino's deployment documentation. Confirm the JDK build and patch policy with your platform team. | Same Java 25 requirement (minimum 25.0.1). Confirm that the JVM settings used on workers are the same across all nodes before enabling retries. |
| Spool storage | Not applicable. No exchange manager or intermediate spool location is needed. | Required for the TASK policy: an exchange manager spooling to S3, Azure Blob Storage, Google Cloud Storage or HDFS. Confirm the bucket or path, quota and lifecycle rules with the storage team. |
| Spool cleanup | Not applicable. | Confirm how spooled data is removed after queries finish or fail, and whether lifecycle rules on object storage will catch leftover files from failed queries. |
| Configuration consistency | Keep coordinator and worker settings aligned across releases. Drift is usually visible as a failed query, not a hidden one. | Keep the retry policy, exchange manager and related settings identical across coordinator and workers. Changes often need a rolling restart, so schedule them with query owners. |
| Failure testing | Test node loss and query cancellation as part of normal operations checks, since a failed worker fails the query. | Test forced worker loss during large batch queries to confirm retries complete and that spooled data is cleaned up. Ask your platform team for a written test plan before go-live. |
| Monitoring of spooled data | Monitor query failures, queue times and worker health through standard Trino and JVM metrics. | Add monitoring for spool size, growth, read and write errors and retry counts. Confirm which metrics your monitoring stack can collect from Trino and the storage layer. |
| Trino skills | Staff need working knowledge of coordinator and worker roles, connectors and access control (file-based, Apache Ranger or Open Policy Agent, per Trino's security documentation). | Same skills, plus experience with retry policies, exchange manager configuration and the object storage APIs used for spooling. |
| Java and JVM skills | Staff need JVM tuning, heap sizing and log analysis for coordinator and worker nodes. | Same JVM skills, plus diagnosing retried tasks and spool read or write failures across nodes. |
| Who must confirm | Platform team: release plan, Java patch policy, monitoring coverage. | Platform team with the storage and security teams: spool location, encryption, retention, access to spooled data and failure test results. |
What does enabling fault-tolerant execution cost and take to run?
Does Trino's TASK retry policy need extra object storage?
Yes. The TASK retry policy needs an exchange manager that spools intermediate data to S3, Azure Blob Storage, Google Cloud Storage or HDFS, so you must provision and secure one of those before enabling it. The QUERY policy restarts the whole query on failure and does not use this spooling layer. Storage cost depends on how much intermediate data your largest queries produce, how long spooled data is kept, and whether the bucket or cluster sits in the same region as your workers. Ask your team to measure intermediate data volume on representative batch jobs before sizing the storage tier, and confirm the retention behavior in the Trino fault-tolerant execution documentation.
How does operating effort compare between standard mode and fault-tolerant mode?
Fault-tolerant execution adds components to run, so the effort is higher than standard mode. Standard mode fails a query when a worker fails, which means the operator's main job is rerunning queries. With fault tolerance enabled, the team also owns the exchange manager, its storage credentials, retention settings, capacity planning and the failure modes of that storage path. Those added duties recur every time you upgrade Trino or change worker sizing. Teams that run only short interactive queries may find the extra moving parts hard to justify, while teams running long batch ETL usually weigh the reruns they avoid against the new storage to operate.
Can our own engineers enable fault-tolerant execution without outside help?
Often yes, if your platform team already runs Trino, controls the object storage account and has tested worker failure on a non-production cluster. The configuration change itself is small, but validating it is not. The team needs to confirm that the chosen retry policy works with its connectors and queries, that spooled data is cleaned up, and that monitoring alerts on exchange storage. Teams without that experience usually get more value from a short review of their design before changing production settings. DeployOpen plans, implements and documents self-hosted platforms in your own infrastructure, so the choice between doing it internally and bringing in help is yours to make from the scope you define.
What should a quote for fault-tolerant execution include?
A useful quote lists the work as written deliverables tied to your environment, since DeployOpen agrees scope, deliverables and timelines in writing before work starts. Ask for:
- The current Trino version, deployment method and the connectors and catalogs in use
- The chosen retry policy and the reason for it, with the workloads it targets
- The exchange manager target (S3, Azure Blob Storage, Google Cloud Storage or HDFS), its access model and retention settings
- Testing of worker failure on a non-production cluster, with the acceptance criteria written down
- Documentation, runbooks and handover, plus whether ongoing upgrades or support are included as a separate retainer
DeployOpen does not publish a price list, so a quote follows from that written scope.
How long does enabling fault-tolerant execution take?
The duration depends on scope, not on a fixed figure. The main drivers are how many clusters and catalogs are involved, how much of the existing batch workload needs testing, whether the object storage target already exists and is approved for this use, and how many review cycles your change process requires. A single cluster with one storage target and clear test cases moves much faster than a multi-region estate with several catalogs and strict change control. Ask for a timeline in the written scope, and ask which of the open items would need your input before the work can start.
What does keeping fault-tolerant Trino up to date require?
Ongoing work recurs with each Trino release. Trino publishes new numbered releases on a regular schedule, and the most recent ones (for example release 484, published on October 15, 2026) can change configuration or behavior, so the team needs a repeatable test pass before each upgrade. Plan for regular testing of the TASK path, checks on exchange storage capacity and cost, review of spooled data cleanup, and documentation updates. Whether you carry that work in-house or under a retainer is a budgeting question. Ask any provider to state which upgrade tasks, test runs and monitoring reviews are included and how often they happen.
Does fault tolerance replace good query design in Trino?
No. Fault-tolerant execution reduces the cost of a failed worker during a long query, but it does not make an inefficient query cheaper. A query that scans far more data than it needs still consumes compute and storage, and spooling intermediate results to object storage adds I/O that poorly filtered joins make worse. Good practice still includes partition pruning on Iceberg or Hive tables, selective predicates, sensible join ordering and right-sized workers. Treat fault tolerance as protection for long batch jobs, and review query plans before enabling it so you do not pay to spool data you never needed to produce.
How to roll out fault-tolerant execution on a batch workload
This sequence moves a long batch workload from Trino's standard execution to fault-tolerant execution in controlled stages, so the team ends with a tested retry configuration, a known spool storage footprint and monitoring that shows whether retries are doing their job.
Identify the long batch queries
The data engineering lead and a Trino administrator review query history and pick the pipelines that run long enough, and touch enough data, that a single worker failure would force a full restart under standard execution. Short interactive dashboard queries usually stay on standard execution. The step is done when the team has a short written list of candidate pipelines with their owners, runtimes and failure history, and has agreed which one will move first.
Choose the exchange manager storage backend
The platform team confirms which storage the cluster can reach and is permitted to use for intermediate data. Trino's documentation names S3, Azure Blob Storage, Google Cloud Storage and HDFS as supported spooling targets for the TASK retry policy. The team should check bucket or container access, encryption requirements, lifecycle rules and network paths from every worker, and record the choice with its owner. The step is done when the storage target is approved and a test write from a worker succeeds.
Configure the exchange manager and retry policy
A Trino administrator enables fault-tolerant execution in a non-production cluster, sets the retry policy and points the exchange manager at the chosen storage. Trino's fault-tolerant execution is off by default, and the TASK policy is the one the documentation recommends for large batch queries. Confirm the exact property names, required values and any coordinator or worker restart steps against the fault-tolerant execution page for the Trino release in use. The step is done when the configuration is version-controlled and the cluster starts cleanly with it.
Test failure cases in a non-production cluster
The QA or platform team runs the candidate pipeline against a staging copy of the data and deliberately stops workers during execution, then checks whether queries finish after retry or fail. They also compare results and runtimes against a standard-execution run of the same query to confirm the output matches. Record which failure modes were tested, which were not, and which queries still fail. The step is done when the team has a written test log and has accepted the remaining risks.
Move one batch pipeline first
The pipeline owner switches the agreed pipeline to the tested configuration in production, typically during a scheduled window with a rollback plan that returns it to standard execution. Keep the rest of the cluster unchanged so any new failure can be traced to one workload. The step is done when several scheduled runs complete on schedule, results reconcile with the prior baseline and the team has signed off in writing.
Review spool storage and monitoring
The platform team checks how much intermediate data the spool location accumulates per run, how cleanup behaves after success and failure, and what the storage bill looks like on the provider side. They also add dashboards and alerts for failed or retried queries, spool growth and storage errors. Ask the vendor or the Trino documentation for the current guidance on cleanup and capacity, since these details change between releases. The step is done when spool growth and retry rates have a named owner, an alert and a documented review cadence.
When batch ETL on Trino justifies the extra setup
Batch ETL that reads through Trino connectors and runs for a long time has the most to gain from TASK retries. A job that scans Iceberg, Delta Lake or Hive tables, joins and writes results over hours loses the most work when a worker fails in standard mode, because the whole query must restart. Short interactive queries rarely justify the change: they finish quickly, a failed run is cheap to resubmit, and the spooling setup adds moving parts to a path that dashboards and analysts depend on. Before enabling anything, check the job durations in your query history and count how many of the long-running queries failed for infrastructure reasons rather than SQL errors.
Trino does not store the data it reads. It queries tables in place through connectors, so a Trino lakehouse still needs separate object storage and a table catalog such as an Iceberg REST catalog or Hive Metastore. Many organizations already run these for Spark, Flink or other engines, and if so, the catalog and bucket costs are shared rather than new. The TASK policy still needs its own exchange manager location, typically a dedicated bucket or container with lifecycle rules and access grants, so confirm with your platform team who owns that storage and how spooled data is cleaned up. Check the Trino fault-tolerant execution documentation against your connector versions before you commit.
The case is decided by a few questions. Do your batch jobs run long enough that a restart is expensive, and do they fail often enough to matter? Does your storage team accept a spooling bucket with its own retention and encryption controls? Are the connectors you rely on compatible with fault-tolerant execution, and have you tested them with a realistic data volume? Who will tune the exchange and retry settings, and who will watch spool growth? If the answers are mostly yes, pilot TASK on one nightly job and compare completion rates and runtimes. If most of your workload is interactive, keep standard execution and use scheduling and resource groups instead.
How DeployOpen scopes and delivers Trino execution changes
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, and that includes changes to Trino's execution mode. We start from your query workload and failure history, then agree in writing what changes, what gets tested and what your team receives before any work begins.
How we scope
- We review which workloads are interactive dashboards and which are long batch queries, since only the batch side usually needs retries.
- The written scope names the retry policy under consideration, either QUERY or TASK, and the reasons for choosing it or staying on standard execution.
- We confirm the exchange manager target (S3, Azure Blob Storage, Google Cloud Storage or HDFS) and who owns that storage, retention and access before work starts.
How we build
- We configure fault-tolerant execution and the exchange manager in a non-production cluster that mirrors your coordinator and worker layout.
- We rerun representative batch queries with induced worker failures to confirm that retried tasks complete and results match the baseline.
- We check the catalogs, connectors and access control paths the queries depend on, so retries do not expose data through a different route.
How we hand over
- You receive configuration files, a record of the test queries and failure scenarios, and the results from each test run.
- The documentation covers how to switch between standard and fault-tolerant modes, how to size and clean up exchange storage, and how to roll back.
- We walk your platform and data engineering staff through operating the cluster, including the signals that show spooling is filling up.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, so Trino version changes are retested against your query set.
- Upgrades to later Trino releases are planned as separate written scopes, with release notes reviewed for changes to retry behavior.
- Confirm with us during scoping which parts of the exchange storage and cluster operations sit with your team and which sit inside the retainer.
