How Apache Airflow runs Trino SQL in your pipelines
The supported route for a team that schedules pipelines in Airflow 3 and needs them to query a shared Trino cluster is the Trino provider, used together with SQLExecuteQueryOperator from the common SQL provider. The Trino provider brings in the Trino client and the common SQL package, so the operator can open a connection to Trino, send SQL statements in task order, and pass results to downstream tasks. Airflow decides when a statement runs, in what order, and what happens on failure. The work itself happens inside the cluster. Trino does not hold the data it queries. It reads through connectors to systems such as Hive, Iceberg, PostgreSQL or Snowflake, and the data stays where it already lives. For a lakehouse, that means separate object storage and a table catalog such as an Iceberg REST catalog or Hive Metastore must exist before any DAG can produce useful results. Airflow therefore orchestrates SQL and does not move rows through its workers. Keeping heavy joins and aggregations inside Trino keeps Airflow workers small and makes the cluster the place where capacity planning matters. Before writing DAGs, confirm which connectors the shared cluster exposes, which catalogs each team may read and write, and whether fault-tolerant execution is enabled for long batch queries. Those answers determine how the Airflow connection is scoped, how retries behave, and which pipelines can safely share one coordinator. The Trino provider package page is the next read for the provider's installation requirements and package extras.
What you need before connecting Airflow to Trino
Connecting Airflow to Trino depends on three sides agreeing: the Airflow 3 control plane, the Trino provider packages on your workers, and a Trino cluster your workers can reach with an account that is limited to the catalogs the pipelines need. Confirm each item below against your own environment before you write the first connection.
Airflow 3 and its Python and database baseline
Start from Airflow 3, not Airflow 2. Airflow 2 is end of life, so it gets no fixes. The Airflow 3.3.2 prerequisites list Python 3.10 to 3.14 and PostgreSQL 14 to 18 or MySQL 8.0, 8.4 and Innovation releases for the metadata database. MariaDB is not supported, and SQLite should not be used in production. Confirm the exact versions your platform team has approved.
Trino provider and its minimum Airflow version
The Trino provider, release 6.7.1, requires Airflow 2.11.0 or later, so a Trino integration on Airflow 3 pulls in the provider version your Airflow release supports. The package depends on the trino client and on the common SQL provider, which supplies SQLExecuteQueryOperator. Pin provider versions with your Airflow constraints file and confirm the install resolves cleanly on every worker image.
Java 25 on the Trino cluster
Trino's deployment documentation requires a 64-bit Linux system and a 64-bit Java 25 runtime, at minimum 25.0.1. Ask the Trino owners to confirm the runtime on every coordinator and worker, since a version mismatch surfaces as a cluster startup failure rather than an Airflow error. Confirm whether a dedicated coordinator is planned, which the docs recommend for larger clusters.
Network paths from workers to the coordinator
Every Airflow worker that runs a Trino task must reach the Trino coordinator on the HTTPS port your cluster exposes. In CeleryExecutor or KubernetesExecutor setups, that means the worker subnets or pod network, not only the scheduler. Check firewall rules, DNS names, and any proxy between the two. Also confirm the path from the coordinator to any object storage used by GCS or other transfers.
Service account with least-privilege catalog access
Create a dedicated Trino service account for Airflow rather than reusing a personal or administrator login. Grant access only to the catalogs, schemas and tables each DAG family uses. Trino's authentication options include password file, LDAP, OAuth 2.0, Kerberos, certificates and JWT, and its access control can be file-based, Apache Ranger or Open Policy Agent. Confirm with the Trino owners which mechanism they run.
Confirm the release line before you build
Trino publishes releases often, and the most recent release in the supplied notes is 484, published on October 15, 2026. Record which Trino version your cluster runs, because the Airflow client and the cluster should be tested together. Ask the platform team to confirm the Trino version, the Airflow patch, and the provider pin in a staging environment before any production DAG depends on them.
Configuring the Trino provider and first SQL task
This sequence takes a self-hosted Airflow environment from a clean install of the Trino provider to a verified SQL task that runs against a shared Trino cluster. Each step closes with a check so the team knows the connection and the task behave before any production pipeline depends on them.
Confirm the Airflow and Python baseline
Before installing anything, confirm that the Airflow environment is on a supported version and Python release. The Trino provider in its 6.7.1 release requires Airflow 2.11.0 or later, and Airflow 3 is the target for new work, so the team should check its Airflow version against the version life cycle and the prerequisites page. Confirm the Python version falls within the range listed for the Airflow release in use. The check is a written record of the Airflow version, Python version and provider release that the team will install against, agreed by the platform owner.
Install the Trino provider with constraints
Install the Trino provider package together with the Airflow constraints file that matches the running Airflow version, so the provider's dependencies resolve to versions tested against that release. The provider pulls in the trino client and the common SQL provider, so the install should cover both in one resolved set. Platform engineers usually run this in the image build for the workers and schedulers, not on a live host. The check is a clean pip check on the built image and an import of the Trino hook in the Airflow environment without errors.
Create the Trino connection in a secrets backend
Create an Airflow connection of the Trino type that points at the coordinator host, port, catalog and schema, and supply the user and any authentication material. Decide first whether the connection lives in the metadata database or in a secrets backend such as Vault or a cloud secrets manager, because that decision drives who can read the credentials. For a shared Trino cluster, a secrets backend is usually the safer default. Confirm with the Trino administrator which authentication method the cluster enforces, since the security documentation lists several options. The check is that the connection test or a simple lookup returns the expected host and catalog without exposing the password in logs.
Write a DAG with a SELECT 1 test task
Write a small DAG that uses SQLExecuteQueryOperator with the Trino connection ID and a trivial query such as SELECT 1. Leave autocommit at its default, which is off, and do not set it unless a later step requires it. Keep the DAG in a test folder or a staging deployment so it does not run against production schedules. The DAG should have no schedule, so it runs only when triggered manually during verification. The check is that the DAG parses without import errors in the Airflow UI and appears with the expected task.
Run the test query and read the task log
Trigger the DAG manually and let it finish. Open the task log and confirm that the connection resolved, the query was sent to the Trino coordinator, and the task reached a success state. The log should show the statement that ran and the returned row. If the task fails, check the authentication method, network path from the worker to the coordinator, and catalog name before changing the DAG. The check is a successful run with a visible log line that shows the SELECT 1 result.
Enable statement splitting and last-result return
Where a task sends several statements in one call, enable the split statements option so each statement runs in order, and enable the option that returns only the last result when downstream tasks need only the final output. Use these options only for the tasks that need them, since returning every result to XCom can make the metadata database grow quickly. Test a two-statement query in staging to confirm both statements run and the returned value matches the final SELECT. The check is a task log that shows each statement executed in sequence and an XCom value that contains only the expected last result.
Record the configuration for review
Before the first real pipeline uses the connection, record the provider version, Airflow version, connection storage location, authentication method, catalog and the autocommit and split settings in the team's runbook. Ask the Trino administrator and the security owner to confirm the credential scope and the network path. Keep the test DAG as a regression check that runs after each Airflow or provider upgrade. The check is a reviewed runbook entry and a passing manual run of the test DAG after the record is approved.
When GCS to Trino transfers fit a pipeline
A GCS to Trino transfer fits when files already land in Google Cloud Storage and the business needs them as Trino-queryable tables on a schedule. The Trino provider documents a GCS to Trino transfer operator, and the google extra of the provider is the piece that installs the Google dependencies that operator needs, so the team should plan that extra into the Airflow image or virtual environment from the start rather than discovering the gap at the first run. If the files are already read by Trino through a catalog, a plain SQL step over the existing table is usually simpler to operate than a copy step.
Before building the transfer, confirm the destination first. Name the Trino catalog and schema that will receive the data, and confirm that the catalog's connector can actually write the table you intend to create. Confirm the file formats in the bucket, because the transfer and the target table must agree on how the data is encoded, and confirm whether the table is a managed table in a lakehouse format or a plain external layout. Trino's Iceberg connector, for example, reads and writes Parquet, ORC and Avro, while the table format and catalog choice determine what schema changes and rewrites are possible later.
Confirm credentials as a separate workstream. The Airflow connection for Google Cloud and the Airflow connection for Trino should each use a dedicated service identity with the narrowest bucket and catalog permissions the job needs, stored in a secrets backend rather than in DAG code. Ask the operator's owner which bucket prefixes the transfer may read, whether the Trino user can write only to the target schema, and how a failed run is rolled back or re-run without duplicating rows. Document those answers before the first production schedule, because they decide whether the transfer is safe to retry.
Choosing lineage and catalog options for Trino pipelines
Installing the Trino provider with the openlineage extra adds lineage emission to Trino-backed tasks, so each run can report its inputs, outputs and job metadata to a lineage backend. The value depends on that backend existing and being owned by someone. Lineage data is only as useful as the store that receives it, so the decision starts with which backend the organization will run, who can query it, and how long events are kept.
- Backend ownership: Confirm who runs the lineage service, its availability targets and its upgrade path, since a broken backend can make Airflow emit events into nothing.
- Access to lineage records: Decide which teams can see table relationships, because lineage can reveal sensitive dataset names and pipeline structure.
- Retention: Agree how long run-level lineage is kept so it matches audit and incident-review needs.
- Network path: Check that workers can reach the lineage endpoint from inside a restricted network, without opening broad egress.
- Version alignment: Confirm the OpenLineage provider version works with your Airflow 3 release and the Trino provider version you pin.
Before enabling it, confirm the lineage backend, retention and access rules in writing with the team that will operate them.
Running without OpenLineage keeps the Trino provider lighter and removes one external dependency from the pipeline. Operators still see task logs, run history and SQL in the Airflow UI, but they do not get a cross-pipeline map of which Trino tables feed which outputs. This suits small estates, early pilots or teams that already track lineage in another tool and are not ready to add a second path.
- Smaller dependency surface: Fewer packages and no lineage endpoint to patch, monitor or certify.
- Audit evidence shifts to logs: Auditors will ask for run records, SQL history and change records instead of automated lineage graphs.
- Manual impact analysis: Changing an upstream Trino table requires engineers to search DAG code rather than query a graph.
- Later adoption: Enabling lineage later is possible, but historical runs will not have lineage events, so the gap should be documented.
- Consistent naming: Without lineage, table and dataset names in SQL and DAG code become the only cross-reference, so naming standards matter more.
Ask the team whether any regulator, internal control or data owner requires lineage evidence before choosing this path.
Trino queries data through connectors and does not hold its own storage, so a lakehouse needs object storage plus a table catalog. The Iceberg connector works with Hive Metastore and AWS Glue, among other catalogs. Choosing one of these means the metadata service becomes a shared dependency for every Trino and Airflow job that reads or writes Iceberg tables.
- Hive Metastore: Common where an existing Hadoop-era estate already runs a metastore, with operational knowledge already in place.
- AWS Glue: Fits organizations already operating on AWS, with catalog access controlled through AWS identity policies.
- Table format support: The Iceberg connector supports format versions 1 and 2, with version 3 documented as experimental, so confirm the version your tables use.
- Write operations: INSERT, UPDATE, DELETE and MERGE are supported, so pipeline designs must plan for concurrent writers and snapshot handling.
- Audit trail: Snapshot history and time travel give a table-level record of changes, which helps with recovery and evidence requests.
Confirm the catalog choice, the storage location and the metastore backup and recovery plan with your data platform owners before building DAGs against it.
The Iceberg connector also works with REST, Nessie and Snowflake catalogs. A REST catalog or Nessie fits teams that want a catalog service separate from a metastore, with branching or versioned catalog behavior in Nessie's case. A Snowflake-backed setup changes the picture because Trino then reads or writes an external Snowflake account, which brings Snowflake access controls and costs into the pipeline path.
- REST catalog: Requires a service you run and secure, with its own availability, upgrade and access model.
- Nessie: Adds versioned catalog behavior, which can support review and rollback workflows, but adds another system to operate.
- Snowflake connector: Queries and creates tables in one Snowflake database per catalog, so multi-database designs need several catalogs.
- Dual control points: Snowflake roles and Trino access rules both apply, so auditors may need evidence from both systems.
- Other connectors: Trino lists many connectors, including PostgreSQL, MySQL and BigQuery. Each one changes credential handling and network paths.
Ask the catalog owner which system is the system of record for table definitions and who approves changes to it.
The lineage and catalog choices combine into an operating model. A team with OpenLineage and a governed Iceberg catalog can answer questions about data origin, change history and run-level activity from system records. A team without lineage and with a simpler catalog depends more on logs, naming discipline and documented procedures. Either model is valid, but each leaves a different set of evidence and a different set of failure modes.
- Incident scope: Lineage shortens the time to find affected downstream tables, while logs-only setups need manual tracing.
- Change control: Catalog changes and DAG changes should be linked in the change record, whichever catalog is chosen.
- Access evidence: Each catalog and the lineage backend need their own access reviews and logging.
- Recurring work: Lineage backends, metastores and catalog services add patching, backup and capacity tasks that continue after the build.
- Scope of cost: Build effort depends on the number of catalogs, connectors and environments, so ask for a quote that lists each system separately.
Before committing, map each choice to the specific audit questions your organization expects to answer.
Mapping Airflow connection fields and identities to Trino
This table maps each Airflow connection field, and the identity behind a DAG run, to the Trino setting it feeds. Use it as a checklist when designing the connection and access model, and confirm every key name against the current Trino provider documentation before building. The main takeaway is that Airflow roles govern the Airflow UI and API, so table permissions must be granted to the Trino user that the connection presents.
| Airflow item | Trino setting or control | What to confirm |
|---|---|---|
| Host and port | Coordinator hostname and HTTP or HTTPS port for the cluster | Point at the coordinator, not individual workers, and confirm the port your cluster exposes |
| Schema | Default schema for unqualified table names in SQL | Confirm where the catalog is set, since schema alone does not select a catalog |
| Login | Trino user name presented on each query | Confirm the provider sends this value as the Trino user that access control evaluates |
| Password | Credential for password file or LDAP authentication | Confirm which authentication types the cluster enables and where the provider reads the secret |
| Token or extras for OAuth 2.0 or JWT | Bearer token for token-based authentication | Confirm whether the provider reads the token from a field or an extra, and how it refreshes |
| Extras for TLS | HTTPS scheme and certificate verification settings | Confirm the key names and CA bundle path, and keep certificate verification enabled on a shared cluster |
| Airflow role or team | No direct Trino permission; Airflow roles only control Airflow access | Confirm that table grants are written in Trino rules, not in Airflow auth manager roles |
| Group membership | Groups from LDAP or file-based rules that grant table and schema access | Confirm which group source your cluster uses and who owns the mapping from teams to groups |
| Access control backend | File-based rules, Apache Ranger or Open Policy Agent evaluate each query | Confirm the backend's policy format and test denied queries from a DAG before go-live |
Securing connections to a shared Trino cluster
A shared Trino cluster serves Airflow and other clients, so the controls around it need to be decided before the first DAG connects. The cards below cover how Airflow authenticates to Trino, where its credentials live, how traffic is encrypted, how service accounts are split by team, and how catalog rights are limited for the people who write DAGs.
Choose an authentication method
Trino documents password file, LDAP, OAuth 2.0, Kerberos, certificate and JWT authentication. For Airflow, a non-interactive service identity is usually the question: a password file or certificate suits a small, controlled cluster, while LDAP or Kerberos fits an existing directory. Confirm with your Trino administrator which method your cluster enforces and whether the Trino provider connection can present it.
Keep credentials out of DAG code
Store the Trino connection username, password or certificate reference in an Airflow connection backed by a secrets store, not in DAG files, environment variables baked into images or task parameters. DAG files are readable by every author with repository access, so a password written there is effectively shared with the whole team. Ask who can read the secrets backend and whether rotation is possible without a DAG change.
Encrypt traffic to the coordinator
Enable TLS on the Trino coordinator and make the Airflow connection verify the server certificate rather than skipping verification. Plaintext access over a shared network exposes queries, results and any password-based credentials. Check that workers talk to the coordinator over TLS as well, and that the certificate authority trusted by Airflow workers and triggerers is the one your security team manages.
Use per-team service accounts
Give each team or pipeline group its own Trino service identity instead of one shared account used by every DAG. Separate identities make Trino query logs attributable to a team, let you revoke one team without stopping the others, and allow different catalog grants per team. Confirm how the identity is passed to Trino in your setup, since this affects which access rules can match it.
Restrict catalog access with rules
Trino controls access through file-based rules, Apache Ranger or Open Policy Agent. Write rules so each service identity can read and write only the catalogs, schemas and tables its pipelines need. Pipeline service accounts should not hold broad rights over every catalog, and the DAG author account should not hold the service identity's rights at all. Test the rules with queries from each identity before production use.
Limit what DAG authors can see
DAG authors can read the SQL their tasks run, the connection IDs they reference and the schedule that triggers them. They should not receive broad catalog rights through their own Trino identity, or be able to read the connection secret. Design it so authors develop against a non-production catalog or sandbox, and production catalog grants go only to the scheduler-run service identity. Confirm this split with a review of who can edit both DAG code and secrets.
Common failures when Airflow queries Trino and how to fix them
This table lists the failures teams most often meet when an Airflow task runs SQL against a shared Trino cluster, with the likely cause and first fix for each. Start with the symptom in the first column, then check the cause against your connection settings and cluster configuration. Most failures trace back to credentials, version pins, transaction settings or cluster configuration, so confirm these before a DAG is promoted.
| Failure | Cause | Fix |
|---|---|---|
| Authentication rejected | The Airflow connection credentials do not match the authentication method the coordinator enforces. | Confirm the method on the coordinator, then update the connection fields to match. Keep secrets out of DAG code. |
| TLS certificate mismatch | The certificate from the coordinator does not match the host name or trust chain set in the connection. | Use the host name in the certificate, and add your internal CA to the trust store used by Airflow workers. |
| Provider or client version mismatch | The Trino provider release 6.7.1 requires Airflow 2.11.0 or later, and its trino client and common SQL dependencies must resolve together. | Pin provider versions with the constraints file for your Airflow release, then review dependencies after each upgrade. |
| Transactions left open | SQLExecuteQueryOperator has autocommit off by default, so writes may not persist until the transaction is committed. | Enable autocommit for single DDL or write statements, or commit explicitly in multi-statement blocks. Confirm behavior with your Trino connector. |
| Statements fail when split incorrectly | Splitting a script on semicolons breaks statements that contain semicolons in strings, comments or procedural blocks. | Enable statement splitting only for plain scripts, test each script against the target catalog, and run complex statements as single tasks. |
| Catalog or schema not found | The catalog in the query does not match a catalog configured on the cluster, or the connection's default schema does not exist. | List catalogs and schemas from the cluster, then align the query, the connection schema and the catalog configuration file names. |
| Fault-tolerant retry policy misconfigured | Fault-tolerant execution is off by default. A TASK retry policy needs an exchange manager spooling to object storage or HDFS. | Use the TASK policy only for large batch queries with a configured exchange manager. Confirm the setup against the Trino fault-tolerant execution docs. |
Questions teams ask before connecting Airflow to Trino
Can our team connect Airflow to Trino without an implementation partner?
Yes, a team with Airflow and Trino experience can usually do the connection work in-house, and the Trino provider and common SQL provider are open source and documented. The harder part is the surrounding decisions: how the Trino cluster authenticates users, who owns the connection secrets, how lineage is emitted, and how fault-tolerant execution is configured. If your team has not run a shared Trino cluster before, the cost of learning those decisions on production pipelines is the main reason to bring in a partner. A written scope lets you compare that cost with your own staff time.
What drives the effort of a Trino integration?
The effort depends on how much of the Trino environment the pipelines touch. The main drivers are:
- The number of catalogs and connectors Airflow tasks query, since each catalog needs its own connection and access review.
- The authentication model, such as password file, LDAP, OAuth 2.0, Kerberos, certificates or JWT, and whether service accounts are separate from people.
- Whether OpenLineage events must be traced through to a lineage backend.
- TLS configuration between Airflow workers and the coordinator.
- GCS to Trino transfers, which add the Google provider and cloud permissions.
- The current state of the Trino cluster, including version, coordinator sizing and any fault-tolerant execution settings.
Each of these should be listed in the written scope so the quote reflects them.
What do we need to provide before the work starts?
You need to provide access to the Trino coordinator and a test cluster or staging environment, a named identity owner who decides how service accounts map to Trino users, and the secrets backend where Airflow connections will live. You also need a change window for production Airflow and Trino changes, and a contact who can confirm which catalogs and schemas each pipeline should read and write. Ask your Trino administrators to confirm the authentication and access control model, whether file-based, Apache Ranger or Open Policy Agent, before scoping begins.
How do quotes follow from the written scope?
DeployOpen works from a written scope that names the Airflow and Trino versions, the connectors and catalogs in use, the authentication model, the lineage and TLS requirements, the GCS transfers if any, and the deliverables and acceptance checks. The quote follows from that document, and the timeline is agreed in writing before work starts. If the scope changes, such as adding a catalog or switching the authentication method, the change is recorded and priced as an amendment. Ask any provider for the list of assumptions behind the quote so you can see what would change the price.
Does fault-tolerant execution in Trino need extra storage?
Yes, if you enable the TASK retry policy. Fault-tolerant execution is off by default and can be enabled with a QUERY or TASK retry policy. The TASK policy, which Trino recommends for large batch queries, needs an exchange manager that spools intermediate data to S3, Azure Blob Storage, Google Cloud Storage or HDFS. That storage is separate from the Airflow metadata database and must be sized, secured and monitored. The QUERY policy has different requirements, so confirm with your Trino administrators which policy fits your batch workloads before you plan the storage.
How does Trino cluster state affect the Airflow integration?
Airflow tasks submit SQL to the Trino coordinator, so the health of the cluster directly affects pipeline success. Trino is described as one coordinator plus workers, and Trino's documentation recommends a dedicated coordinator for larger clusters. Before connecting Airflow, confirm the Java runtime version, the node sizing and whether queries from scheduled jobs compete with interactive users. Set task timeouts and retries with that load in mind, and agree on who is alerted when queries queue or fail.
What does ongoing support cover after the integration goes live?
Ongoing support is available as a retainer and covers the parts that change after go-live: upgrades of the Trino provider and common SQL provider, Trino version upgrades, changes to connections and catalogs, and checks after Airflow upgrades. Ask any provider for the written list of what the retainer includes and excludes, including response expectations, because those terms are set in the agreement rather than assumed. Your own team should still own pipeline logic, data quality rules and access requests for new datasets.
How DeployOpen scopes and supports the Trino integration
DeployOpen plans and implements the Airflow and Trino integration inside your own infrastructure, whether that is your cloud accounts, data centers or an air-gapped network. Most engagements are fixed-scope, so the Trino provider configuration, the connection and secrets design, the lineage setup and the test runs are described in a written scope with deliverables and timelines agreed before work starts. Ask for the scope to name the Airflow and Trino versions in use, the Trino cluster topology, the authentication method for the shared cluster, the catalogs that DAGs may reach, and the systems that must be exercised in testing. A quote follows from that written scope, so the more specific the scope, the more comparable the quotes will be.
Ongoing support, upgrades and continuous testing are available as retainers after the initial delivery. For this integration, that matters because the Trino provider and the common SQL provider move on their own release cycles, Trino publishes new releases several times a year, and each Airflow minor version has its own support window, so a retainer can cover provider pinning, version checks and regression runs of representative DAGs. Confirm with the vendor which Airflow and provider versions are current on the day you scope the work, and agree in the retainer who performs upgrades in staging before production.
DeployOpen prepares systems, configuration and evidence for compliance assessments, such as connection inventories, access settings, run logs and change records for the Trino integration. DeployOpen does not issue audit reports, attestations or certifications. Those come from independent auditors and certification bodies, so ask your assessor early which evidence they expect from the Airflow and Trino path and confirm that the planned records will satisfy them.
