Apache Superset and Databricks integration: what it connects and how
A team running self-hosted Superset that needs dashboards over Databricks data should treat the databricks extra as the supported starting point. Installing that extra adds the Databricks driver and SQLAlchemy dialect to the Superset image, and the connection itself is then created inside Superset with a token-based connection string that points at a SQL warehouse or cluster. The package names, pinned versions and exact connection string format are not fixed across releases, so confirm them against the Superset release you run and against Databricks' own driver documentation before you write the image build. DeployOpen plans and builds this integration inside your environment, including the image change, the connection and the verification steps, so that the driver version and the Superset version are chosen together rather than discovered during testing.
The decisions that matter most sit before the connection string is written. Decide which Databricks workspace, catalog and schema Superset may read, and whether each Superset user will reach data through one shared service identity or through per-user credentials. The token-based connection in this setup means a personal access token or service principal token is stored in Superset's database configuration, so the token owner, its lifetime, its rotation process and the warehouse's permissions become part of your access model. Catalog handling needs the same care: confirm how the connection's default catalog and schema interact with the three-level names your analysts use in datasets, because a mismatch produces tables that appear in one place and fail in another. Ask your Databricks administrator to confirm which compute resources the token may use and whether they are sized for dashboard concurrency.
Verification should cover the image, the connection and the data path. Confirm that the installed driver and dialect load in the running web and worker containers, test the connection from the Superset interface, and run a representative dataset query that uses your catalog naming. Record the Superset version, driver version and Databricks compute type in your change record, because each upgrade of either side can change the connection behavior and needs the same test again. Quote requests should list the Superset release, the Databricks workspace type, the authentication method, the number of catalogs in scope and the expected dashboard concurrency, since each of these changes the amount of configuration and testing involved.
What to confirm before you connect Superset to Databricks
Six items decide whether a Databricks connection works on the first attempt and stays working after upgrades. Each one depends on the others, so confirm the Superset release and image build before you commit to a warehouse or cluster design.
Superset release and Python environment
Start by fixing the Superset release you will run, then confirm the Python version that release supports. The Databricks driver and SQLAlchemy dialect must match that Python and Superset combination. Check the Superset documentation for your release and the Databricks driver documentation for compatible versions, because this page does not fix a version. Record both so an upgrade later does not break the connection.
Image build with the databricks extra
Superset does not ship the Databricks driver in every install path, so the databricks extra must be added when the image is built. Decide who owns the Dockerfile or build pipeline and how the extra is pinned. A rebuilt image with unpinned driver versions can change dialect behavior without any change in Superset itself.
Databricks workspace and compute target
Confirm whether the connection will target a SQL warehouse or an all-purpose or job cluster, since the HTTP path and sizing differ. Ask the Databricks platform team which workspace, catalogs and schemas the analysts need, and whether compute auto-stops. Dashboards that refresh often can keep a warehouse running and drive cost, so agree on the warehouse size and scaling limits before go-live.
Service identity and access token
Decide which identity Superset will use to query Databricks. A personal access token tied to an individual will stop working when that person leaves, so confirm whether a service principal or another token type fits your Databricks policy. Limit the identity to the catalogs and tables the dashboards need, and confirm with your platform team how tokens are issued, rotated and stored in Superset.
Network path from Superset workers
Queries run from the Superset web and worker processes, so those hosts need outbound access to the Databricks endpoint over HTTPS. Check egress rules, proxies, DNS and any private connectivity your cloud provides. If Superset runs on a restricted network, confirm the path with your network team before you write the connection string, not after the first failed test.
Metadata database and caching layer
Confirm that the metadata database is PostgreSQL or MySQL in a supported version, and not the SQLite default. Alerts, reports, async queries and thumbnails also need the caching layer and Celery workers, which most installs run on Redis. Databricks queries from these features run through those workers, so plan their capacity alongside the connection.
SQL warehouse or cluster: which Databricks target to connect
A SQL warehouse is the compute target Databricks provides for SQL queries, and it fits the pattern most Superset users generate: many short queries from dashboards, filters and charts issued by people who never open a notebook. Superset connects to it through the SQL warehouse's HTTP path, and the warehouse's sizing, type and auto-stop settings decide how long the first query waits and how many queries run at once.
- Startup behavior: A stopped warehouse must start before it answers the first query. Auto-stop settings decide how often that happens, so a dashboard opened after a quiet period may load slowly.
- Concurrency: Simultaneous dashboard loads share the warehouse's capacity. Ask how many concurrent users the warehouse is sized for and whether scaling is enabled.
- Cost: Billing follows warehouse uptime and size rather than individual queries. Auto-stop, cluster size and Superset's caching layer all influence how often the warehouse runs.
- Governance: Access to catalogs, schemas and tables is controlled through Unity Catalog grants on the principal Superset uses, so the connecting identity should hold only the read privileges the dashboards need.
- Confirm with your administrator: The warehouse type, auto-stop duration, scaling limits, which catalogs and schemas the service principal can read, and whether the warehouse is shared with other workloads.
Before you approve a warehouse for Superset, confirm the auto-stop and sizing choices against the dashboards you plan to publish.
An all-purpose or job cluster is general compute that runs notebooks, pipelines and jobs. Teams that already maintain data engineering workloads there sometimes point Superset at the same cluster to avoid a second compute target. This route puts dashboard traffic on infrastructure built for other work, so the trade-offs need to be understood before the connection is made.
- Startup behavior: A cluster can take time to start, and job clusters exist only while a job runs. Dashboards may fail or stall when the cluster is not running.
- Concurrency: Dashboard queries compete with notebooks and scheduled jobs for the same executors. A heavy pipeline can slow every chart that reads from the cluster.
- Cost: Cluster time is billed while the cluster runs, and a cluster kept alive for dashboards can run idle between refreshes. Ask who pays for that uptime and how it is tagged.
- Governance: Permissions are set on the cluster, its data access mode and Unity Catalog objects. Confirm which access mode applies and that the Superset identity cannot reach data beyond its role.
- Confirm with your administrator: The cluster's access mode, its auto-termination setting, whether the cluster is shared, the Databricks runtime version, and the catalog and schema grants for the connecting identity.
A dedicated cluster for Superset, separated from production jobs, is a common way to limit the impact of each workload on the other.
Most teams can decide between the two targets by answering a few questions about their dashboards and their existing Databricks use. The answers usually point clearly to one option, and the checklist below helps you record them before any credentials are issued.
- Query pattern: Mostly interactive SQL from dashboards points toward a SQL warehouse. Mostly Spark transformations or notebook-based datasets point toward a cluster.
- Freshness and latency: Ask whether dashboards must respond quickly during business hours and how long a cold start is acceptable.
- Isolation: Decide whether Superset traffic may share compute with other workloads, and who owns the capacity decision.
- Cost ownership: Name the team that pays for uptime and the chargeback tags that will show it.
- Ownership of access grants: Identify the Databricks administrator who will create the service principal, grant catalog and schema access and review it.
Record these answers in the design document so the connection string, caching settings and grant model are reviewed together, not one after another.
Configuring the Databricks connection in Superset, step by step
This sequence takes a self-hosted Superset from a clean image to a verified Databricks connection, with each step ending in a check that confirms the driver, identity, connection and data are working before dashboards depend on them.
Add the databricks extra to the image build
The platform team adds the databricks extra to the Superset install, typically in the Dockerfile or the requirements file used for the image build, so the Databricks SQLAlchemy dialect and its driver are present in the running web and worker containers. Install the same extra on every container that runs queries, including Celery workers if you use alerts, reports or async queries. Confirm the installed driver and dialect versions inside a running container with pip show, and check them against the Databricks connector requirements listed in the Superset database documentation for your Superset release. The step is closed when the versions are recorded in your build notes and the import of the Databricks dialect succeeds inside the container.
Create a dedicated Databricks service identity and token
A Databricks administrator creates a service principal or dedicated user for Superset, not a personal account, and issues a personal access token or equivalent credential for it. Grant that identity the narrowest permissions it needs: CAN USE on the SQL warehouse or cluster, USE CATALOG and USE SCHEMA on the catalogs and schemas the dashboards read, and SELECT on the specific tables or views. Do not grant workspace admin rights or write privileges. Store the token in your secrets manager and plan a rotation date with an owner. The step is closed when a query run with that identity succeeds against the approved tables and fails against a table outside the grant, which proves the boundary holds.
Build the SQLAlchemy connection string
From the SQL warehouse or cluster page in Databricks, copy the server hostname and the HTTP path, and decide which catalog and schema the connection should default to. Assemble the SQLAlchemy URI for the Databricks dialect with the token, host, HTTP path and catalog values, and keep the full string out of version control. Put the token in the connection configuration as a secret reference or environment-injected value rather than pasting it into the UI as plain text where your deployment allows it. The step is closed when the URI has been reviewed by a second person and every placeholder has been replaced with a value confirmed in the Databricks console.
Register the database and set catalog and schema
In Superset, an administrator opens the database connections screen, adds a new database using the Databricks SQLAlchemy URI, and sets the catalog and schema the connection should expose. Review the advanced options, including any setting that controls which SQL functions or statements are permitted, and keep the defaults unless your security review says otherwise. Limit which users can see the connection, because its credentials reach every dataset built on it. The step is closed when the database entry saves with the expected catalog name visible in the connection details and no credential appears in plain text in the UI.
Run the test connection and browse the schema
Use the test connection action in the database dialog to confirm that Superset reaches the warehouse, authenticates with the token and returns the metadata it needs. Then open the SQL Lab schema browser and expand the catalog to confirm that the schemas and tables visible match the grants given to the service identity, not a wider set. If the test fails, check network routing from the Superset containers to the Databricks host, the token validity, and the HTTP path, in that order. The step is closed when the test passes and the browser lists only the approved catalog, schema and tables.
Verify a sample query returns expected rows
In SQL Lab, the analyst or engineer who owns the data runs a small query against a table whose row count and sample values are already known from Databricks, using a fully qualified name that includes the catalog and schema. Compare the returned row count and a few column values with the result from the Databricks SQL editor run under the same grants. Then create one dataset and one chart in Superset to confirm the query path works end to end. The step is closed when the counts and values match the Databricks reference, and the chart renders without errors.
How Databricks catalogs, schemas and identities map to Superset
This table compares each Databricks object or identity with the Superset object it becomes in the connection, and lists what to confirm in each layer. Read it as a checklist: a Databricks grant does not automatically define what a Superset user sees, so verify access at both the Databricks and Superset levels.
| Databricks object or identity | Superset object it becomes | What to confirm |
|---|---|---|
| Catalog | Database connection, scoped by the catalog named in the connection string | Which catalog each Superset database connection targets, and whether one connection should be created per catalog for separate data domains |
| Schema | Schema selectable within the Superset database connection | That schemas the dashboards need are visible to the service identity, and that unrelated schemas are not exposed in the Superset schema picker |
| Service token or service principal | Credentials stored on the Superset database connection | Token owner, expiry and rotation process, plus whether the token is limited to the SQL warehouse or cluster the connection targets |
| Databricks user or group | Superset user or role assigned through the Superset authentication setup | How identities reach Superset, which roles they receive, and whether Superset roles match the Databricks groups an administrator expects |
| Table and view grants | Datasets Superset can list and chart | Whether the connection identity holds read access to each table or view a dataset uses, and what happens when a grant is removed |
| Superset roles and dataset permissions | Access applied on top of Databricks grants inside Superset | Which dashboards, datasets and databases each Superset role can open, and that these rules are tested separately from Databricks grants |
| Row-level rules in Superset | Filters added to queries issued from Superset | Where row-level rules are defined, which roles they apply to, and whether Databricks row or column controls are also in place for direct SQL access |
| Shared service identity versus per-user identity | Single connection used by many Superset users | Whether queries show the real user in Databricks audit logs, and whether a per-user identity is needed for the access model you must meet |
Why catalog choice and token scope decide what users can query
The catalog and schema set in the Superset connection, together with the scope of the Databricks token behind it, decide which tables a dashboard viewer can reach. Superset does not read Unity Catalog permissions on its own. Its SQL Lab and chart queries run as the identity that the connection string and token represent, so the token's grants are the effective access boundary for every user who can open a chart built on that database connection. A default catalog or schema in the connection string also shapes how unqualified table names resolve, which can make a query appear to hit one dataset when it reads another. Before build, record which catalog and schema each Superset database object points to, which Databricks principal owns the token, and which grants that principal holds on each catalog. Confirm these with your Databricks administrator, including whether the token belongs to a service principal or a user and whether it can expire or be revoked centrally.
A single shared service identity makes per-user accountability harder. When every chart and SQL Lab query arrives at Databricks from one token, query history and audit records show the service identity, not the analyst who pressed run. Superset's own logs can help reconstruct who did what, but they are a separate trail that auditors and incident responders will need to correlate with Databricks records. Adding a second catalog usually means a second Superset database connection or a new token with different grants, so access rules have to be duplicated and kept in step. Rotating a token changes nothing about what users can see, but it does interrupt every dashboard and alert that depends on it until the new secret is in place, so plan the rotation window with the people who own those dashboards. Moving a dashboard to a different warehouse changes the compute, the catalog it reads and sometimes the grants that apply, so the chart definitions may return different numbers without any visible error.
Hardening the Databricks connection and its token lifecycle
The Databricks connection is a standing credential that reaches your lakehouse data from a web application many users can reach. The cards below cover where the token lives, what it is allowed to read, how the connection is verified, who can change it, and how the Superset secret key protects stored credentials. Your team configures each control, and DeployOpen guides the choices and checks them in your environment.
Keep the token out of the database entry
Superset stores database connection details in its metadata database, so a token pasted into the SQLAlchemy URI is readable by anyone with access to that database or a backup of it. Prefer the Superset secret store or your deployment's secret manager, injected as environment variables or mounted files at startup. Check that the metadata database backups are encrypted and that no dump or export of connection details leaves your controlled environment. Confirm the token never appears in logs or exported dashboard definitions.
Scope the Databricks identity to read-only
Create a dedicated Databricks service principal for Superset rather than reusing a personal token. Grant it only USE CATALOG, USE SCHEMA and SELECT on the catalogs, schemas and tables that dashboards need, and no write, ownership or workspace administration rights. Ask your Databricks administrators to confirm these grants in the Unity Catalog model your workspace uses. Review them whenever a new dataset is added to a dashboard.
Keep TLS verification on
Connections to the Databricks workspace endpoint should validate the server certificate chain and hostname. Disabling verification to clear a connection error hides a misrouted or intercepted connection that would carry the token and query results. If your network uses a private or internal certificate authority, add that CA to the Superset image or trust store instead of turning verification off. Test the connection after every image rebuild, since a changed base image can change the trust store.
Limit who can edit connections
Anyone who can create or edit a database connection in Superset can point it at a different host, change the token, or add a connection that exposes data the owner never intended. Restrict database connection permissions to a small platform group, and keep analysts on chart and dashboard roles. Confirm the role mapping in your authentication setup so that group membership from your identity provider cannot grant connection editing by accident. Audit changes to database entries on a regular schedule.
Set token expiry and rotation
Give the Databricks token or service principal secret an explicit lifetime, and record the expiry date in your operations calendar. Rotation should follow a written procedure: create the new credential, update the secret store entry, restart or reload Superset, verify a test query, then revoke the old credential. Ask your Databricks administrators which token lifetimes and revocation controls your workspace supports. Rehearse the procedure in staging before the first production rotation.
Review the secret key and its rotation
Superset requires a user-specified SECRET_KEY to start, and encrypts stored connection credentials with it. Keep it in your secret manager, never in the repository, and do not change it casually, because values encrypted under the old key stop decrypting. The configuration documentation describes PREVIOUS_SECRET_KEY for rotation, so use it to keep the old key available while credentials are re-encrypted. Confirm the procedure against the documentation for your release before you change the key.
Failures that show up after the connection test passes
This table pairs common symptoms with their likely causes and a check the team can run. Read each row from the symptom you see, then work down the check column before changing the connection or the warehouse. Compare any exact error text with the Databricks driver release notes before acting on it.
| Symptom | Likely cause | What to check |
|---|---|---|
| Connection test passes but schema browsing returns no catalogs or schemas | The connection points at a catalog or schema the token's identity cannot see, or the default catalog is not set as expected | Run a SHOW CATALOGS and SHOW SCHEMAS query as the same identity in the Databricks SQL editor, then compare grants on each catalog and schema |
| Queries fail with authentication errors after a personal access token is rotated | Superset still holds the old token in the SQLAlchemy URI or in Secure Extra, or the rotated token belongs to a different service principal | Confirm which secret store entry Superset reads, update it, restart the web and worker processes, and rerun a test query |
| First query after a idle period takes much longer than later ones | The SQL warehouse was stopped and must start before it accepts the query | Check the warehouse state and its auto-stop setting, then time a query after the warehouse is running to separate startup delay from query cost |
| Dashboards time out when many users load them at the same time | The warehouse concurrency and cluster size cannot serve the parallel queries each dashboard issues, and Superset caching is not covering them | Count queries per dashboard load, review warehouse queue time in Databricks, and confirm caching and async query settings in Superset |
| Charts that worked before fail after a Superset upgrade | The new Superset release expects a different Databricks dialect or driver version than the image now contains | Compare the installed databricks-sql-connector and SQLAlchemy dialect versions with the Superset release notes and UPDATING.md, then rebuild the image with pinned versions |
| Error text differs from the documented behavior or from a colleague's report | Driver and dialect releases change error wording, and the same root cause can appear under different messages | Copy the exact error and driver version, then match it against the Databricks driver release notes before changing grants or the connection string |
What teams ask before they hire help with the Databricks connection
How much effort does the Databricks connection take compared with the full Superset build?
The connection itself is usually a small part of a Superset project. Adding the databricks extra, confirming the driver and dialect versions, and testing a token-based connection to one SQL warehouse is narrow work. Most of the effort sits in the surrounding build: the metadata database, the caching layer and workers if you need alerts or async queries, identity, roles, and the dashboards that depend on the connection. Ask any partner to split the quote into the connection work and the wider platform work, so you can see which part scales with your data estate.
What drives the cost of a Databricks connection project?
Five factors move the price most:
- The number of catalogs and schemas Superset must expose, since each needs its own access decision and testing.
- The identity model, such as shared service tokens or per-user access, which changes how roles and catalog permissions are designed.
- The network path between Superset and the Databricks workspace, including private connectivity and any proxy or firewall rules.
- The number of dashboards to validate after the connection works, because each one needs its queries and results checked.
- The upgrade cadence you expect, since Superset and the Databricks driver both change over time and need regression testing.
Ask for each factor to be priced separately in the written scope.
Can our internal team set up the Databricks connection without a partner?
Yes, if your team already runs Superset in a container or virtual machine and has someone comfortable with Databricks workspace administration. The steps are documented, and the risk is mostly in access design: which tokens exist, which catalogs they can read, and how the connection string is stored. A partner adds value when the team lacks Databricks administration experience, when several catalogs and identity models must be designed at once, or when audit evidence for access decisions must be produced. Whichever route you take, have a second engineer review the token scope before any dashboard reaches users.
What must our organization provide before the Databricks connection work starts?
Plan for three inputs:
- Workspace administrator access, or a named administrator who can create service principals or tokens, grant catalog and warehouse permissions, and confirm the SQL warehouse or cluster to use.
- A test SQL warehouse or cluster with representative data, so the connection and dashboards can be validated without touching production tables.
- Named data owners who can sign off on which catalogs, schemas and tables Superset may expose, and who approve dashboard results.
Also confirm the Superset hosting location, the Superset version in use, and where secrets will be stored, because these shape the build.
How long does the Databricks connection work take?
Duration depends on scope, not on a fixed template. A single catalog, one warehouse, a token-based connection and a short list of dashboards is a small, bounded task. Multiple catalogs, per-user identity, private network paths and a larger dashboard set extend the work across design, testing and sign-off. The schedule also depends on how quickly your administrators and data owners respond. Ask for a timeline that lists each dependency and the client input it needs, and confirm that the agreed scope and timeline are written down before work starts.
What recurring work follows after the Databricks connection goes live?
The connection needs ongoing attention in four areas:
- Driver and dialect version tracking, because the Superset release notes and UPDATING.md can change connector requirements between versions.
- Token rotation and review of which service principals or tokens still have access.
- Catalog and warehouse permission reviews when data owners add or retire datasets.
- Regression checks on key dashboards after each Superset upgrade, following the backup, staging and upgrade steps in the project documentation.
Decide early who owns each item, and whether a retainer or your own team will cover it.
Which parts of the Databricks setup carry the most security risk?
The main risks are over-broad tokens, connection strings stored in plain configuration, and dashboards that expose more rows than intended. A token with workspace-wide rights gives Superset more access than most dashboards need, so scope it to specific catalogs and warehouses. Keep the token out of version control and use the secret handling your platform team already trusts. Finally, test whether the data each dashboard shows matches the access model your data owners approved, because the connection alone does not enforce row-level rules.
How DeployOpen scopes and delivers the Databricks integration
DeployOpen approaches the Apache Superset and Databricks integration as a written, fixed-scope engagement inside your own cloud accounts or data centers. We agree the warehouse targets, identity model and connection design before any build starts, then test, document and hand the working connection to your team.
How we scope
- Inventory which Databricks SQL warehouses and clusters Superset must reach, and which catalogs and schemas each group may query.
- Confirm the Databricks connector and SQLAlchemy dialect versions against your Superset release and container image.
- Agree how token-based authentication, service principal ownership and token rotation will work before any credentials are created.
How we build
- Install the databricks extra into the Superset image and build the connection string with the HTTP path and catalog settings you approved.
- Configure catalog and schema handling so each Superset database entry points at the intended Unity Catalog scope.
- Test query behavior, timeouts and permissions from a staging Superset instance before any dashboard depends on the connection.
How we hand over
- Deliver the image build definition, configuration files and connection setup documentation in a form your platform team can rerun.
- Document the token rotation procedure, the owner of each service identity and the steps to add a new warehouse.
- Walk your operators through verification queries and the checks to run after a Superset or connector version change.
How we support
- Offer ongoing support, connector upgrades and periodic testing as a retainer, scoped in writing after handover.
- Review driver and dialect version changes when you plan a Superset upgrade, so the Databricks connection is tested before cutover.
- Ask for a quote based on the written scope, which states the warehouses, identity model and deliverables included.
