Why Kafka delegation tokens matter for Spark jobs
A delegation token lets a Spark driver or worker authenticate to a secured Kafka cluster without each executor holding its own keytab or username and password. A platform team needs this when a Spark job reads from or writes to Kafka, the cluster requires SASL authentication, and the job runs on many worker nodes that the team does not want to provision with long-lived secrets. The token replaces the per-process credential with a short-lived secret that the driver obtains from the cluster and passes to the workers.
Apache Kafka is an Apache Software Foundation project released under the Apache License 2.0, so the broker, its security features and the delegation token mechanism are available without a paid edition from the project. Security is optional in Kafka. Clusters can run unsecured, or mix secured and unsecured listeners, so the team must enable SSL or SASL on each listener that frameworks will use. Before designing token distribution, confirm which SASL mechanisms your listeners accept and whether the token mechanism you plan to use is documented for the Kafka release you will run. The Apache Kafka security overview is the starting point for that check.
Spark runs its workers under one of three cluster managers: its own standalone manager, Hadoop YARN or Kubernetes. The choice matters because it decides where executors start, how they receive secrets and how long they live, but the Kafka side of the design is the same: the driver needs a credential that the brokers accept and the executors need a way to use it. Confirm against the Kafka 4.3 security documentation the exact token issuance, renewal and expiry behavior, the principal that a token carries and the ACLs that principal needs before you commit to a pattern. Treat those details as items to verify in a test cluster, not as settled facts.
What the token carries: the ACL principal and its owner
A delegation token authenticates as a principal, not as the Spark job that requested it, so the ACLs written for that principal decide what every executor holding the token can read and write. Settle the principal model before the first token is issued. A token issued to a broad principal, such as one shared by several teams or one granted wildcard topic access, gives every Spark application that uses it the same broad reach. Confirm in your Kafka version exactly which principal a token maps to, including how the principal name is derived from the requesting identity and any principal mapping rules, because the ACL entries you write must match that name precisely.
The token carries two roles that are easy to conflate. The owner is the principal the token was issued to and whose access it inherits. The renewer is the principal allowed to extend the token's lifetime. The renewer does not gain the owner's data access by renewing, but a renewer who can keep a token alive can keep a leaked token valid for longer. For a Spark workload, keep the owner as a dedicated application principal and name the renewer explicitly, so the identity that renews the token is one you can list, audit and revoke.
Authorization runs on the StandardAuthorizer, which stores ACLs in the cluster metadata log. Resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so a topic that no one has written an ACL for is closed to a token holder by default. Leave that setting at its default in production. Super users bypass ACLs entirely, so a token issued to a super user is unrestricted, and no Spark job should ever run under one. Write the ACLs for the Spark principal before issuing tokens, and test them with a token that has only that principal's rights.
Choosing the token owner, renewer and lifetime settings
This table compares the decisions that shape a delegation token: who owns it, who may renew it, how long it lives, how tokens are scoped to jobs, how the initial credential is obtained and how Spark passes tokens along. Read each row as a trade-off between blast radius, renewal load and job duration. Confirm every setting name, default and limit in the Kafka 4.3 security documentation and your Spark release notes before you commit to a value.
| Decision | Option and when it fits | Trade-off | What to confirm |
|---|---|---|---|
| Token owner principal | A dedicated service principal per framework or platform, used only to mint tokens for jobs. Fits shared Spark clusters where many teams submit work. | A compromised owner can mint tokens for every job it serves, so keep its ACLs narrow and its credential tightly held. | Which principal the broker records as token owner and how ACLs on that principal are written |
| Token renewer principal | The job or platform principal allowed to renew a token. Fits long-running streaming jobs that outlive a single token lifetime. | Renewal rights extend a token's life, so a renewer with broad rights can keep a stolen token alive. | How the renewer is named at creation and whether it can be changed after issue |
| Maximum lifetime | Short maximum lifetime with frequent renewal. Fits batch jobs with predictable runtime and tight exposure limits. | Short lifetimes increase renewal traffic and make failed renewals stop jobs sooner. | The broker-side maximum lifetime setting and its default in the Kafka 4.3 security documentation |
| Renewal window | Renewal set well before expiry with retry headroom. Fits streaming jobs where a missed renewal would stall processing. | Wide margins mean longer exposure for a leaked token, while narrow margins leave little room for broker or network faults. | The renewal interval Spark uses and how it relates to the token lifetime in your release |
| Token per job or per application | One token per job for narrow scope and clean revocation. Fits regulated workloads and jobs that need separate audit trails. | Per-job tokens multiply creation and renewal calls across the cluster, adding operational load. | Whether tokens can be revoked individually and how the token ID appears in broker logs |
| SASL mechanism used to obtain the token | SCRAM-based or Kerberos-based login to the broker that requests the token. Fits environments that already run one of these identity sources. | The initial credential still exists, so its storage, rotation and scope remain the main exposure. | Which SASL mechanisms may request tokens in your version, and whether TLS is required on that listener |
| Spark token fetch and pass-through | Driver obtains tokens at submission and distributes them to executors. Fits clusters where executors cannot reach a keytab or password store. | Token distribution depends on how Spark on your resource manager stores and forwards credentials, which affects who can read them. | How your cluster manager (standalone, YARN or Kubernetes) handles token storage and the Spark settings for it |
| Expiry and failure behavior | Jobs fail fast when a token cannot renew. Fits pipelines where partial output is worse than a restart. | Fail-fast behavior raises alert volume and can restart large jobs during broker maintenance. | What Spark does on renewal failure in your version and how your scheduler retries the job |
Setting up delegation tokens for a Spark job
This sequence takes a Kafka cluster from a secured listener to a Spark job that authenticates with a delegation token. Each stage ends with a check showing that the token carries only the access the job needs and stops working after expiry.
Confirm a secured listener and SASL mechanism
The platform team checks that the brokers expose a SASL listener over TLS, using a mechanism the Kafka security overview documents, such as SCRAM-SHA-256 or SCRAM-SHA-512. Delegation tokens are issued to already authenticated sessions, so an unsecured listener cannot start this flow. Connect a test client with the intended mechanism and confirm the connection succeeds. Then confirm that plaintext access to the same port is refused or absent. Record the listener configuration so later checks can be compared against it.
Create the job principal with minimal ACLs
The Kafka admin creates a dedicated user principal for the Spark job and grants it only the operations it needs on the topics and consumer groups the job uses. In KRaft clusters, the StandardAuthorizer stores these ACLs in the metadata log and blocks resources with no ACL unless allow.everyone.if.no.acl.found is set to true, so confirm that setting before testing. Review the grants with
kafka-acls.sh, then confirm that a second, unrelated principal is denied on the same topic. The check is complete when both results match the design.Enable delegation token issuance on brokers
Set a delegation token master key on every broker through the
delegation.token.master.keysetting, and set the token lifetime limits to the values your policy allows. Keep these settings identical across brokers, then restart them in a rolling fashion so the change applies. Confirm that the principal allowed to request tokens holds the token creation permission. Usekafka-delegation-tokens.shto create a token and describe it, checking the owner, renewers, issue time and expiry time. Confirm exact setting names and behavior in the Kafka security documentation for your release.Configure Spark to obtain and distribute the token
Configure the Spark application so the driver authenticates to Kafka, obtains a delegation token for the job principal, and passes it to executors through Spark's credential handling. No keytab or password should appear in the job image, the job submission or cluster configuration files. The exact Spark settings depend on your Spark version and cluster manager, so confirm them in the Spark documentation for the release you run. The check is complete when driver logs show a token obtained and executors start without any stored Kafka password.
Run a produce and consume test on a lab topic
Run the job against a non-production topic that holds only synthetic records, using a consumer group created for this test. The job should produce and consume successfully with the token, and the output should show the expected message count and offsets. Then run the same test with a principal that lacks the grant, and confirm the operation fails with an authorization error. Both results together show that the ACLs enforce what the token carries.
Verify renewal across a job longer than the lifetime
Run a test job in a lab cluster that lasts longer than the configured token lifetime. Confirm from broker and driver logs that renewal happens before expiry, under the renewer principal you set. Watch closely at each renewal boundary, because a missed renewal stops the job mid-run. The check passes when the job keeps producing and consuming across every renewal without authentication errors.
Confirm access ends after token expiry
In the lab cluster, let a token expire without renewal, then confirm that requests made with it fail with an authentication error. Cancel a second token explicitly and confirm that it is rejected as well. Compare the recorded expiry time with the broker clock, since clock drift changes when a token stops working. Only after these checks pass should the pattern move toward a shared or production environment.
Risks of token leaks and how to limit them
A delegation token works as a bearer credential for its owner's Kafka access, so the risk sits in how it is issued, carried, logged and revoked. The cards below cover the exposure points that matter most when Spark workers hold tokens, and the check to run in your own environment for each.
Limit the window of a leaked token
A leaked token grants the owner's Kafka access until it expires or is cancelled. Set a maximum lifetime that fits your longest batch job, and make sure the renewal cycle does not extend it indefinitely. Confirm the token lifetime and max-lifetime values your Kafka version applies, and test that an expired token is refused by a broker.
Scope the token owner narrowly
A token carries the identity of the principal that requested it, so a broad service principal turns one leak into access across many topics and groups. Issue tokens only for dedicated principals per job or team, with prefixed ACLs. Confirm which principal each Spark application requests tokens as and what that principal can reach.
Keep token material out of logs
Token identifiers and passwords can appear in driver logs, the Spark UI, executor environment dumps or cluster manager job metadata. Anyone who can read those surfaces can reuse the token. Review how your framework and cluster manager display configuration, and restrict access to logs and UIs. Confirm which views show submitted configuration in your deployment.
Restrict who can renew tokens
The renewer is the principal allowed to extend a token's life, so a compromised renewer can keep a stolen token alive. Grant renewal rights only to the framework service principal that runs the renewal loop, not to every worker. Confirm which principal is set as renewer and that no general-purpose account holds that permission.
Encrypt listeners so tokens stay private
Tokens sent over a plaintext listener can be read by anyone on the path, which makes a network capture as damaging as a stolen file. Require TLS on every listener that accepts token-based SASL and block plaintext ports. Confirm that workers, brokers and controllers all reach the secured listener, and that no client falls back to plaintext.
Tie every action to a token owner
Without audit trails, Kafka actions made with a token cannot be traced back to the job or person that obtained it. Record authorization decisions and token issuance events, and map each token to its originating application. Confirm your logging pipeline retains those events and that someone can query them by principal during an investigation.
How the token flow differs across Spark cluster managers
Spark's standalone manager is its own cluster manager, so it schedules executors on worker nodes without an external resource manager handling identity for the job. The driver program coordinates those executors from wherever the application is submitted or from a host the cluster selects, depending on deploy mode. Because no outside scheduler carries credentials for the job, the token flow is largely something your team designs and tests.
- Driver location: Confirm whether the driver runs on a submitting client or on a worker in cluster deploy mode, because that decides which host first holds the token.
- Secret delivery: Tokens usually reach hosts as configuration or files. Restrict file permissions on every worker and remove the files when the application ends.
- Executor refresh: Long-running executors on many workers need renewed tokens before expiry. Confirm how renewal reaches each worker, not only the driver.
- No built-in secret store: Spark does not provide a secrets service for this. Plan integration with a vault or secrets service your organization already operates.
- Submission path: Applications are submitted with the Spark submit tooling on hosts you manage, so access to those hosts is part of the token boundary.
Before relying on this model, test token expiry and renewal on a non-production standalone cluster and record which hosts ever hold a live token.
On Hadoop YARN, the driver runs either in the client process or inside the YARN application master in cluster mode, and executors run in containers scheduled on NodeManagers. YARN is responsible for placing containers, so the token delivery path depends partly on what your Hadoop distribution does with credentials for each application. That makes the platform team's confirmation step more important than it is on standalone clusters.
- Driver location: In cluster mode the driver sits in the application master on a NodeManager host, which changes who can read its local files.
- Keytab placement: Kerberos material for the submitting or service principal usually starts on a gateway host. Limit who can read that keytab and where copies are kept.
- Container localization: Files distributed to containers are copied to NodeManager hosts. Confirm whether a token file is localized, how long it persists, and who can read it.
- Log exposure: Aggregated container logs can capture configuration values. Confirm that tokens are never logged and that log retention matches your policy.
- Submission path: Applications are submitted through YARN client tooling, not an operator, so the gateway host and its access controls remain the main control point.
Confirm with your distribution's documentation how it handles Kafka tokens for YARN applications, and verify on a test cluster that renewal succeeds for long-running jobs.
On Kubernetes, the driver runs in a pod and executors run in separate pods, each with its own service account and volume mounts. Secrets are usually delivered as Kubernetes Secrets, so the delivery method and rotation behavior depend on how those Secrets are mounted. The Apache Spark Kubernetes Operator is an Apache-2.0 subproject of Spark that manages Spark applications with the operator pattern, which changes how applications are submitted.
- Operator submission: When the operator is used, an application is declared as a custom resource that the operator reconciles into driver and executor pods. Token configuration then belongs in that resource and its referenced Secrets, not in a shell command.
- Mounted Secrets: Mounting a Secret as a file lets pods read updated values after rotation, while values injected as environment variables are fixed when the pod starts. Confirm which method your application uses.
- Pod visibility: Anyone who can read pod specifications or describe Secrets in the namespace can see how tokens are delivered. Restrict those permissions with role-based access control.
- Executor churn: Executors are created and removed during a job, so each new pod needs a valid token at startup. Test renewal across executor restarts.
- Operator lifecycle: The operator is its own component with its own upgrades. Confirm which operator and Spark versions you run together and who patches them.
For the operator itself, review the project repository before adoption and confirm that the token Secrets follow your namespace isolation rules.
What to confirm before you rely on delegation tokens
Delegation tokens let Spark workers authenticate to Kafka without carrying keytabs or passwords, but the token flow depends on the exact Kafka, Spark and client versions you run. The cards below list the limits your team should verify in a test cluster before any production job depends on them.
Kafka and Spark release pairing
Confirm which Kafka release and which Spark release together support the token flow you plan to use. Spark 4.2.0 was released July 14, 2026, with 4.1.3 and 4.0.4 on July 15, 2026 and 3.5.9 on July 16, 2026, according to the Spark downloads page. Test the exact pair you will deploy, because support for a flow can differ between a Spark maintenance line and its feature release. Check the Kafka upgrade notes for your broker version as well.
SASL mechanisms that accept tokens
Confirm which SASL mechanisms on your listeners accept a delegation token for authentication. Kafka documents GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER, but the token-capable combinations are not the same as the full list. Verify each mechanism with a token before you assume that a listener you already run will accept one. The security overview is the starting point for this check.
Token renewal for long streaming jobs
Confirm how token renewal behaves when a streaming job runs longer than one token lifetime. Check the maximum lifetime, the renewal window and what happens when renewal fails, because a missed renewal can stop a job partway through a stream. Test a job that runs past the expiry point in a non-production cluster and record whether the job recovers without manual steps.
Clients outside Spark
Confirm whether Kafka Streams applications and custom consumers can obtain and present a delegation token the same way Spark does. Token handling is often implemented in the client library or in the framework, not in Kafka alone. Check each client library version and its configuration keys before you plan a shared token process for several teams.
Spark Connect client limits
Confirm whether your Spark Connect clients can use the same token path as classic Spark jobs. Spark Connect uses a client-server design in which unresolved logical plans go over gRPC, and PySpark clients are supported from 3.4 while Scala clients are supported from 3.5. Verify where the Kafka connection is opened, on the server or the client, because that decides where the token must be available.
Rotation of the token-issuing credential
Confirm how your team rotates the credential that requests delegation tokens, since that credential is the root of trust for every token issued from it. Check whether rotating it invalidates tokens already in use, how long a token outlives its issuing credential and who is alerted when a rotation fails. Write the rotation steps down and rehearse them before the first production job relies on tokens.
Operating tokens: renewal, expiry and incident response
Once Spark jobs authenticate with delegation tokens, the work that protects the cluster is operational: watching whether renewals succeed, alerting on authentication failures at the brokers, and having a tested way to kill a token that may be exposed. Renewal failures are the most common early warning. A job that cannot renew its token will fail partway through a run, often hours after the renewal window was missed, so the platform should alert on renewal errors from the framework side and on the matching authentication errors in broker logs. Set the maximum lifetime of a token deliberately for long-running streaming jobs, and decide who approves a longer lifetime, because a token that lives longer is a larger window if it leaks. Confirm the token renewal and maximum lifetime settings your Kafka version supports in the Kafka security documentation and in the Spark configuration reference for your release.
Incident response depends on being able to expire or revoke tokens for a principal without restarting every broker or rotating the underlying SCRAM credentials for all users. Write a runbook that names who can revoke tokens, how the owner is identified from the token owner and renewer fields, which ACLs the affected principal holds, and how to confirm that existing jobs fail closed. Test the revocation commands in a non-production cluster running the same Kafka version you operate, because revocation behavior and the admin client options for it have changed across releases and should be confirmed for your version. Because the Kafka time based release plan supports rolling upgrades from each of the last three releases to the latest, a broker upgrade is routine and should trigger a retest of token issuance, renewal, expiry and revocation before the change is accepted. Rehearse the full loop at least once per platform change, with the framework team, the platform team and the security owner taking their roles.
Questions teams ask before they hire help with Kafka delegation tokens
How much effort does a delegation token rollout take?
The effort depends on how many Spark applications need tokens, how mature your Kafka ACL design is, which cluster manager runs the jobs, how far your SASL setup already goes, and how deep the testing must be. A small set of applications on a cluster with clean prefixed ACLs is a very different job from dozens of jobs with shared service accounts and no ACL owner. Ask for the effort to be broken into discovery, broker configuration, framework configuration, token renewal and expiry testing, and handover, so each driver is visible and can be changed in scope. No duration is stated here because it should follow from a written scope, not from a general rule.
Can our team set up Kafka delegation tokens without outside help?
Yes, many teams can, provided they already run a secured Kafka cluster and have engineers who understand both Kafka's SASL and ACL model and the token handling in their framework. The harder parts are usually not the broker settings but the coordination: deciding which principal each job carries, checking that renewal outlasts long-running streaming jobs, and writing ACLs that do not grant more than the job needs. Outside help is most useful when the team lacks one of these skills or needs an independent review before production use. Your team can run a short internal pilot with one non-critical application to judge the gap before deciding.
What do we need to provide for a delegation token engagement?
The work needs access that matches its scope, which normally includes administrative access to the Kafka cluster for configuration and a test cluster that mirrors production security settings. You should also name owners for ACLs, because every token carries a principal and someone must approve what that principal can read and write. Spark administrators need to share how jobs are submitted, which cluster manager they use and how long their jobs run. Keep a list of the applications in scope with their topics and consumer groups. Agree on who receives credentials and how they are stored before work begins, and confirm that change windows and a rollback path are available.
How long does a delegation token project take?
The duration depends on the written scope, so no general timeline applies. The scope fixes how many applications, clusters and cluster managers are covered, what testing is included and whether production cutover is part of the work. Ask the provider to state the phases, the dependencies on your change process and the points where your staff must review or approve. Ask what happens if an application turns out to need a different ACL model or a framework upgrade. A timeline that is not tied to these items is not a timeline you can plan around.
Do delegation tokens replace Kerberos or SCRAM in our Kafka cluster?
Delegation tokens do not replace your primary authentication. Kafka authenticates clients with SSL or SASL, using GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 or OAUTHBEARER, and a delegation token is issued to an identity that has already authenticated. Whether tokens fit your environment depends on your existing setup: if you use Kerberos with keytabs on the driver, tokens can remove the need to distribute keytabs to workers, and if you use SCRAM, the token design must work with the credentials your jobs already hold. Confirm the current mechanism and the principal mapping before choosing a design, and review the Kafka security overview to check which mechanisms your cluster runs.
What should a delegation token quote include?
A quote should follow from a written scope and list the work in sections your team can check. Ask for the inventory of applications and the cluster managers in scope, the Kafka and Spark versions the work targets, and the principal design, including which principals will carry tokens and which ACLs they receive. Each deliverable should be named with its acceptance test, such as renewal beyond a job's expected runtime and expiry behavior after revocation. The quote should separate one-time work from recurring support, state what your staff must provide, list assumptions about your environment, and say how changes to scope are priced and approved. Ask which tests are included and whether a handover session is part of the fixed scope.
What goes wrong most often with delegation tokens?
The most common problems are tokens that expire before a long-running streaming job finishes, renewal that fails silently after a broker or network change, and ACLs that grant a shared principal more access than any single job needs. Teams also find that a token issued to one principal is accepted for work they did not intend, because ACL scoping was never reviewed. Plan tests that run jobs longer than the token lifetime, that revoke or expire tokens on purpose, and that confirm each application can reach only its own topics and groups. Check the Kafka documentation for the delegation token settings in your release before you rely on defaults.
How DeployOpen scopes and delivers Kafka security work
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including Kafka delegation token setups for Spark and other distributed processing frameworks. Engagements are fixed-scope, with the token lifecycle, principals and ACLs agreed in writing before any cluster change is made.
How we scope
- Inventory which frameworks, jobs and workers need delegation tokens and which still rely on keytabs or passwords.
- Define the token owner, renewer and fetcher principals, and the ACLs each principal needs on topics and groups.
- Agree in writing the token maximum lifetime, renewal interval and expiry handling for long-running streaming jobs.
- Confirm with the Kafka and framework documentation which features your exact versions support before scoping begins.
How we build
- Enable SASL and TLS on the listeners the framework uses, then configure the broker token secret and delegation token settings.
- Configure the framework to obtain, renew and cancel tokens, and verify that workers receive tokens without stored passwords.
- Write ACLs that scope each token principal to the topics, consumer groups and transactional IDs it actually needs.
- Test expiry and renewal under load, including a job that outlives its initial token lifetime, before the build is accepted.
How we hand over
- Deliver configuration as code, with the broker, framework and ACL definitions versioned in your own repositories.
- Document the token lifecycle, principal design, rotation of the token master secret and the runbook for revoking a token.
- Run a handover session with your platform and data engineering teams on issuing, renewing and revoking tokens.
- Provide a test plan listing each verification step, with results recorded for your compliance and audit files.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer, with the scope agreed in writing.
- Review token and ACL changes when Kafka or Spark versions change, since the Kafka upgrade notes and Spark release notes may alter behavior.
- Advise on quotes by listing the frameworks, cluster count, principals and environments in scope so proposals stay comparable.
