Why ACLs and SCRAM credentials need their own migration plan
A ZooKeeper-based cluster that uses AclAuthorizer and SCRAM users has to move its security state to KRaft as a deliberate workstream of its own, because the access rules and credentials do not follow the data or the clients automatically. Kafka 4.0 supports only KRaft mode and removed ZooKeeper mode, so according to the Kafka upgrade notes, a ZooKeeper-based cluster must finish migrating to KRaft before it can move to any 4.x release. Security is therefore on the critical path, not an afterthought at the end of the broker move.
Treat the security state as its own stream with its own owner, inventory and test plan. Moving partitions, switching producers and consumers, and replacing controllers each change how the cluster behaves, but none of them proves that every principal still has the same permissions. Effort depends mainly on how many principals, ACL bindings and client applications the cluster serves, and on how many of those clients are owned by teams you must coordinate with. Confirm those counts before you agree on a schedule, because a partner quote should follow from them.
Start by exporting the current ACLs and SCRAM user list and confirming who owns each principal. In a KRaft cluster the StandardAuthorizer keeps ACLs in the metadata log, and resources without a matching ACL are limited to super users unless allow.everyone.if.no.acl.found is set to true. That default means a missed ACL can cause an outage rather than a silent gap, so each phase needs a documented access test against the expected principals. The Kafka authorization documentation describes the StandardAuthorizer behavior you will verify against.
Which ZooKeeper security settings map to KRaft and which do not
This table compares each ZooKeeper-era security element with its KRaft counterpart and marks what carries over, what changes location and what is removed. Read the rows as a checklist: anything marked as moved or removed needs a replacement control, a configuration change or a test before the cluster leaves migration mode. The main takeaway is that ACLs and SCRAM credentials move into the metadata log, while encryption of that data is no longer provided by Kafka.
| Element | ZooKeeper-era setting or store | KRaft equivalent and what to check |
|---|---|---|
| Authorizer class | kafka.security.authorizer.AclAuthorizer reading ACLs from ZooKeeper | StandardAuthorizer, which stores ACLs in the cluster metadata log. Confirm the class name in your broker configuration before the switch. |
| Where ACLs live | ZooKeeper znodes written by the AclAuthorizer | Cluster metadata log. Export and review existing ACLs first, then recreate or verify them after the controllers are running. |
| Default for resources without ACLs | Governed by the ZooKeeper-based authorizer's own configuration | Resources with no ACL are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Decide this value deliberately and test it. |
| SCRAM credentials | Stored in ZooKeeper | Stored in the metadata log. Credentials can be created at format time with kafka-storage.sh or afterward with kafka-configs.sh. Re-create or verify each user. |
| ZooKeeper password encoder | Password encoder settings protect sensitive values written to ZooKeeper | Removed in KRaft mode. Sensitive data is stored in metadata records without encryption, so encryption at rest must come from the platform (disk, volume or storage layer). |
| Super users | super.users configuration | Still required for administrative access after the switch. Confirm the principal names match the identities your clients use. |
| Client authentication mechanisms | SASL mechanisms and TLS listeners | Listener and SASL settings carry over, but SCRAM users must exist in the metadata log before clients reconnect. Test each listener. |
| Authorizer plug-in model | Pluggable authorizer | Authorization remains pluggable and can integrate with external services. Confirm any custom authorizer supports KRaft before relying on it. |
Which documented paths move ACLs and credentials
The documented route for authorization rules on a KRaft cluster is to recreate them. Once the cluster runs with the StandardAuthorizer, ACLs live in the cluster metadata log, so you define them again with the admin tooling and confirm each one matches the ZooKeeper-era rule. Start by exporting the current ACL list from the source cluster so you have a written reference for every principal, resource type, pattern type and operation in use.
- Inventory before writing: List every ACL on the source cluster, including prefixed and literal resource patterns, and mark which ones belong to service accounts that are being retired.
- Recreate with the admin tool: Use
kafka-acls.shagainst the KRaft controllers to add each rule, grouped by application so failures are easy to trace. - Account for super users: Under the StandardAuthorizer, resources without ACLs are restricted to super users unless
allow.everyone.if.no.acl.foundis set to true, so an omitted rule can block a client rather than fail open. - Diff the result: Compare the KRaft ACL list with your inventory and record every difference, including rules you intentionally dropped.
- Test per principal: Confirm that each application can still produce or consume on the resources it needs and is refused on the ones it should not reach.
Confirm the exact flags and behavior for your Kafka version in the authorization documentation before you script anything.
SCRAM credentials are stored in the metadata log in KRaft mode, and the documented way to create them on a running cluster is the configuration tool. You recreate each SCRAM user with the new mechanism and password, then update the client configuration that uses them. The password itself is not carried over from ZooKeeper, so this step is also a credential rotation point.
- Pick the mechanism per user: Choose SCRAM-SHA-256 or SCRAM-SHA-512 for each user and keep the choice consistent with the listener configuration.
- Create users with kafka-configs.sh: Add each user's credential through the configuration tool and record who received which secret through your normal secret handling process.
- Plan the cutover of clients: Clients using old credentials will fail once listeners stop accepting them, so schedule the switch with each application owner.
- Remove what you no longer need: Delete users that are retired so their credentials do not remain valid in the metadata log.
- Verify authentication: Log in with each updated client and check the broker logs for authentication failures during the window.
Check the SASL documentation for your version to confirm the command syntax and whether a credential can be updated in place.
The other documented route for SCRAM is at storage format time. The storage tool can write initial SCRAM credentials into the metadata log while formatting a new controller or broker directory. This suits a new KRaft cluster built alongside the old one, where you want bootstrap credentials in place before any node starts. It does not move existing ZooKeeper data, so the users it creates still need to be matched to the applications that depend on them.
- Use it for new nodes: Apply the format-time option when you build fresh controllers, so the first metadata state already includes the bootstrap admin credential.
- Keep formatting consistent: Every node that formats the same cluster needs the same cluster ID and matching credentials, so script the format command once and reuse it.
- Limit the bootstrap user: Create only the administrative user at format time and add application users afterward, so bootstrap secrets stay small in number.
- Store secrets carefully: Credentials passed to the tool can appear in shell history and process listings, so use files or a secret store where your environment allows it.
- Confirm the flags: Check the format command options against the SASL and storage documentation for your release before running it on production hardware.
After formatting, run a login test with the bootstrap user and then repeat the per-user checks from the other tabs.
Kafka's authorization layer is pluggable, and the security overview states that it can integrate with external authorization services. If your organization already manages access in a central policy engine, the ACL recreation work may be replaced by mapping existing policies to an authorizer implementation. This is a design decision rather than a copy step, and it changes who owns the authorization record. Treat it as a separate project with its own testing scope.
- Decide where policy lives: Keep Kafka ACLs in the metadata log if the platform team owns access, or move decisions to an external service if security governance already runs there.
- Check the authorizer contract: Confirm with the vendor or project that the external implementation supports the operations, resource types and principal names your clients use.
- Plan for availability: An external decision point adds a dependency to every authorization check, so define behavior when the service is unreachable.
- Audit the mapping: Record how each existing ACL maps to the external policy so reviewers can trace decisions across the migration.
- Test the same cases: Run the per-principal tests from the ACL tab against the external setup before cutover.
Confirm supported authorizer options and their configuration keys in the security documentation for your Kafka version before you choose this route.
How to sequence the security cutover around the migration phases
This sequence moves ACLs and SCRAM credentials out of ZooKeeper and into the KRaft metadata log without a window where clients lose access. Each phase ends with an access test matrix, and ZooKeeper settings are removed only after those tests pass.
Inventory principals, ACLs and listeners
Before touching the cluster, export every ACL, every SCRAM user and every listener with its security protocol, and map each to the client group that depends on it: producers, consumers, Kafka Connect workers, Streams applications, admin tools and inter-broker traffic. Record which principals are used by which applications and who owns each one. The step is done when the team has a single inventory document that the access test matrices in later steps can reference. Confirm with your Kafka administrators which ACLs exist on the live cluster, since ACLs added by hand are easy to miss in older runbooks.
Build the KRaft authorizer and SCRAM users in a test cluster
In a separate KRaft test cluster, configure the StandardAuthorizer and recreate the same ACLs from the inventory. Create the SCRAM users with kafka-configs.sh, or with kafka-storage.sh at format time if you bootstrap credentials that way. In KRaft, the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true, so confirm that setting explicitly for your environment. The step closes when every principal in the inventory authenticates and passes its test matrix, and each denied action is confirmed to be intentionally denied.
Test access from each client group before dual running
Run a matrix that records, for each client group, the authentication mechanism, the topics and groups it touches, the operations it performs (read, write, describe, create, alter) and the expected result. Include negative cases such as a consumer attempting to produce and a Connect worker reading a topic outside its prefix. Repeat the matrix after every change to the authorizer configuration. The output is a signed-off matrix that the platform owner and each application owner have reviewed, which becomes the baseline for the production phases.
Run the dual-running phase with matching security configuration
During dual running, the ZooKeeper-based cluster stays live while controllers are brought up in migration mode. Keep the existing ZooKeeper authorizer and credentials in place so clients continue to work, and verify that the KRaft-side ACLs and SCRAM users written during the test cluster work are in sync with the ZooKeeper data for every principal in the inventory. Rerun the access matrix from each client group against the migrating cluster. The step closes when the matrix passes with no unexpected denials and the team has confirmed that no client depends on an ACL that exists only on one side.
Migrate to KRaft with controllers in place
Move brokers to KRaft mode in the sequence described in the migration documentation, with the KRaft controllers running and the cluster in migration mode. Rerun the full access matrix after each broker role change, not only at the end, so a failure can be traced to a specific change. Check that the authorizer, the SCRAM credentials and the listener security settings are reading from the KRaft metadata and that clients reconnect with the same principals. Finalize only after the matrix passes on every phase and the team has reviewed the revert options in the sibling rollback topic, because finalization is one-way.
Remove ZooKeeper settings and authorizer config
Only after the access matrix passes on the finalized KRaft cluster, remove the ZooKeeper connection settings, the ZooKeeper password encoder settings and the AclAuthorizer configuration from broker and controller configuration files. Keep a copy of the previous configuration in version control so the change can be reviewed. Note that in KRaft mode Kafka stores sensitive data in metadata records without encrypting it, so the team should confirm how credential material is protected at the storage and access layers. The step is done when the cluster starts cleanly with the reduced configuration and the final access matrix passes again.
Record evidence and schedule access reviews
Archive the final access matrix, the configuration diff, the list of SCRAM users and the ACL export from the KRaft cluster. Assign an owner for each principal and set a recurring review of ACLs and SCRAM users against the inventory from the first step. Confirm which review cadence your own policies require, since this page does not set one. The step closes when the evidence bundle is stored where the security and compliance teams can retrieve it and each principal has a named owner.
What drives the effort of moving Kafka security to KRaft
This table lists the inputs that set how much work a ZooKeeper to KRaft security move takes and whether each one is a one-time task or an ongoing operation. Use it to size your own inventory before asking for a quote. The main takeaway is that ACL and credential volume drive the migration itself, while listener layout and environment count drive the testing that follows.
| Driver | Why it changes effort | One-time or recurring | How to keep it down |
|---|---|---|---|
| ACL rules and resource patterns | Each ACL must be exported, mapped to the StandardAuthorizer model and verified, including prefixed and literal patterns. | One-time migration work, then recurring review of new grants | Consolidate rules into prefixed patterns and remove stale principals before the move |
| SCRAM principals and provisioning method | Credentials must be recreated in the metadata log with kafka-storage.sh or kafka-configs.sh, so each user needs a deliberate re-creation step. | One-time for existing users, recurring for every new user or rotation | Script provisioning from one source of truth so the same command path runs in every environment |
| Client applications and their auth configs | Every application's SASL mechanism, JAAS settings and credentials must be checked against the new cluster before cutover. | One-time per application, recurring when clients change mechanisms | Inventory clients by mechanism and owner, and group them into batches for cutover |
| Custom authorizers or external authorization services | Pluggable or external authorizers add code and integration points that must be re-validated against the KRaft authorizer behavior. | One-time redesign or retest, then recurring maintenance of the plugin | Confirm whether the plugin can be retired in favor of ACLs in the metadata log before building work |
| Listener layout with mixed secured and unsecured paths | Clusters that run unsecured or mixed listeners need each path classified and tested separately, since security is optional in Kafka. | One-time classification and test design, recurring when listeners are added | Remove unsecured listeners where possible and document each remaining path with its owner |
| Number of environments to test | Each environment (development, test, staging, production) needs its own access tests after each migration phase. | One-time per environment for the migration, recurring for regression tests on upgrades | Automate access tests so the same suite runs in every environment after each phase |
| Allow.everyone.if.no.acl.found setting | If this setting is enabled, resources without ACLs stay open, which hides gaps in the migrated ACL set during testing. | One-time review before cutover, then recurring audit of the setting | Keep the default restrictive behavior and confirm every resource has an explicit ACL before cutover |
What usually breaks access after a KRaft move
Access problems after a ZooKeeper to KRaft move rarely come from one bad setting. They come from gaps between where ACLs and credentials used to live, where they live now, and what clients still expect. Each card below names one failure pattern, why it hits this migration, and the prevention step to put in the plan before cutover.
Resources left without ACLs
In KRaft clusters the StandardAuthorizer stores ACLs in the metadata log, and any resource without an ACL is limited to super users. Topics, consumer groups and transactional IDs that existed in ZooKeeper-era ACLs may not have been carried over one for one. Before cutover, export the current ACL list, map each principal to each resource type, and confirm that every production client still has an explicit grant on the resources it uses.
Open fallback set by mistake
The allow.everyone.if.no.acl.found setting, when set to true, lets any principal access resources that have no ACL. Teams often turn it on during migration to stop failures quickly, then leave it in place. Treat it as a temporary diagnostic setting, record who changed it and when, and confirm in a review that it is false or removed before the cluster takes production traffic.
SCRAM credentials not re-created
SCRAM credentials are stored in the metadata log, so they are not carried over automatically from the old ZooKeeper store in the way many teams assume. Create each user's credential with kafka-storage.sh at format time or with kafka-configs.sh after the cluster is up. Keep the source of each password in your secrets manager, and check that every SCRAM user can authenticate on the new cluster before the old brokers are retired.
Secrets stored without encryption
In KRaft mode the ZooKeeper password encoder settings are removed, and sensitive data in metadata records is stored without Kafka encrypting it. Any SCRAM credential or other sensitive value that was protected under the old encoder now depends on the protection of the metadata log directories and their backups. Ask your security team to confirm disk encryption, file permissions, and backup encryption for every controller and broker log directory.
Clients keeping ZooKeeper-era settings
Client configurations, scripts and tooling sometimes still point at ZooKeeper addresses or use options that belonged to the old mode. Those clients fail or connect in unexpected ways once the ZooKeeper settings are removed from the brokers. Inventory every admin script, deployment template and application config that references ZooKeeper, switch them to bootstrap servers and current options, and test each one against a staging cluster in KRaft mode.
Tests that only cover producers
Smoke tests that publish and read a single topic can pass while consumers fail. Consumer groups have their own ACL resource and offset access, so a consumer may be denied even when producers work. Build the test plan to cover produce, consume, consumer group join and offset commit for each principal, plus SCRAM login for every user, and rerun it after each migration phase rather than once at the end.
Questions teams ask before moving Kafka security to KRaft
Can ZooKeeper ACLs and SCRAM users be copied to KRaft automatically?
Not as a single switch. ACLs must be re-expressed for the StandardAuthorizer, because in KRaft they are stored in the cluster metadata log rather than in ZooKeeper. SCRAM credentials also live in the metadata log and can be created with kafka-configs.sh or at format time with kafka-storage.sh. Your team should export the current ACL and SCRAM inventory from ZooKeeper, map each principal and resource, and then recreate them deliberately. Review each entry during the rebuild instead of trusting a bulk copy.
What drives the cost of moving Kafka security to KRaft?
Cost follows the number of principals, topics, consumer groups and client applications that depend on existing ACLs, plus how many authentication paths you run (SASL mechanisms, TLS listeners, and any external authorization service). Most of the effort is discovery: finding which clients use which credentials and which ACLs are no longer needed. Ask any provider for a written scope that separates the inventory, the ACL redesign, the testing per phase and the handover. Ongoing costs are limited to credential rotation and access reviews that your team runs after the migration.
How long does the security part of a KRaft migration usually take?
The duration depends on the size of your access model and how many teams must test their clients, not on a fixed rule. Plan time for the inventory, a review of default-deny behavior, a dry run in a non-production cluster, and a verification window after each migration phase. Kafka's upgrade notes state that clusters must move to KRaft before upgrading to 4.x, so that date often sets the deadline. Confirm the timeline against your own environment and your change calendar.
What must our internal team provide before the work starts?
Your team needs to provide a current list of clients, service accounts and their owners, a copy of the existing ACL definitions, and the SASL mechanisms each application uses. You also need access to the ZooKeeper ensemble for export, a staging cluster that mirrors production listeners, and a named approver for each application team. Without owners for each principal, nobody can confirm whether a missing ACL is a cleanup or an outage. Agree these inputs in the written scope before work begins.
Can our own engineers move Kafka security to KRaft without a partner?
Yes, if they have hands-on experience with Kafka authorization and SASL and enough time for a staged rollout. The steps are documented in the Kafka security and KRaft migration references, so an experienced team can plan the work internally. A partner adds value mainly when the access model is large, the change window is short, or no one on staff has run a KRaft migration before. Whichever path you choose, the decision should rest on a written plan and a tested rollback.
How do we test client access without causing an outage?
Test each phase against a representative set of clients before you move on. Confirm that each principal can still produce and consume on its expected topics, that consumer groups keep their offsets, and that unexpected principals are denied. Keep the StandardAuthorizer default behavior in mind: resources without an ACL are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so a missing rule fails closed. Run denial tests as well as allow tests, and keep the ZooKeeper-based settings available until each phase passes.
Do we need to rotate SCRAM passwords as part of the move?
Rotation is not required by the migration itself, but it is a sensible checkpoint when credentials are recreated in a new store. Plan rotation as a separate, scheduled change so that a failed login can be traced to either the credential or the authorizer. Document which secrets your applications read, where they are stored and who can update them. Confirm your rotation policy with your security and compliance owners before the cutover.
Official Kafka documentation for this migration
The three Apache Kafka 4.3 pages below cover the security model, the credential store and the metadata differences that a ZooKeeper to KRaft move depends on. The upgrade notes set the Java and version prerequisites that must be met before the cluster can move to 4.x. Read them together before writing the cutover plan, because each one changes a different part of the security configuration.
Authorization and ACL model
This page explains how the StandardAuthorizer replaces AclAuthorizer in KRaft clusters and where ACLs live once they are migrated: in the cluster metadata log rather than in ZooKeeper. Check how the allow.everyone.if.no.acl.found setting behaves, because resources without an ACL are restricted to super users unless that setting is true. Compare every existing ACL against the new model before you switch authorizers, and test each principal's read, write and admin rights on a non-production cluster first.
SASL authentication and SCRAM
This page covers the documented SASL mechanisms, including SCRAM-SHA-256 and SCRAM-SHA-512, and how SCRAM credentials are stored in the metadata log. It also describes creating credentials with kafka-storage.sh at format time or with kafka-configs.sh on a running cluster. Confirm which users need credentials in the new cluster, since credentials created in ZooKeeper do not carry over automatically and must be recreated and tested for each client.
KRaft versus ZooKeeper differences
This page lists the configuration that changes between the two modes, including the removal of the ZooKeeper password encoder settings in KRaft mode. It also states that sensitive data in metadata records is stored without encryption, so the metadata log and its disks need the same protection you apply to ZooKeeper data. Use it as a checklist to remove obsolete ZooKeeper properties from broker configuration files and confirm that no broker still expects a ZooKeeper connection string.
Upgrade prerequisites and Java requirements
The upgrade notes state that ZooKeeper-based clusters must migrate to KRaft before upgrading to 4.x, and recommend that KRaft clusters older than 3.3.x first move to 3.9.x. Since Kafka 4.0, brokers, Kafka Connect and tools require Java 17, while clients and Kafka Streams applications require at least Java 11. Inventory every broker, Connect worker and client runtime against these numbers before scheduling the migration window, because a missing JVM upgrade can block a rolling restart.
How DeployOpen scopes and delivers the security cutover
DeployOpen fixes the scope of the security cutover in writing before any work starts. The written scope names the clusters involved, the authorizer change from AclAuthorizer to StandardAuthorizer, the SCRAM users to be recreated, and the access tests that define done. Quotes follow from that written scope, so the price reflects the number of principals, topics, consumer groups and applications that actually need to move, not an estimate made before anyone has looked at the cluster. Before the scope is signed, the team inventories every principal and ACL currently held in ZooKeeper, and records which of them are still in use. Unused service accounts and stale ACLs are often the largest source of avoidable rework, so the inventory is where the reader should expect the most decisions to be made.
Once the scope is agreed, the KRaft authorizer configuration, the SCRAM credentials and the ACL set are built and tested inside the client's own infrastructure, in a non-production cluster first. Each access test is run against the cluster in the phase it belongs to, so a failed test points to one change rather than to a whole migration. Passwords and SCRAM secrets are handled through the client's own secret store, because KRaft stores credential records in the metadata log without encrypting them, and the client's controls around that log should be confirmed before go-live. DeployOpen prepares the evidence for the client's assessment, such as ACL exports, test results and change records, but it does not issue audit reports, attestations or certifications. The client's independent auditor or certification body makes that determination. After cutover, a retainer can cover ongoing access reviews, credential rotation and upgrades if the client wants continued support, and that retainer is quoted separately from the migration scope.
