When Kafka share groups can take over queue-style work
Teams that run a separate message queue next to Kafka often ask whether share groups let them retire that second system. The answer depends on the work. A share group changes how consumers read a topic: instead of each consumer owning whole partitions, records are handed out to individual consumers in the group, and each consumer acknowledges the records it processes. That per-record model is what resembles a work queue, so share groups are worth evaluating for task-style processing. Confirm the exact delivery and acknowledgement behavior in the Kafka 4.3 documentation before you design around it, because queue semantics such as redelivery timing and acknowledgement handling are where teams most often make wrong assumptions.
Share groups fit workloads where records are independent units of work, where processing order across records does not matter, and where you need more consumers working on one topic than the partition count would normally allow. Typical candidates include background jobs such as sending notifications, resizing images or running document checks. Poor fits include workloads that depend on strict ordering per key, that rely on replaying a log from a stored offset, or that need the topic to feed stream-processing logic. Illustrative example: a team with 6 partitions wants 20 workers pulling image-resize tasks. A standard consumer group would cap useful parallelism at 6, so the team should test whether a share group meets that need, and whether duplicate processing after a failed acknowledgement is safe for their job.
Which queue-style workloads fit share groups
Share groups suit work where any available worker can take any record and order across records matters little. The cards below cover common queue-style patterns, the condition that would argue for a dedicated queue instead, and what to confirm against the Kafka 4.3 documentation before you commit.
Task dispatch to a worker pool
Independent tasks such as sending notifications, resizing images or calling an external API can be published to a topic and taken by any idle worker in a share group. Each record is handed to one consumer, so adding workers spreads load without repartitioning. Choose a dedicated queue instead when each task must run on a specific worker, when tasks need priority lanes with strict ordering, or when your team already runs a queue broker with per-message routing you rely on. Confirm the delivery and acknowledgement behavior in the 4.3 documentation.
Background jobs with retries
Report generation, document conversion and batch exports are often retried after a failure and do not depend on the order of other jobs. A share group lets several workers pull jobs concurrently, which keeps one slow job from blocking the rest. Argue for a dedicated queue when jobs need per-job delay scheduling, dead-letter handling your operations team has already standardized, or visibility into individual message states that your tooling expects. Check how redelivery and poison records are handled in your target release before designing retry logic.
Event fan-out to parallel workers
One event, such as a new order or a file upload, triggers several independent processing steps that run on a shared pool of consumers. Share groups let the pool scale on demand while every record is still processed once by the group. A dedicated queue or a separate topic per consumer type is the better fit when each downstream system needs its own complete copy of the stream, because that is a consumer group pattern rather than a share group pattern. Confirm the fan-out model with your team before choosing.
Email and notification delivery
Outbound messages are usually independent, idempotent at the provider level and tolerant of small reorderings. Workers can take messages from a share group and acknowledge them after the provider accepts them. Keep a dedicated queue when messages must follow a strict sequence per recipient, when provider rate limits need per-queue throttling your platform already provides, or when compliance rules require a separate audit trail for each message. Verify acknowledgement timing against the 4.3 documentation so duplicates are handled deliberately.
Work that needs strict per-key order
Records for one account, device or order must be processed in sequence, for example a ledger update or a state machine transition. Standard partition-based consumer groups preserve order within a partition, which a share group does not provide across consumers for the same key. Do not use share groups here, and keep a consumer group with one consumer per partition. Confirm the ordering guarantees in the 4.3 documentation before any workload is moved.
Scaling workers beyond partition count
A common reason to consider share groups is that a topic has fewer partitions than the workers available to process it, which caps parallelism in a classic consumer group. Share groups let more consumers take records from the same partitions. Partition count alone rarely justifies a switch if the workload is already well balanced and ordering matters. Ask your platform team to measure consumer lag and throughput first, then confirm the feature status for your Kafka version.
Share group settings to decide before rollout
This table lists the configuration and operational choices a team should settle before enabling share groups. Read each row as a question to answer for your own Kafka version, clients and access model, and confirm the specifics against the documentation for your release. The main takeaway is that the broker feature state, client runtime and ACLs must agree before any consumer is pointed at a share group.
| Decision | When it fits | What to confirm |
|---|---|---|
| Broker feature enablement | Any cluster where share groups are planned, checked against the Kafka version in use | Confirm in the Kafka 4.3 documentation whether share groups are enabled by default on your release, and how the feature is turned on for your brokers before rollout |
| Group naming | Teams that will share a cluster and need to trace which application owns each queue-style consumer group | Agree a naming convention with the platform team and confirm how group names map to ACL resources and monitoring dashboards |
| ACL grants for the group | Clusters using the StandardAuthorizer, where resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set | Grant the consuming principal access to the share group and its topics explicitly, then test a consumer under the production principal before go-live |
| Client Java version | Consumer applications that must run on a Java runtime supported by the client library release in use | Clients require at least Java 11 according to the upgrade notes, so confirm the runtime each team uses and the client library version that supports share consumption |
| Topic partition count and retention | Queue-style workloads that still rely on partitions for ordering or throughput, and need predictable retention for redelivery | Size partitions for expected throughput and set retention to cover the redelivery and reprocessing window your application needs, then validate both with a load test |
| Lag and in-flight monitoring | Every production share group, so backlog and stuck records are visible before users notice | Confirm which share group metrics your Kafka release exposes and wire them into alerting, including consumer lag and records that are not acknowledged |
| Security controls on the transport | Clusters with SSL or SASL listeners, where queue clients must authenticate the same way as other producers and consumers | Confirm the authentication mechanism (GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 or OAUTHBEARER) and TLS settings used by the share consumer clients |
| Rollback path | Teams that may need to fall back to a standard consumer group or another queue system if share groups do not meet requirements | Document how consumers would switch back, which topics would need to be drained, and who approves the change |
How share groups behave when workers fail or records are retried
Before a team puts queue-style work on a share group, it needs answers to three questions: how each record is acknowledged, what happens to a record when the consumer holding it crashes or stalls, and how a poison record that never processes successfully is handled. Confirm each answer in the Kafka 4.3 documentation for the release you plan to run, then check it on a test cluster. A useful test kills a share group consumer while it holds a batch, lets a processing step exceed its time limit, and sends one record that always fails. Record what the group does in each case, including which consumer receives the record next and how many times it is delivered.
Redelivery is the behavior that most affects application design. When a consumer does not acknowledge a record, the record becomes available to another member of the share group, so the same record can be processed more than once. Ask the team that owns the cluster which settings control how long a record stays with a consumer, how many delivery attempts are allowed, and what happens after the limit is reached, such as archiving or leaving the record for operator action. Do not assume a default. Read the values from the documentation and your cluster configuration. Do not assume exactly-once processing or ordering across records for share group consumers unless the documentation for your version confirms it.
These answers change the application. Handlers should be idempotent, typically by writing results under a stable record key or an event identifier that lets a duplicate be detected and ignored. Error handling should separate transient failures, which justify another attempt, from permanent failures, which should be moved to a dead-letter topic that your producers or consumers write to explicitly. Operations should alert on redelivery counts and on records that reach the attempt limit, so poison records are visible before they stall a queue. Plan these changes in the design phase, because a team that assumes each record is processed once will find out the hard way in production.
Setting up a share group pilot on a Kafka cluster you control
This sequence takes a test cluster from a supported Kafka release to a working share group pilot, with a check at each step so the team knows the cluster behaves as expected before any application depends on it. The result is a documented pilot that shows how records are delivered and redelivered in your environment.
Confirm a supported release and Java level
The platform team checks the running brokers against the Kafka downloads page, which lists 4.3.1, 4.2.2 and 4.1.2 as supported releases. The Kafka upgrade notes state that since Kafka 4.0, brokers require Java 17 and clients require at least Java 11. Run the broker version and Java runtime on every node and compare the results with those lists. The step is done when every broker reports a supported release and a Java version the upgrade notes allow, and any older cluster has an agreed upgrade path before the pilot starts.
Enable the share group feature
Share groups are switched on through a feature setting that the Kafka 4.3 documentation describes for the release you run. Confirm the exact feature name and enablement command for your version in the upgrade notes, and apply the change to a non-production cluster first. Record the feature levels before and after the change so you have a known rollback point. The step is done when listing the cluster's feature levels shows the share group feature at the enabled value on every broker.
Create a test topic for the pilot
The operator creates a dedicated topic sized for the pilot's test plan rather than for production volumes. Choose a name with a prefix that makes the topic easy to scope in ACLs, for example a prefix that marks it as a share group pilot (illustrative). Use the admin tooling for your Kafka version to create the topic, then describe it to confirm the partition count and replication factor. The step is done when the description shows every partition with a leader, and a test producer can write one record to it.
Grant group and topic ACLs with the StandardAuthorizer
In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log. Resources without an ACL are limited to super users unless allow.everyone.if.no.acl.found is set to true, so the pilot principal needs explicit grants for the test topic and the share group. Confirm the resource types and operation names in the Kafka authorization documentation for your release, and do not set allow.everyone.if.no.acl.found for the pilot. The step is done when the pilot principal can produce and consume on the test topic, and an unlisted topic is denied.
Run a pilot consumer pool in the share group
Start several consumers in the same share group against the test topic, all authenticated as the pilot principal. The count should be higher than the partition count so the test shows that consumers take individual records rather than whole partitions (illustrative setup). Produce a numbered batch of records and have each consumer acknowledge what it processes. The step is done when the acknowledged record IDs cover the whole batch, and any duplicate deliveries are logged so the team knows the delivery behavior it must design around.
Stop a worker and observe redelivery
While records are in flight, stop one consumer in the pool and leave the others running. Watch the share group state and identify the records the stopped consumer had received but not acknowledged. Confirm in the documentation for your version how long those records stay locked before the group can deliver them to another consumer, and do not assume a value. The step is done when a surviving consumer acknowledges every record in the batch, and the team has written down the observed redelivery delay and any duplicates.
Choosing between a share group, a consumer group and a separate queue
A team that already runs Kafka for event streams has the strongest reason to consider share groups, because the brokers, topics, access model, monitoring and client libraries are already in place. The decision then turns on whether the new worker pools need a queue-style delivery contract or whether another consumer group on the same topics is enough.
- Operational reuse: Brokers, security configuration and dashboards already exist, so the added work is mostly topic design and client changes.
- Pool isolation: Several worker pools can read the same topic with different delivery needs without each pool forcing its own partition layout.
- Partition ceiling: In a classic consumer group, parallelism is limited by partition count. Ask whether your expected concurrency exceeds what the current partitions allow.
- Version dependence: Share group behavior depends on your Kafka release. Check the status for your version in the Kafka downloads page before planning around it.
Before you choose, confirm which release line you run and whether your client libraries expose the share group interface you need.
Consumer groups preserve order within a partition, and the message key determines which partition a record lands in. A team that depends on per-key sequence usually designs around one consumer per partition, which keeps each key's records in a single, ordered stream. Share groups hand records to consumers individually, so the ordering a team can assume changes.
- Ordering scope: Identify which business entities need strict sequence, such as an account or an order, and which only need eventual processing.
- Concurrency trade-off: Ordering limits how many workers can process one key at once, which affects throughput for hot keys.
- Retry behavior: A failed record that is retried may be processed after later records, so define whether retries must block the key.
- Documented guarantees: Read the ordering statements for share groups in the Kafka 4.3 documentation for your version rather than assuming partition-level order carries over.
Write down the ordering rule per key and test a retry scenario against your chosen model before the build starts.
Teams that run a queue product today often have integrations, dead-letter handling, dashboards and runbooks built around it. Replacing that product with Kafka means reproducing those behaviors, not only moving messages. Keeping the queue product for some workloads and using Kafka for others is also a valid outcome.
- Integration inventory: List every producer, consumer and connector, and note which ones depend on the queue product's protocol or client library.
- Delivery semantics: Map acknowledgment, redelivery, visibility timeouts and dead-lettering to what Kafka and your application code will provide.
- Routing patterns: Confirm whether the queue product's routing rules, priorities or delayed delivery have equivalents in your design.
- Dual running: Plan how both systems will run during cutover, and who owns each path during that period.
- Skills and tooling: Estimate the operations work for the new platform with your own staff, and ask a partner to quote the migration work in writing.
Before deciding, document each existing queue behavior your teams rely on and check whether Kafka covers it without custom code.
In regulated environments, the question is what evidence shows that each record was produced, delivered, processed and retained. Kafka keeps records according to retention settings, but a processing acknowledgment is an application decision. Auditors will ask where that acknowledgment is recorded and who can change it.
- Acknowledgment record: Decide whether processing outcomes are written to a separate audit topic, a database or both, and who can read them.
- Access control: In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log. Review how Kafka authorization treats resources without ACLs and confirm the setting for allow.everyone.if.no.acl.found in your environment.
- Authentication: Confirm which SSL or SASL mechanism each client uses, and how credentials are issued and revoked.
- Retention alignment: Match topic retention to the period your policy requires, and confirm how deletion is evidenced.
- Dead-letter handling: Define who reviews failed records and how each decision is logged.
Before choosing, ask your compliance lead and auditor which evidence they will accept for delivery and processing, and confirm it can be produced from your chosen design.
Access control and encryption for share group consumers
Share group consumers reach the cluster through the same listeners, authentication and authorization layer as every other Kafka client, so the controls below apply to them directly. The cards cover transport security, credentials, ACLs for group and topic resources, external authorization and the recurring checks that keep group state access narrow.
Encrypt listeners used by share clients
Kafka encrypts traffic between clients, brokers and tools with SSL/TLS, but security is optional, so a cluster can run unsecured or mix secured and unsecured listeners. Share group consumers should connect only to listeners that use TLS on every path that carries records or group requests. Check the advertised listener for each share consumer, and confirm that no plaintext listener is left open for convenience during testing.
Choose a SASL mechanism per client
Kafka authenticates clients with SSL or SASL. The documented SASL mechanisms are GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Pick one mechanism for each consumer application and record it in the client inventory. Confirm in the 4.3 security overview which mechanisms your share group clients support before standardizing on one.
Rotate SCRAM and client credentials
Share group consumers often run as long-lived services, so their credentials outlive the team that created them. Set a rotation schedule for SCRAM users and any OAUTHBEARER or Kerberos principals, and test that consumers reconnect cleanly after each change. Keep rotation steps in the same runbook as the broker maintenance procedures so a rotation does not interrupt record delivery.
Grant ACLs on group and topic resources
In KRaft clusters the StandardAuthorizer stores ACLs in the cluster metadata log. Resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so a new share group can fail to read records until its principal has access. Grant read access on the topic and the share group, and confirm the exact resource types and operations in the 4.3 authorization documentation.
Review who can read or reset group state
Principals that can describe, alter or delete a share group can see its membership and change how records are delivered. Review which operator accounts and automation identities hold group-level permissions, and remove those held by people who no longer run the queue. Confirm which operations control group state in the 4.3 authorization documentation before writing the review checklist.
Assess an external authorization service
Kafka authorization is pluggable and supports integration with external authorization services. An external service can centralize policy across Kafka and other systems, but it adds a dependency on the policy source for every broker decision. Ask whether your identity team needs share group permissions managed outside the cluster, and test broker behavior when the external service is slow or unavailable.
Limits to confirm before you commit to share groups
Before a team commits a queue-style workload to share groups, it should confirm the limits that apply to the exact Kafka release it will run in production. DeployOpen does not assert which limits apply to a given version until the documentation for that version has been checked. Share group behavior is the kind of feature that changes between releases, so the maturity label, the supported settings and the known restrictions should be read from the documentation for the chosen version, not carried over from older guidance or from blog posts. Start with the release line: the Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as the supported releases, and the team should record which of those it intends to run and what its documentation says about share groups in that line.
Next, confirm client library support in every language the team uses. Share group consumption depends on the protocol and client APIs, and a Java client may expose capabilities that a client in another language does not yet offer. For each client, check whether it supports the share consumer and acknowledgement model the team needs, and test the behavior of its acknowledgement and redelivery handling against a non-production cluster. Also confirm monitoring and admin tooling coverage: the team should verify which metrics, command-line tools and administrative operations exist for share groups in the chosen version, and whether the existing alerting on consumer lag still means what the team expects for queue-style consumption. Where a tool does not cover share groups, plan the manual procedure before go-live.
Two areas need separate checks. The first is cross-site replication. MirrorMaker 2 documents replication of topics, consumer groups with offsets and ACLs, but the team should confirm in the Kafka geo-replication documentation how share group state and in-flight delivery are handled, and what a failover means for records already acquired but not yet acknowledged. The second is upgrade behavior. Rolling upgrades are supported from each release in the past year to the latest version, according to the release plan, but the team should confirm in the upgrade notes for each target release whether share group state, settings or client behavior change, and rehearse the move on a copy of production. Record the answers in the design decision, with the version and the documentation date, so the decision can be reviewed when the platform is upgraded.
Questions buyers ask about Kafka share groups
Does using Kafka share groups add a software license cost?
No license fee applies to the Apache Kafka project itself. Apache Kafka is an Apache Software Foundation project released under the Apache License 2.0 (Apache-2.0), and the project offers no paid edition. Your costs come from the infrastructure that runs the brokers, the engineering time to build and operate the cluster, and any services you buy. Check the distribution, container image or managed service you plan to use, because a vendor's packaged build can carry its own terms and support contract. Before you sign, confirm in the Kafka documentation for the release you run that share groups are included and marked as production ready for your use case. Ask any implementation partner to separate software costs from their fixed-scope fee and to list every third-party component the design adds.
What drives the effort for a share groups pilot?
The effort depends on how much of your current queue workload you want to test and how much of the Kafka platform already exists. The main drivers are:
- The Kafka release you run or plan to run, since the supported releases differ in feature status and upgrade path.
- The number of topics, producers and consumer applications that will read through a share group.
- How much your consumers depend on message ordering, redelivery behavior and acknowledgement handling, because these change the application code.
- Whether the cluster already uses KRaft mode, ACLs and TLS, or whether those controls must be added for the pilot.
- How many test scenarios you need, such as failed consumers, poison messages and load peaks.
A narrow pilot on an existing cluster with one or two consumer applications is far smaller than a pilot that also builds the platform.
How does production effort differ from a pilot for share groups?
Production adds the controls a pilot can skip. A pilot proves that consumers can read and acknowledge records as expected. Production also needs the cluster sized for real traffic, monitoring and alerts on consumer behavior, access control through ACLs and TLS or SASL, tested runbooks for failed consumers and broker incidents, and a backup and recovery plan. You also need an upgrade plan that fits the Kafka release cycle, because rolling upgrades are supported only from recent releases to the latest version. Each of these adds design, testing and documentation work, and the total depends on how many applications move and how many teams will operate them. Ask for each control to appear as a named deliverable in the written scope.
How long does a Kafka share groups pilot take?
The duration depends on the factors that set the work, not on a standard number of weeks. The main inputs are the Kafka version you will use, whether a suitable cluster already exists, how quickly your team can grant access to the environment, how many consumer applications are in scope, and how many approval cycles your security and change processes require. A pilot with ready infrastructure and one well-defined consumer moves faster than one that waits for new networks, identity integration or application changes. Engagements are normally fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. Ask for the timeline to list each dependency and the party responsible for it.
What must our team provide for a share groups engagement?
Your team provides the access, decisions and workload details that the work cannot proceed without. Typical inputs include:
- Access to the target environment in your cloud account, data center or restricted network, with the approvals that apply to it.
- The Kafka release, cluster layout and current security configuration, or the plan to build one.
- Representative message samples, volumes, ordering rules and failure cases for each queue workload.
- The owners of each consumer application who can review code changes and test results.
- Named approvers for security requirements, change windows and acceptance tests.
If these inputs arrive late, the schedule moves. Confirm the list with the partner before the scope is signed.
Can an internal platform team adopt share groups without outside help?
An internal team can adopt share groups alone if it already operates Kafka and has engineers who can change consumer code. Apache Kafka is open source and its documentation is public, so the knowledge is available. The questions to answer honestly are whether the team has run KRaft clusters, configured ACLs and TLS, handled upgrades between releases, and tested failure scenarios under load. The team also needs spare capacity for the pilot, because share group work competes with day-to-day operations. If one of these is missing, an outside partner can cover that gap for a fixed scope while your team keeps ownership of the platform afterward. Hiring help does not remove the need for internal operators, so plan who will run the cluster after handover.
When is moving a queue to Kafka share groups worth the work?
A queue migration is worth the work when the benefits of Kafka apply to your workload and the features you depend on are confirmed in Kafka. It is most likely to pay off when you already run Kafka, want to reduce the number of messaging systems to operate, need records retained and replayed for other consumers, or find that per-partition consumer scaling limits your queue workload. It is less likely to pay off when your current queue provides behaviors you rely on, such as message-level delays, priorities or retry rules, and you cannot confirm that the Kafka release you run supports them. Write down each behavior your applications use, test it against the Kafka documentation, and compare the result with the cost of changing consumers and operations.
How DeployOpen scopes and delivers a share group pilot
DeployOpen plans, implements, tests and documents self-hosted Kafka inside your own infrastructure, so a share group pilot runs in your environment under your controls. We start from a written scope that states which queue-style workloads move, which Kafka version and client libraries are in use, and how success will be checked before any build begins.
How we scope
- Inventory the queue-style workloads, their ordering needs, and the message volumes each one carries
- Confirm the Kafka version and whether the share group feature is available in that release before design
- Agree in writing which consumers, topics and access controls are in the pilot and which stay out
How we build
- Configure share groups on a non-production cluster that matches your broker, TLS and SASL settings
- Set ACLs for share group access so only approved consumer applications can read the pilot topics
- Test acknowledgement, redelivery and poison message handling against your own message samples
How we hand over
- Document the share group configuration, client settings and the decision record for queue versus separate queue
- Provide runbooks for checking group lag, stuck deliveries and consumer restarts in your monitoring stack
- Walk your platform and application teams through the pilot results and the open risks before handover
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after the pilot closes
- Track Kafka release status and plan any upgrade of the share group setup in your own change process
- Revisit the pilot scope in writing when workloads expand beyond the agreed topics and groups
