How many partitions does a Kafka topic need?

How many partitions does a Kafka topic need before you create it?

The partition count for a Kafka topic should follow from three inputs: the target throughput, the number of consumers that must read in parallel, and whether records need strict ordering by key. It is hard to reduce later, because Kafka can add partitions to an existing topic but cannot remove them, and adding partitions changes how keyed records map to partitions. Treat the number as a planning decision, not a default to accept when you run the create command on your own cluster.

The Kafka documentation describes partitions as the unit of both parallelism and ordering. Each partition is an ordered log read by at most one consumer in a consumer group at a time, so the partition count caps how many consumers in that group can work in parallel. Ordering is guaranteed only within a partition, so records that share a key must land in the same partition if their order matters. The platform team should own this decision together with the application teams that produce and consume the topic, because the application team knows its ordering rules and consumer capacity, while the platform team knows what the brokers and operating system can sustain.

Because this page covers open-source Apache Kafka running in infrastructure you control, the count also has an operational cost on your own brokers and host limits. Before you create the topic, ask the application team for expected peak throughput, the consumer instance count they plan to run, and the key they will partition on, then have the platform team confirm broker capacity and open file limits for the planned total. Keep the reasoning in the topic request so the next person who changes the count understands why it was chosen.

What a partition count actually controls

The partition count on a topic sets four things at once: how records are ordered, how many consumers can work in parallel, how much load each partition carries, and how much work the brokers do. Each effect pulls the number in a different direction, so decide them together rather than picking a number from a rule of thumb. Per-partition throughput depends on your hardware, message size and replication settings, so measure it in your own environment before you commit to a count.

Ordering guarantees per key

Kafka guarantees order only within a single partition. Records with the same key are routed to the same partition by the default partitioner, so all events for one customer, order or account arrive in sequence for consumers of that partition. Changing the partition count later changes the key-to-partition mapping for new records, which can split a key's history across old and new partitions. Check which key you need ordered, and whether any consumer depends on that order across a resize.

Parallel consumers per group

Within one consumer group, each partition is assigned to at most one consumer at a time. A partition count of 12 therefore caps useful parallelism at 12 consumers in that group, and extra consumers sit idle. Ask each team how many instances it expects to run at peak and how fast it must scale, then size the partition count to that ceiling with some margin. Consumers in separate groups read the same partitions independently, so this limit applies per group.

Throughput share per partition

Each partition is a single ordered log that one leader broker serves, so its throughput is bounded by the leader's disk, network and the producer and consumer clients on that partition. Total topic throughput is roughly the sum of partition throughputs, but a single hot key can overload one partition no matter how many you add. Run a load test with your real message sizes, compression and acks settings to find the per-partition rate, then divide your target rate by that figure.

Replicas and log files on brokers

Every partition has a log on disk for each replica, and each log uses segment files, indexes and open file handles. Multiplying partitions by the replication factor gives the number of partition replicas the cluster must elect leaders for, replicate and recover after a broker failure. More partitions also lengthen leader elections and controller metadata work after failures. Confirm the total partition replica count per broker and the operating system open-file limit before you add partitions across many topics.

Partitions can grow but not shrink

Kafka lets you increase the partition count on an existing topic but does not let you reduce it. Adding partitions also changes which partition new keyed records land on, so ordering for existing keys may break at the cutover point. Plan a starting count with headroom, and document a procedure for consumers that rely on key order if you expect to grow. If you need fewer partitions later, the usual path is creating a new topic and migrating producers and consumers to it.

Partition settings and when each one fits

This table compares the four settings that decide how many partitions a topic gets and how records are ordered. Read each row for when the setting fits and what it costs to change later. The main takeaway is that partition count can be increased on an existing topic but not reduced, so the initial value deserves the most care.

Setting	When it fits	What it costs to change later
Per-topic partition count at creation	Set explicitly for each topic, using the Kafka admin tooling or topic configuration, once you know the target throughput and consumer count for that topic.	Partitions can be added to an existing topic but cannot be removed. Adding partitions also changes key-to-partition mapping for keyed records, which can break ordering for consumers that depend on it.
num.partitions (cluster default for new topics)	Useful as a fallback when teams create topics without specifying a count. Confirm the default value for your version in the Kafka 4.3 configuration reference.	Changing the broker setting affects only topics created afterward. Topics already created keep their counts, so a default that is too low or high is paid for topic by topic.
default.replication.factor and the topic replication factor	The topic-level replication factor sets how many replicas each partition has. Choose it per topic based on durability needs, and use default.replication.factor only as a fallback. Confirm defaults for your version.	Raising the replication factor later needs a partition reassignment that copies data to new replicas, which adds network and disk load. Replicas multiply storage for every partition, so count them when sizing brokers.
Record key (ordering by key)	Use a key when all records for one entity, such as one customer or order, must be consumed in sequence. Records with the same key go to the same partition, so ordering holds within that key.	Ordering is guaranteed only within a partition. Changing the key or the partition count later moves keys to different partitions, so in-flight ordering across the change is not preserved. Records without a key have no per-key ordering guarantee.

How to size partitions from throughput and consumer parallelism

This sequence turns business traffic and consumer design into a partition count you can test, so the number you choose is grounded in measurements from your own message sizes and settings rather than a rule of thumb.

Record peak and average ingest per topic
Owners of each producing application measure the sustained average and the peak message rate, in messages and bytes per second, for every topic that will exist on the cluster. Use production logs, load balancer counters or existing broker metrics where they exist, and label any forecast separately from observed numbers. The step is done when each topic has a peak figure, an average figure, and a stated growth assumption that the business owner has signed off. Ask the team to confirm whether peaks are bursty or sustained, because a short burst often needs more buffer capacity than a steady peak.
Define the largest consumer group and its target
For each topic, identify the consumer group with the most instances you expect to run, and write down the scaling target, such as the number of parallel consumers needed to keep up at peak. Remember that within a group, one partition is read by at most one consumer instance, so the partition count caps useful parallelism. The check is that the consumer team confirms the target instance count and the processing time per message they measured, since a slow handler can make a larger group necessary than the throughput alone suggests.
Measure per-partition throughput on a test cluster
Run a load test on a test cluster that matches your planned broker hardware, Kafka version and replication factor. Use message sizes and key distributions drawn from production samples, and set the producer compression, batching, acks and idempotence values you intend to ship, because these change the result. Record the sustained throughput one partition can carry before latency or broker resource use exceeds your limits. The step is complete when you have a per-partition figure with its test conditions written next to it, and the performance team can reproduce it from the saved configuration.
Divide target throughput by measured throughput
Divide the peak target rate for each topic by the measured per-partition throughput to get the partition count needed for throughput. Use the peak figure, not the average, unless the team has evidence that peaks can be buffered without breaching latency targets. Round up to a whole number and record the calculation beside the test results so reviewers can trace each input. This is an estimate to validate: a result from a single test does not guarantee performance on production hardware, with real key skew, or under broker failure.
Take the larger of throughput and consumer needs
Compare the throughput-based count with the consumer requirement from step two and select the larger value as the base partition count. If the consumer requirement is higher, the extra partitions are there to allow parallel reads, so confirm that the consumer team can run that many instances without exceeding their resource limits. Check that the chosen number still leaves each key on a single partition in a way that keeps the ordering your application needs. Ask the application owner to sign off on the ordering assumption before the number is fixed.
Add headroom for growth and reassignment
Increase the base count with a margin for projected growth and for moving partitions between brokers during maintenance or broker replacement. The margin size is a team decision based on the growth forecast from step one, and it should be written down with its reason. Confirm that the resulting total partitions per broker, across all topics and replicas, stays within the limits your brokers and operating system have been tested against. The output is a proposed partition count per topic with its headroom documented.
Validate the plan under production settings
Before creating the topics in production, rerun the load test with the final partition counts, producer settings and consumer configuration, and include a broker restart or partition move in the test. Compare the observed latency, lag recovery and broker resource use with the targets agreed earlier. Record any gap and adjust either the count or the settings, then repeat the test. The step closes when the results are signed off by the platform, application and consumer owners, and the approved numbers are stored with the topic definitions in version control.

Why too many partitions slow brokers and hit operating system limits

Each partition is more than a queue entry on a broker. Every replica keeps its own log directory with segment files and index files, and each open segment holds file handles while the broker serves it. Multiply that by partitions, replicas and retained segments, and a cluster can carry a large number of open files long before disk throughput or CPU becomes the limit. In a KRaft cluster, the controllers also track every partition's leader, replicas and in-sync state in the metadata log, so a higher partition count means more metadata to write, replicate and apply on each broker. Operating system limits on open files are often the first ceiling a team hits, so check the open file limit on every broker host and compare it with the file handle budget implied by your planned partitions, replicas and segment counts.

Excess partitions also make failures slower to recover from. When a broker stops, the controller must elect new leaders for every partition that broker led, and the affected brokers must catch up on each partition's log before it serves traffic again. More partitions per broker means more elections, more fetch work and a longer window in which some partitions are unavailable or under-replicated. Heap use grows with partition count too, because brokers keep per-partition state in memory for replication, request handling and metrics. The exact threshold depends on broker hardware, record sizes, replication factor, retention settings and how many clients connect, so no single partition count is safe for every cluster.

Treat any candidate partition count as a hypothesis to test. Load the cluster to your target throughput with the partition count you plan to run, then stop a broker and measure leader election time, under-replicated partitions and client latency while it recovers. Watch open file descriptors, heap use and controller metadata activity during the test. The Apache Kafka downloads page lists the supported releases you should test against, since broker behavior changes between them. Record the results alongside the host limits, so the partition count you choose is backed by measured headroom rather than by a guess.

Partition counts by workload type

Topics such as account events are often sized around ordering rather than raw throughput. Kafka guarantees order only within a single partition, so records that share a key are routed to the same partition by the producer's partitioner and reach consumers in the order written. The partition count therefore sets how many distinct key groups can be processed in parallel, and it has to be large enough for consumer parallelism but not so large that each consumer instance holds many small, lightly loaded partitions.

Key cardinality: Count the active keys and their peak event rates. Many keys with uneven traffic can hide a hot key that saturates one partition.
Consumer parallelism: A consumer group can use at most one consumer per partition, so the count caps how far a group can scale out.
Ordering scope: Ordering holds only inside a partition. Changing the partition count later re-maps keys to different partitions, so ordering across the change is not preserved.
Growth path: Adding partitions is possible, but keyed streams need a plan for in-flight ordering during the change.
Illustrative example: A topic for account events with a few thousand active accounts might start with a count chosen for expected peak consumer instances, not for total volume.

Before choosing, confirm the ordering requirement with the owning team, the expected peak consumer count, and whether any single key can produce disproportionate traffic.",

Setting partition counts on a new Kafka cluster

This sequence turns the sizing inputs into topics with explicit partition and replication settings, then checks that the cluster spreads load and holds consumer lag within the agreed bound before production traffic arrives.

Record sizing inputs in a topic register
The platform owner and the producing and consuming teams record each planned topic in a shared register before any command runs. Each row holds the target throughput in messages and bytes per second, the peak-to-average ratio, the key set and whether per-key ordering matters, the number of consumer instances planned for each group, the retention period and the replication factor. Teams should also record the partition count they expect to need in 12 to 18 months, so growth is planned rather than discovered. The step is done when every topic has a row, every row has a named owner, and the sizing arithmetic is written next to the figures it came from, so a reviewer can rerun it.
Set cluster defaults deliberately
Review the broker defaults for num.partitions, default.replication.factor and min.insync.replicas, then set each one to a value the register supports instead of accepting the installer's default. Automatic topic creation should be decided explicitly: many teams disable it in production so that every topic goes through the register. Confirm the configured values on each broker with the configuration tooling for your release, since the exact commands and config keys are documented for the version you install (4.1.x, 4.2.x or 4.3.x). The step is done when a recorded configuration export matches the values in the change request, and the export is stored with the cluster build records.
Create topics with explicit settings
Create every topic through the Kafka admin tooling or through infrastructure as code, such as a Terraform provider or a Kubernetes operator, with partition count, replication factor and topic-level overrides written out in the definition. Do not rely on the broker defaults for any topic in the register. Keep the definitions in version control so each change is reviewed and reproducible. Check the exact flags and resource fields against the documentation for the Kafka release you run, because the admin commands and operator schemas change between versions. The step is done when the topic definitions in version control match the register row for each topic, and the apply run reports no drift.
Verify topic metadata and leader distribution
After creation, describe each topic and compare the reported partition count, replication factor and in-sync replica sets against the register. Then check the leader distribution across brokers. Partitions should be spread across all brokers, and leadership should be balanced so that no single broker carries a disproportionate share of partitions. Where it is not, adjust placement or run the balancing tooling your release documents before load arrives. The step is done when every topic reports the planned settings, every partition has its full in-sync replica set, and the per-broker leader counts differ by no more than the team has agreed to accept in the change request.
Build the load test plan
Write a load test plan that matches the register. Producers should send at the target rate, including the planned peak, using the same keys, message sizes, compression and acks settings the production clients will use. Start the consumer group with the number of instances in the register, not an arbitrary number, so the test reflects the real parallelism ceiling. Agree in writing the lag bound the consumers must hold and the test duration, and name who watches the results. The step is done when the plan is approved by the topic owners and the expected per-partition rate for each topic is written in the plan.
Run the test and check consumer lag
Run the producers and consumer group for the agreed duration, sampling consumer group lag and per-partition throughput throughout. Lag should stay within the bound set in the plan during steady state and recover within an agreed time after the peak. Watch for hot partitions, where one partition's lag grows while others stay flat, which usually signals key skew rather than too few partitions. Also confirm that broker CPU, network and disk usage stay within the headroom the team set. Record every result against the register row. The step is done when the test report shows lag inside the bound for the planned load, and any failed run leads to a documented change to the register, not a silent increase in partitions.
Confirm the sign-off and keep the register current
Topic owners review the test report, sign the register rows, and the platform owner files the change record with the cluster build documents. Confirm before go-live that the process for adding partitions is understood, because partition counts can be increased later but cannot be reduced in place, and increasing them changes key-to-partition mapping for keyed topics. The register becomes the input for every future topic request. The step is done when each topic has a signed row, the change record references the test report, and the register is stored where new topics must be requested.

Why partitions can be added later but never removed

Kafka lets you increase a topic's partition count after it exists, but it does not let you reduce it. That asymmetry should drive the sizing decision on this page. Adding partitions is a routine operation on a running cluster. Removing them is not, so a count that looks too low can be corrected later, while a count that looks too high is expensive to undo. Confirm the exact behavior for your release in the Kafka documentation before you rely on it, starting with the Apache Kafka downloads page to identify the supported version you will run.

Adding partitions has three effects you need to plan for. New partitions start empty, so existing records stay in the partitions where they were written and are not redistributed. Records are assigned to partitions by key hash, and the partition count is part of that calculation, so a key that previously landed on one partition may land on another after the change. Per-key ordering is therefore preserved only within each period before and after the change, not across it. Consumer groups also rebalance when the partition set changes, so the consumer application must tolerate reassignment and must not assume the old key-to-consumer mapping still holds.

If a topic must keep strict per-key ordering, decide the partition count before producers write production data, and leave headroom for expected growth instead of relying on a later increase. If you do need fewer partitions, the usual path is to create a new topic with the target count, run producers and consumers against it in parallel or through a controlled cutover, and retire the old topic once consumers have drained it. Write that migration as a runbook, rehearse it in a non-production environment, and measure consumer lag and ordering behavior during the test before committing to a production window.

Limits, access control and what to confirm in your environment

A partition count is only as good as the environment that runs it. The cards below cover who is allowed to change topics, the host and JVM limits that cap partitions per broker, the rebalance cost of large consumer groups, and the caps your platform team sets. Each one names where to confirm the answer before you finalize a plan.

Who may create or alter topics

Partition changes are topic administration, so they fall under the authorization model. In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Confirm that setting on every cluster, then list which principals hold topic create and alter rights. Check this in the Kafka 4.x authorization documentation and in your own cluster's broker configuration.

Open file limits and JVM memory

Each partition replica adds log segments and open file handles on the broker, so the partition count per broker is bounded by the operating system before Kafka itself becomes the constraint. Check the open file limit for the user that runs the broker process, and confirm the JVM heap and off-heap headroom on your broker hardware. Verify both with a load test that creates the planned partition count, not with a default configuration review.

Rebalance time for consumer groups

A consumer group reassigns its partitions whenever members join, leave or fail, and the time this takes grows with the number of partitions each member must take over. A high partition count can turn routine deployments into long pauses in consumption. Measure rebalance duration in your own test with the consumer versions and assignment strategy you plan to run, and set the planned member count against that result.

Cluster-level caps and quotas

Platform teams often cap total partitions per cluster, per team or per topic, and set client quotas on produce and fetch rates. These limits can block a partition plan even when the brokers have capacity. Ask the platform owner which caps and quotas apply to your namespace or cluster, whether they are enforced at topic creation, and who approves an exception. Record the answers in the design document so the partition plan and the quota plan match.

Questions teams ask before sizing partitions

Can our team size Kafka partitions ourselves or do we need outside help?

Many teams can size partitions in-house, provided someone on the team understands Kafka's ordering and consumer group rules and can run a realistic load test. The deciding factors are the number of topics that need planning, how strict the ordering requirements are across keys, how much load testing effort the workload justifies, and how much hands-on Kafka experience the team has. A platform with a few topics and clear throughput targets is usually easier to size internally than a shared cluster where many teams publish events with different ordering needs. If the team lacks production Kafka experience or has no test environment that resembles production, an outside review can shorten the path to a plan that holds up under load.

What does a partition sizing engagement cost and how long does it take?

DeployOpen does not publish a price list. The cost follows from the written scope, which is agreed before work starts and is typically fixed-scope. The main cost drivers are the number of topics and consumer groups in scope, how many distinct ordering requirements must be reviewed, the amount of load and failure testing needed, whether a test cluster must be provisioned in your environment, and how many producer and consumer code paths must be inspected. Ask for a quote that separates the review of requirements, the test design and execution, and the written recommendations, and that states which items are one-time and which are recurring, such as a retainer for re-sizing as traffic grows. DeployOpen gives a fixed-scope quote from the written scope, and the timeline is agreed in the same document.

What do we need to provide before partition sizing work can start?

The work goes faster when you provide a short list of inputs in advance:

Representative message payloads, including typical and largest sizes, so tests reflect real record size and compression behavior.
The producer and consumer code paths, including how records are keyed, how batches are sent, and how consumers process and commit offsets.
Target throughput figures for each topic, with peak and average rates and the expected growth over the planning period.
Consumer parallelism constraints, such as the number of consumer instances you can run and any downstream system limits.
A test environment with brokers, storage and network similar to production, along with access to metrics for the cluster and clients.

Confirm with your own engineers which figures are measured and which are estimates, because the plan is only as reliable as those inputs.

Can we increase or decrease the partition count of a topic later?

You can increase the partition count of an existing topic, but you cannot decrease it through the normal topic configuration. Adding partitions raises parallelism for new consumers, yet it changes how keys map to partitions for records produced after the change, so existing key ordering assumptions can break for those keys. Adding partitions also increases broker metadata, open file handles and replication work. Removing partitions requires creating a new topic with the desired count and migrating producers and consumers to it, which means planning a cutover and handling records still in the old topic. Because of this asymmetry, it is usually safer to start with a count that covers near-term throughput and consumer parallelism, and to test the growth path before relying on it.

How do we validate a partition plan before production?

Validate the plan with two kinds of tests. Load tests send records at target and peak rates with representative payloads and measure end-to-end latency, broker resource use and consumer lag under the planned consumer count. Failure tests stop a broker, trigger leader changes and rebalance consumer groups while traffic is running, then check that throughput recovers and that ordering holds for keys that must stay ordered. Include a test of a partition increase on a copy of the topic if you expect growth. Record the results against explicit acceptance criteria, such as lag returning to baseline and no error rates above an agreed threshold. Use the results to set the final count and to document the reasoning for future reviews.

How do partition counts affect broker resources and operating system limits?

Each partition is a log on disk with its own segments, index files and replication state, so a higher count increases open file handles, memory use, leader election work and recovery time after a broker restart. Operating system limits on open files and memory maps can become a constraint long before disk or network capacity is reached. Check the file descriptor and memory map limits on every broker against the total partition replica count per broker, and review the broker heap and page cache headroom. Confirm the exact guidance for your Kafka release in the upgrade and operations documentation, because the practical ceiling depends on hardware, replication factor and the version in use.

Does more partitions always mean higher throughput?

No. Throughput rises with partition count only until brokers, disks, network links or consumers become the bottleneck. Partitions set the upper bound on how many consumers in one group can work in parallel, and they spread load across brokers, but each partition has a single leader and its records are ordered only within that partition. Adding partitions beyond what your consumers and brokers can use adds overhead without adding speed. For illustrative purposes, a topic with twelve partitions and only four consumers in a group still processes at the speed of those four consumers, so the extra partitions mainly add broker cost. Measure the bottleneck in tests before increasing the count.

How DeployOpen scopes and delivers Kafka partition planning

DeployOpen approaches partition planning as a written decision, not a guess. We work from your throughput targets, consumer design and ordering needs, then build and document a partition plan for the Kafka cluster you run in your own infrastructure.

How we scope

Collect target produce and consume rates per topic, with peak and average figures from your own systems.
Map which topics need per-key ordering and which consumer groups must scale out.
Agree the written scope, deliverables and timeline before any cluster or topic changes begin.

How we build

Model partition counts per topic against throughput, consumer parallelism and broker capacity, with the assumptions stated.
Test the chosen counts on a non-production cluster with payloads that resemble your real traffic.
Set topic defaults, naming and creation rules so new topics follow the same planning method.

How we hand over

Deliver a partition register listing each topic, its count, key strategy and the reason for that count.
Document the expansion path, including the point at which adding partitions changes key-to-partition mapping for consumers.
Walk your platform and application teams through the plan and the runbook they will use.

How we support

Review partition counts again when throughput, consumer counts or broker hardware change materially.
Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Confirm with your team the monitoring signals that should trigger a partition review in your environment.