Making Kafka fast and right-sized: partition counts, producer and consumer settings, compression, storage and retention sizing, large messages and compacted topics.
Sizing partitions from target throughput and consumer parallelism: per-partition throughput, ordering by key, the cost of too many partitions on brokers and OS limits, and why partitions can be added but not removed.
Producer settings that matter: batch.size and linger.ms, compression, acks and idempotence defaults, buffer memory and in-flight requests, with example profiles for high throughput and low latency.
Consumer settings that decide throughput and stability: fetch.min.bytes and fetch waits, max.poll.records, processing time against max.poll.interval.ms, and scaling consumers up to the partition count.
Picking a compression codec for Kafka: CPU versus network and disk savings, batch size effects on compression ratio, producer versus topic level settings and testing with real payloads.
Calculating broker disk needs: ingest rate times retention times replication factor, headroom for reassignment and failures, compacted topics, and when tiered storage changes the math.
Sending records above the 1 MB default: message.max.bytes and topic max.message.bytes, matching producer and consumer limits, compression, and the claim-check pattern of storing payloads elsewhere.
Using compacted topics: cleanup.policy compact and delete,compact, tombstones and delete.retention.ms, what consumers can rely on, and typical uses such as Connect offsets and Streams changelogs.
Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.