Day-two work on a running Kafka cluster: metrics and alerting, consumer lag, partition moves, broker removal, admin tools, troubleshooting and KRaft metadata recovery.
The short list of Kafka metrics that catch real problems: under-replicated and offline partitions, active controller count, ISR shrinks, request latency, KRaft quorum metrics and consumer lag, with alert thresholds to start from.
Getting Kafka JMX metrics into Prometheus: the JMX exporter Java agent, Strimzi's built-in metrics and Kafka Exporter, scraping controllers and brokers, and dashboards for brokers, topics and consumers.
Measuring how far consumers are behind: records-lag-max from clients, kafka-consumer-groups describe, exporters for Prometheus, lag per partition versus total lag, and alerting on lag that grows instead of lag that exists.
Adding brokers and moving partitions: generating and executing reassignment plans, throttling replication during moves, verifying completion and balancing leadership afterwards.
Using Cruise Control to balance load and handle failures: goals and proposals, anomaly detection and self-healing, adding and removing brokers, and running it through Strimzi on Kubernetes.
Removing a broker or disk safely in Kafka 4.3: cordoning log directories, building a reassignment plan for every replica, unregistering the broker and checking that no partitions are left behind.
Giving teams a UI for Kafka without a commercial console: kafbat UI features, authentication and RBAC, read-only modes for production, and what to keep in the CLI and automation instead.
Replaying or skipping data by resetting offsets: the reset scenarios, dry runs versus --execute, stopping the group first, per-topic and per-partition scope and recording what was changed.
Finding why replicas fall out of sync or partitions go offline: broker and disk failures, slow followers, ISR shrink metrics, min.insync.replicas errors on producers, and the recovery steps for each cause.
Stopping rebalance storms: max.poll.interval.ms and slow processing, static membership, the classic versus consumer group protocol from KIP-848, and the metrics and logs that show why a group keeps rebalancing.
Keeping brokers from filling up: retention by time and size, segment settings, compacted topics that never shrink, per-topic size metrics for alerts, and emergency steps when a broker runs out of disk.
Investigating controller and metadata problems: kafka-metadata-quorum status and replication output, decoding metadata logs and snapshots with kafka-dump-log, the metadata shell and dynamic log levels on controllers.
Recovering a controller whose metadata directory was lost: waiting for a caught-up majority, checking lag and fetch timestamps, reformatting with the cluster ID and the narrow case for --ignore-formatted.
Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.