How Druid loads Kafka topics for real-time analytics
Druid's Kafka ingestion runs supervisors on the Overlord, which manage Kafka indexing tasks that read each topic by partition and offset. A team that needs Kafka events to be queryable in Druid within seconds should understand this model first, because it determines how data is read, how it is committed, and what happens when a task fails. The streaming path depends on the *druid-kafka-indexing-service* extension, which must be loaded on the Druid cluster, and on a Kafka version of 0.11.x or higher. Druid also reads streaming data from Amazon Kinesis through supervisors, but Kafka is the path this page covers. DeployOpen implements this path in infrastructure the client controls, including the Druid services, the Kafka brokers it reads from, and the network paths between them.
A supervisor is a long-running specification that Druid keeps running. It defines the source topic, the target datasource, the input format and the tuning for indexing tasks, then creates and monitors those tasks as they read assigned partitions. Because each task tracks the offsets it has processed for its partitions, Druid can resume from a known position after a task or server restart, which is how the Kafka indexing service delivers exactly-once ingestion. Teams should confirm the offset and checkpoint behavior for their Druid version against the Apache Druid Kafka ingestion documentation before they design alerting around it.
The architecture decisions come before the first supervisor is submitted. Confirm which Druid release will be deployed and check its Kafka client compatibility, identify the metadata store and deep storage that the Overlord and Middle Managers depend on, and agree which team owns topic partition counts, since task parallelism is bounded by them. A quote or plan should state how supervisors are created and versioned, how the extension is installed on every affected node, and how lag and task failures will be watched in the environment the client runs.
What you need in place before the first supervisor
A Druid supervisor only starts reading once the Kafka side, the Druid services and the network path between them are all ready. Check these components in order, because a gap in any one of them shows up as a supervisor that never leaves its starting state.
Kafka version and release
Druid's Kafka ingestion requires Kafka 0.11.x or higher, but that is a floor, not a target. Run a Kafka release that is still supported, and confirm the current supported list on the Kafka downloads page before you pick a version. Older brokers may lack features your team expects, so plan the upgrade path before you build the pipeline on top of it.
Druid Kafka indexing service extension
Kafka ingestion is not in the base Druid install. You must load the druid-kafka-indexing-service extension on the Overlord and on the task-running nodes, and every node must run the same Druid version. Confirm the Druid release you plan to run on the apache/druid releases page, then check the extension list in your build before scheduling any supervisor.
Overlord and task-running nodes
Supervisors run on the Overlord, which launches Kafka indexing tasks on Middle Manager or Indexer nodes. Historical nodes then serve the segments those tasks publish. Size the task-running tier for the number of partitions you plan to read in parallel, and confirm that Historicals have enough disk to hold segments for your query retention window.
Deep storage, metadata store and ZooKeeper
Druid needs deep storage for segments, such as S3, HDFS or local disk, and a metadata store, typically PostgreSQL, MySQL or Derby. Supervisor specs and segment metadata live in the metadata store, so back it up before the first supervisor. Confirm whether your Druid release still requires ZooKeeper for discovery and leader election in the architecture docs, and provision it with the same care as the rest of the cluster.
Network routes from tasks to brokers
Indexing tasks open their own connections to Kafka brokers, so the route must exist from every Middle Manager or Indexer host, not only from the Overlord. Check that brokers advertise listener addresses these hosts can resolve and reach, that the required TLS or SASL settings are available to the tasks, and that firewall rules allow the broker ports in both directions.
How to configure a Kafka supervisor in Druid
This sequence takes a Kafka topic from a provisioned source to a running Druid supervisor that reads every assigned partition, so you can confirm that rows arrive and queries return them. Each step ends with a check you can run before moving on.
Enable the Kafka indexing service extension
Confirm that the druid-kafka-indexing-service extension is loaded on the Druid services that run ingestion, which in a standard layout means the Overlord and the Middle Managers or Indexers that run the indexing tasks. The platform team makes this change in the Druid common runtime properties and restarts the affected services. Check that the Overlord web console lists Kafka as a supported ingestion type and that the Overlord API accepts a supervisor spec of type kafka, which confirms the extension loaded.
Confirm the Kafka version and topic access
Verify that your Kafka cluster runs a version Druid supports, which the Druid documentation sets at Kafka 0.11.x or higher, and that the Druid service account has a principal with read access to the source topic and its consumer group. Create the topic with the partition count your ingestion plan requires, since Druid assigns tasks to partitions. Check the access by consuming a few records from the topic with a Kafka client using the same credentials Druid will use, and confirm the partition count with the topic description command.
Write the supervisor spec
Write a JSON supervisor spec with type kafka and an ioConfig that names the topic, the consumer properties (bootstrap servers and the security settings your cluster requires), and the inputFormat that matches your message encoding, such as JSON. Add a dataSchema that sets the datasource name, the timestamp column, dimensions, and metrics, and a tuningConfig that sets task counts and segment behavior. Keep the spec in version control so changes are reviewed. Check the file by validating the JSON locally before you submit it.
Submit the supervisor to the Overlord
Post the spec to the Overlord supervisor API, or submit it from the web console data loader, so the Overlord creates the supervisor and starts managing Kafka indexing tasks for the topic. The Overlord stores the spec and schedules tasks from it. Check the response, which returns the supervisor ID, then confirm the supervisor appears in the supervisor list of the web console.
Verify supervisor status and task creation
Query the supervisor status endpoint and confirm the state is RUNNING. Then open the task list and check that indexing tasks exist, each assigned a set of partitions from the topic. The tasks should cover every partition between them, with no partition left unassigned. If the state shows an error or tasks fail repeatedly, read the supervisor status payload and the task logs before changing the spec.
Check offset progress against the topic
Compare the offsets recorded by the supervisor with the latest offsets on each partition of the topic. Druid reads by partition and offset, which is how it provides exactly-once ingestion, so the stored offsets should advance as producers write new records. Check that the lag between the topic end offsets and the supervisor offsets stays small under normal load, and that it shrinks after a producer burst. A lag that keeps growing points to undersized task counts or a failing task.
Query the ingested rows
Run a Druid SQL query against the new datasource, using the web console query workbench or the JDBC or HTTP API, and confirm that the row count and the latest timestamps match what your producers have written. Compare a sample of records against the source topic to confirm that fields were parsed as the dataSchema defines them. The sequence is complete when the supervisor is RUNNING, tasks cover all partitions, offsets advance, and queries return the expected rows.
Sizing ingestion tasks for your partition layout and query needs
A topic with few partitions and modest message rates needs a small supervisor footprint. Kafka indexing tasks divide the topic's partitions among themselves, so once the task count exceeds the partition count, the extra tasks have nothing to read and still hold worker slots. The main risk in this situation is overprovisioning, which takes capacity away from other datasources on the same middle managers or indexers. Illustrative example: a topic with three partitions gains nothing from a task count above three.
- Match task count to partitions: Set
taskCountat or below the number of partitions so that every task reads at least one partition. - Keep replicas low: Each replica runs a full copy of the same partition assignments, which doubles the load for a topic that does not need extra availability.
- Avoid long task durations by default: Short-lived tasks publish segments sooner, but a low-volume topic may not need the extra segment churn.
- Check idle slots: Review supervisor status to see whether tasks are idle while the topic is quiet.
- Confirm lag stays small: A few tasks may keep up easily, so verify that consumer lag stays acceptable at normal rates.
Before you settle on a count, confirm the topic's partition count and steady-state message rate with the team that owns the producers, and check the result against the Druid Kafka ingestion documentation.
A high-throughput topic often has many partitions, and throughput scales with the number of tasks that can read them in parallel, up to the point where cluster resources become the limit. Each task consumes memory, CPU and disk for intermediate segment files, so adding tasks changes the load on the middle managers or indexers as well as the topic. Sizing starts with how many partitions each task will handle and how many worker slots your Druid cluster can give to this supervisor without starving queries.
- Spread partitions evenly: Choose a task count that divides the partitions into groups of similar size, so no task becomes a hot spot.
- Budget worker capacity: Confirm that the number of tasks, multiplied by replicas, fits the worker slots left after other workloads.
- Multiply replicas deliberately: Each replica repeats the work for the same partitions, so a replica count of two roughly doubles ingestion load.
- Balance task duration and granularity: Longer tasks create fewer, larger segments, which usually reduces per-segment overhead on a busy topic.
- Plan for peaks: Size against peak rates rather than averages, since backlog built during a spike must be drained by the same tasks.
Ask the platform team to confirm worker capacity and the producers' peak rates in writing before the supervisor goes live, because those two numbers set the practical upper limit on task count.
When analysts or operational dashboards need events to appear soon after they are produced, handoff timing becomes the main design driver. Tasks publish their segments when they finish, so task duration and segment granularity together control how long an event waits before it is queryable. Shorter tasks and finer granularity make data visible sooner, but they produce more, smaller segments that increase metadata work on the coordinator and later compaction work. The right balance depends on how fresh the data must be and how much segment churn the cluster can absorb.
- Shorten task duration for freshness: Shorter tasks hand off sooner, but each handoff creates more segments to track and maintain.
- Choose granularity with query patterns in mind: Finer segment granularity supports fresh queries but adds segment count, so match it to the time ranges your queries use.
- Expect more tasks at short durations: Frequent task restarts mean more task starts and more replicas running at the same moment.
- Plan compaction: Small segments created for freshness should be compacted later to keep query performance steady.
- Measure end to end: Time the path from an event's production to its first query result, since the supervisor's own status does not show it.
Confirm the freshness target with the dashboard owners in writing, then verify it on the live topic before treating it as a service level.
Estimates of task count from partition counts and paper capacity models are a starting point, not a final answer. Real producers vary in message size, burstiness and key distribution, and those factors change how evenly partitions load and how much memory each task uses during a build. A sizing test with representative message rates shows the actual lag, handoff delay and resource use, which estimates cannot reliably predict. Run the test against a topic that carries the same message shapes and rates as production, with the same replicas and task duration you intend to use.
- Replay representative traffic: Use a copy of production events or a load generator that mirrors their size, rate and key pattern.
- Run through peak and recovery: Measure how long the supervisor takes to clear a backlog after a simulated outage.
- Record resource use: Capture memory, CPU and disk on the middle managers or indexers while tasks run at full rate.
- Test failover: Stop a task or worker and confirm that the supervisor reassigns partitions without losing or duplicating data.
- Compare results to targets: Check lag and freshness against the targets agreed with the business owners.
Ask for the test plan, the measured results and the tuning changes in writing, so the production settings can be traced back to evidence.
Mapping Kafka message fields to Druid columns and timestamps
This table compares each Kafka record field and payload element with the Druid column role it can take. Read it before writing the supervisor spec, because the timestamp and rollup choices are hard to change once segments exist. The main takeaway is that the timestamp column, the input format and the rollup setting decide most of the schema work.
| Kafka element | Druid role | Decision to make | Confirm before loading |
|---|---|---|---|
| Record value (payload) | Source of dimensions and metrics | Choose the input format that matches how producers serialize messages, such as JSON or Avro, and confirm the format is supported in your Druid version. | Supported input formats and any extension required for your payload type in the Druid ingestion docs |
| Record timestamp | __time column | Decide whether Druid uses the Kafka record timestamp or a field inside the payload as the primary time column. | Which timestamp the producers set and whether it reflects event time or publish time |
| Payload event time field | __time column (alternative) | Use this when the business event time differs from the broker append time, and parse it with the timestamp spec. | Time zone and format of the field, and whether any records lack it |
| Nested JSON objects and arrays | Flattened dimensions | Flatten nested fields through the input format's flattening rules so each needed value becomes a named column. | Field paths in a sample of real messages, including optional and missing keys |
| Record key | Dimension (optional) | Include the key as a column only if analysts filter by it, since keys do not always identify the entity in the payload. | Whether keys are set on every record and how they are serialized |
| Headers and partition or offset | Metadata columns (optional) | Map headers or Kafka metadata only if queries need them, because they add columns and can change rollup results. | Which metadata fields the chosen Druid version exposes to the input format |
| Numeric payload fields | Metrics with aggregators | Pick sum, count, min, max or sketch aggregators when rollup is on, and keep raw values as dimensions when they are needed unaggregated. | Which fields are counts or amounts and which are identifiers that should not be summed |
| Rollup setting | Pre-aggregation at ingestion | Enable rollup to cut stored rows for summed or counted metrics, and disable it when each event must stay queryable individually. | Query needs for per-event detail, since rollup cannot be reversed for data already ingested |
Why partition offsets make Druid ingestion exactly-once
Druid's Kafka ingestion tracks progress by partition and offset, not by time or message count. A supervisor on the Overlord manages the indexing tasks that read each assigned Kafka partition, and the offsets those tasks have processed are kept as part of the supervisor's state. When a task fails or a Middle Manager restarts, the replacement task resumes from the stored position for each partition instead of guessing where to start. This is what allows Druid to describe its Kafka path as exactly-once: the data is not re-read from an arbitrary point, and records are not skipped because a checkpoint was lost. The mechanics depend on the topic keeping its messages for the period between a failure and recovery, so retention on the topic is part of the design, not a separate concern.
The guarantee is bounded by the ingestion and handoff path, and that boundary is where most surprises appear. Offsets are only one half of the picture: a task must also publish its segments successfully before the work is committed, and a supervisor reset, a change to the supervisor spec, or a topic that has lost data to retention can all move the starting point. Before relying on exactly-once behavior, have your team read the Kafka ingestion guide in the Apache Druid documentation and confirm, in a test environment, what Druid does under task failures, supervisor resets and offset-out-of-range conditions for the Druid version you run. Record the expected outcome for each case so that operators know which failures need a manual backfill and which resolve on their own.
Securing the Kafka to Druid connection
The Druid supervisor is a Kafka client like any other, so it needs an authentication method, encrypted transport and an explicit grant on every topic and consumer group it reads. Each card below covers one of those layers or the secrets that carry them.
Choose the authentication method
Kafka supports SSL client authentication or SASL, and the documented SASL mechanisms are GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Pick the mechanism your brokers already run, because Druid's consumer properties must match it exactly. Confirm with your Kafka team which listener Druid will use before writing the supervisor spec, since a mismatch fails at task startup rather than at deployment.
Encrypt traffic to the brokers
Kafka's security is optional, so a cluster can expose unsecured and secured listeners side by side. Point the Druid supervisor at the TLS listener only, and confirm that the indexing tasks running on Middle Managers or Indexers trust the broker certificate chain. Check the truststore path and format in the consumer properties, because tasks run on different hosts from the Overlord that submitted the spec.
Grant the Druid principal explicit access
In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log, and resources without an ACL are limited to super users unless allow.everyone.if.no.acl.found is set to true. The Druid principal therefore needs an explicit read grant on each source topic and on its consumer group. Ask whether that default is set on your cluster, since a permissive setting changes what a missing ACL means.
Scope the consumer group
Druid tracks partition offsets to guarantee exactly-once ingestion, so the consumer group it uses must stay stable across supervisor restarts. Grant the Druid principal access only to that group, not to a wildcard pattern shared with other applications. Review the group ACL whenever a supervisor is cloned or renamed, because a new group name starts without the grants the old one held.
Keep credentials out of supervisor specs
Supervisor specs are submitted to the Overlord over HTTP and are stored and displayed in the web console, so inline passwords, keystore passwords and OAuth client secrets in the spec end up in more places than intended. Reference secrets through environment variables or a secrets mechanism your platform controls, and confirm how Druid's dynamic configuration handles sensitive values in your version. Restrict who can read and submit specs.
Plan rotation for Kafka credentials
Certificates, SCRAM passwords, Kerberos keytabs and OAuth client secrets all expire or get rotated, and a running supervisor keeps its tasks on the old values until they are restarted. Document who owns each credential, how the new value reaches every Middle Manager or Indexer, and how you verify that tasks reconnect. Test the rotation on a non-production supervisor first.
Common Druid Kafka ingestion failures and how to fix them
This table lists the failures most often seen when a Druid Kafka supervisor reads a topic, with the likely cause, what to check in supervisor status and task logs, and the fix for each. Use it as a first-pass triage guide, starting with the supervisor state before looking at individual tasks. Most of these failures trace back to connectivity, permissions, task capacity, malformed input or retention settings.
| Failure | Likely cause | What to check | Fix |
|---|---|---|---|
| Supervisor in UNHEALTHY state | The Overlord cannot get a healthy status from its Kafka indexing tasks, often after repeated task failures | Supervisor status on the Overlord, the most recent task errors, and the Overlord log for the supervisor ID | Resolve the underlying task error first, then suspend and resume the supervisor so it restarts its tasks |
| Tasks cannot connect to brokers | Wrong bootstrap servers, DNS or network rules blocking the broker port, or TLS settings missing from consumer properties | Task logs for connection or metadata timeouts, bootstrap servers in the supervisor spec, and reachability from the Middle Manager or Indexer host | Correct the bootstrap servers and consumer properties in the supervisor spec, and open the broker listener port from the Druid data servers |
| Authorization errors on topics | The principal used by Druid lacks read access on the topic or the consumer group, or the cluster authorizer denies the request | Task logs for topic or group authorization failures and the ACLs for the Druid principal on the topic and group | Grant the Druid principal read access on the topic and the consumer group it uses, then restart the affected tasks |
| Lag grows because tasks are undersized | Too few tasks or replicas for the partition count and message rate, or tasks that run too slowly to keep up | Consumer lag per partition in supervisor status, task count and partition assignment, and task run time and row throughput in task logs | Increase task count within the partition limit, review task duration and segment granularity settings, and size Middle Manager or Indexer slots to match |
| Schema or parse exceptions from malformed messages | Messages that do not match the configured input format or timestamp column, or a schema change in the producer | Parse exception entries in task reports, sample raw messages from the topic, and recent producer schema changes | Fix the input format or flatten settings, add a tolerant parse policy where the data allows it, and correct the producer or route bad records to a separate topic |
| Offsets out of range after retention expiry | The stored offset points to data that Kafka retention already deleted, usually after a long pause or a retention change | Supervisor offset state, the topic's retention settings, and the time since the supervisor last committed progress | Decide whether to skip to the earliest available offset or reset the supervisor, and record the data gap for analysts; confirm the reset behavior in the Druid documentation for your version |
| Segment handoff delays | Published segments take time to load on Historicals, or deep storage and Coordinator load rules slow the handoff | Coordinator status for segments awaiting load, Historical capacity, deep storage write errors in task logs, and load rules for the datasource | Check Historical disk and load queue capacity, verify deep storage credentials and permissions, and adjust replication or load rules so segments can load promptly |
Questions teams ask before loading Kafka into Druid
What drives the cost and effort of loading Kafka topics into Druid?
The main drivers are the number of topics and datasets, how complex each message schema is, how many fields must be flattened or transformed at ingestion, and how strict the query and freshness requirements are. Each supervisor needs a spec that matches its topic's partitioning, so every additional topic adds design, testing and review work. Cost also depends on who is paid for what: a fixed-scope engagement covers agreed deliverables, while internal staff time is spent on the same design choices. Ask any partner for a quote that itemizes topics, datasets, environments, load testing and handover, and separates one-time build work from recurring operations and upgrades.
How long does a Kafka to Druid ingestion project take?
Duration depends mainly on topic count, schema complexity and how many environments must be validated. Simple, well-documented JSON topics with stable fields move through design and testing faster than topics with nested or evolving schemas, which need more time for field mapping, timestamp handling and rollout checks. Load testing with realistic message rates and task sizing also takes time, because supervisor behavior under lag and failure is what determines production readiness. The engagement scope, deliverables and timeline are agreed in writing before work starts, so ask for the schedule to be tied to the topic inventory and to named acceptance tests.
What does our team need to provide before the work starts?
Three inputs shape almost every design decision. First, a topic inventory listing each topic, its partition count, message format, expected throughput, retention and owner. Second, schema samples or a schema registry reference for each topic, including how fields change over time and which timestamp field defines event time. Third, an access policy: which service principals may read each topic, how they authenticate, and where Druid's own credentials are stored. You should also document the queries the business needs, such as time range, grouping, freshness target and concurrency, because those requirements determine segment granularity, rollup and task sizing.
Can an internal team load Kafka into Druid without a partner?
Yes, an internal team can do this if it has engineers who understand Kafka consumer offsets, Druid ingestion specs and the Druid cluster layout. Druid reads Kafka through supervisors on the Overlord that manage indexing tasks, so the team must be comfortable operating those supervisors, reviewing task failures and planning capacity for Middle Managers or Indexers. The work is most at risk where knowledge is thin: exactly-once behavior depends on correct partition and offset handling, and upgrades must be planned against the supervisor spec. A partner is most useful when the team lacks this experience or needs a fixed-scope delivery with documented handover, testing and runbooks.
Do Druid or Kafka upgrades change the supervisor spec?
Upgrades can require spec changes, so each upgrade should be tested against your supervisors before production. Druid's Kafka ingestion requires Kafka 0.11.x or higher and the druid-kafka-indexing-service extension, so both versions must remain compatible with your chosen release. Kafka's own upgrade path matters too: Kafka 4.0 supports only KRaft mode, and rolling upgrades are supported from each release in the past year to the latest version. Check Druid's release notes for each target version, confirm extension and option names in the Kafka ingestion documentation, and keep supervisor specs in version control so changes are reviewed and reproducible.
How do we check that Druid ingests each Kafka record exactly once?
Druid reads by partition and offset, which is the basis for its exactly-once ingestion guarantee when the supervisor and tasks run as designed. Verification means comparing record counts and offset ranges in Kafka with the segments Druid publishes, and testing what happens during task failures, rebalances and restarts. Ask for test cases that kill tasks mid-run, change the number of tasks, and replay a known offset range, then reconcile counts after each one. The result should be a documented checklist that your data team can rerun after every upgrade or topic change.
What access controls should we plan for Druid reading from Kafka?
Plan access at three points: how Druid authenticates to the Kafka brokers, which principals can read each topic, and who can change supervisors in Druid. Kafka supports SSL and SASL authentication, with the documented SASL mechanisms GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Druid's supervisor API and web console should be restricted to the operators who manage ingestion, since changing a spec can alter what data is loaded and how long it is retained.
Testing the pipeline before production traffic
Before any production topic feeds Druid, run a test plan that proves the supervisor resumes from stored offsets, keeps up with peak rate and returns query results that match the source. Start by replaying representative topics from a staging copy or a retained window of production data into a non-production Druid cluster with the same druid-kafka-indexing-service extension and a supervisor spec close to the production one. Use a topic with realistic key distribution and message size, since skewed partitions hide lag problems that a uniform test topic never shows. Confirm the Kafka version in the test environment matches the one you will run in production, because the Druid documentation sets a minimum of Kafka 0.11.x and your results only apply to the client and broker versions you actually tested.
Next, drive the replay at the peak rate you expect and watch the supervisor status, the consumer lag per partition and the behavior of the indexing tasks. Record how long tasks take to publish segments, whether task count stays within the capacity of your Middle Managers or Indexers, and whether lag recovers after the peak ends. Then test failure behavior on purpose: kill individual indexing tasks, restart the Overlord and confirm that new tasks resume from the last committed partition offsets without gaps or duplicate rows in the query results. Repeat the restart while the replay is running, because a failure during active publishing is the case that matters most. Finally, rehearse a supervisor spec change, such as a new task count or a changed taskDuration, and check that the supervisor applies it, that offsets carry across and that queries stay correct throughout.
Treat the outcomes as acceptance evidence and write them down before production cutover. Keep the supervisor status and lag graphs for each test, the count of rows ingested against the count of messages produced for the same offset range, and query results compared with an independent count from the source system for a fixed time window. Also record the task failure and restart log entries, the time to recover lag after each injected failure and the Druid and Kafka versions used. Agree the pass criteria with the data owner in advance, including what difference in counts is acceptable and how long lag may take to recover, and confirm those thresholds against your own environment and business needs rather than adopting generic values.
How DeployOpen scopes and delivers Kafka to Druid ingestion
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including the Kafka and Druid ingestion path described on this page. Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
How we scope
- We inventory the Kafka topics, partition counts and message formats that Druid should ingest.
- We define the Druid datasources, timestamp and dimension mapping, and the query workloads they must serve.
- The written scope states which supervisors, task settings, Overlord and Middle Manager sizing, and retention decisions are included.
How we build
- We install the druid-kafka-indexing-service extension and confirm the Kafka client version against your cluster.
- We configure supervisors and task counts, then test offset tracking and restart behavior with controlled producer load.
- We validate deep storage, metadata store and ZooKeeper connectivity in your environment before any production topic is attached.
How we hand over
- You receive supervisor specifications, configuration files and a runbook for suspend, resume, reset and task failure handling.
- We document sizing assumptions, the offset and segment checks used during testing, and the known limits we found.
- Knowledge transfer sessions cover how your operators read supervisor status and task logs in the Druid web console.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Retainer work can cover Druid release reviews, supervisor changes as topics evolve and periodic ingestion lag checks.
- Upgrade planning covers Kafka and Druid versions together, so compatibility is checked before either side moves.
