Metering and chargeback on a shared Kafka cluster

The effort depends less on Kafka itself and more on how cleanly your teams map to topics, principals and quotas. Most of the work is in three places: deciding the allocation rules, collecting per-principal throughput and per-topic storage from your metrics stack, and turning both into a report that finance accepts. Expect the largest share to go into the mapping: which topic prefix or service account belongs to which cost center, what happens to shared topics, and how platform overhead such as brokers, controllers and monitoring is split. Effort rises with the number of teams, the number of principals per team, the amount of unowned or legacy topics, and how many metrics sources you must join. It falls when naming conventions and ownership metadata already exist. Ask any partner to break the estimate into discovery, metrics pipeline, allocation model, report build and handover, and to state which of those assume your existing naming and access model.

Yes, a platform team with metrics and data engineering skills can build showback in-house, and many do. The work is well bounded: export quota-related throughput metrics, measure topic log sizes per prefix, apply an allocation rule, and publish a monthly table. The risks are in the edges. Your team must agree on what counts as shared overhead, handle topics that were created outside the naming standard, and keep the pipeline working through broker upgrades that change metric names. Outside help is most useful when the allocation model needs review by finance, when several clusters must be combined, or when the team lacks capacity to own the report after launch. Whichever route you take, ask for the design written down before build, so the logic can be audited and handed over. DeployOpen plans, implements and documents self-hosted platforms in your own infrastructure, and scope is agreed in writing before work starts.

It needs two decisions from finance and one from the platform owner before any code is written. Finance must define the cost pool: which infrastructure lines, such as compute, storage, network transfer and support time, are allocated, and which budget codes receive them. Finance must also agree on the cost units, for example storage by gigabyte-month and throughput by gigabytes moved, and whether shared overhead is spread evenly or by usage. The platform owner must confirm that each team's principals, topic prefixes and service accounts map to a named owner, and must decide who approves exceptions. Without these decisions, the report will be technically correct but disputed. Plan a review cycle where finance checks a dry run against the prior month's invoices or allocation totals before the report becomes official. Ask who signs off on the model and how changes are versioned.

The time to a first report depends on the drivers, not on a fixed number. The main ones are how far your access model already matches team boundaries, whether per-principal quota metrics are already collected, how consistent topic naming is, and how many decisions finance needs to make. A shared cluster with enforced prefixes and named service accounts reaches a first report much faster than one where ownership has to be reconstructed from old tickets. Count separately the time to produce a manual first run, which validates the allocation logic, and the time to automate it on a schedule. Ask any partner for a plan that lists which dependencies sit with your teams, such as metrics access, finance sign-off and naming cleanup, so you can see where delays will come from.

Apache Kafka itself carries no licensing cost. It is released under the Apache License 2.0 (SPDX: Apache-2.0), and the project offers no paid edition. The cost you allocate comes from infrastructure and operations: compute for brokers and controllers, storage for retained data, network traffic between sites or zones, monitoring and logging, and the engineering time to run the platform. Some surrounding tools, such as a commercial schema registry, a managed connector product or a vendor support contract, may carry their own license fees, so include those as separate lines if you use them. When you build the cost pool, list each cost line with its source and whether it is one-time or recurring. Ask your infrastructure team which lines they can meter per tenant and which must be spread across all tenants as shared overhead.

Start with showback, which reports each team's consumption and estimated cost without moving money between budgets. This lets teams see their share, challenge the allocation rules and fix naming or ownership gaps while nothing is billed. A practical sequence is to pick one or two teams for a pilot, produce a monthly report for a few cycles, compare it with the actual infrastructure spend, and adjust the allocation model. Once finance accepts the numbers and the owners stop disputing them, move to chargeback in a planned change with an effective date. Keep the same report format through the transition so that the only difference is whether the amount is transferred. Document the rules in a short model description that names the cost pool, the units, the owners and the review cadence, so that a later audit or budget review can reproduce any month.