Data contracts for Kafka topics

Data contracts for Kafka topics start with agreeing what producers may publish

A data contract for a Kafka topic is the written agreement between the team that produces events and the teams that consume them about what a message looks like, how it may change, and who answers when it does not match. It is worth building once several consumers depend on a shared topic and schema edits have started breaking them, usually without warning. Apache Kafka does not enforce a schema on its own. The broker stores and transports bytes, so the contract has to be built from a schema registry, compatibility settings on each subject, ownership metadata and checks that run in CI before a change reaches a shared environment.

Kafka itself is released under the Apache License 2.0 (SPDX: Apache-2.0), and the project offers no paid edition, so the broker adds no license fee to a contract program. The costs sit elsewhere: the registry you choose, the engineering time to write and review contracts, and the CI pipeline that enforces them. Confirm the license terms of the registry and serializer libraries separately, because they are distributed by different projects and are not covered by the broker license.

Before writing any rules, list the topics that more than one team reads and, for each, name the owning team, the producers allowed to publish, and the consumers known today. Then agree what a breaking change means for that topic, such as removing a field, changing a type or renaming a key. Changes that break consumers should move to a new topic instead of changing the existing one silently, which keeps old consumers running while new ones migrate. Confirm with your consumer teams which version of the contract they deploy against, since that decides how far back a compatibility rule must reach.

Which compatibility rule fits which contract

When the team that owns a producer also owns its consumers, and both sides can deploy in one coordinated release, the compatibility question is mostly about sequencing. The direction of the rule matters less than knowing which side moves first and whether the other side can tolerate that change while the rollout is in progress.

Backward direction fits consumer-first rollouts: a new schema version can be read by consumers still on the previous version, so consumers can be updated before the producer begins writing the new shape.
Forward direction fits producer-first rollouts: old consumers can read records written with the new schema, so the producer can ship first and consumers catch up afterward.
Full direction removes the ordering decision: when either side may deploy first, both must tolerate the other, which narrows the set of changes you can make in place.
Deploy order belongs in the runbook: write the order into both the release plan and the rollback plan, because a reversed order can leave a running consumer unable to decode new records.
A short overlap window lowers the stakes: a single coordinated release limits how long two schema versions coexist, so the choice of transitive checks matters less here.

Confirm that every consumer group on the topic is listed, that the team can deploy both sides in the agreed order, and review the sibling page on compatibility modes for Kafka topics for the full rule set.

Contract elements and when each one fits

This table compares the elements a team can add to a Kafka topic contract, from schema format to the route for breaking changes. Read each row as a decision: when the element earns its keep and what it asks your team to maintain afterward. The main takeaway is that the cheapest elements to start with are usually ownership and a CI gate, while compatibility modes and breaking-change routes carry the ongoing cost.

Contract element	When it fits	Cost to maintain
Schema format (Avro, Protobuf or JSON Schema)	Choose one format per domain when producers and consumers share types across languages; mixing formats raises the number of serializers your teams must run.	One-time choice, then recurring tooling upkeep per language. Format trade-offs are covered on the sibling format page.
Compatibility mode per subject	Use backward or full modes when consumers deploy after producers and may lag several versions; use transitive modes when consumers skip versions.	Low to set, but each rule blocks some changes, so teams must agree on exceptions and deploy order.
Named owner and consumer registry	Needed as soon as more than one team reads a topic, so that a breaking change has a known list of people to notify.	Recurring: owners change roles and the consumer list goes stale unless someone reviews it on a schedule.
Required metadata fields	Add fields such as owner, classification, retention intent and PII flags when catalog or access decisions depend on them.	Recurring: each field needs a value, a validator and a reason to stay populated.
CI gate on schema changes	Fits teams that merge schema changes through pull requests and want incompatible changes rejected before deployment.	Moderate to build; ongoing work to keep test fixtures and registry access current for every pipeline.
Breaking-change route (new topic with a version suffix)	Use when a change cannot stay compatible, such as a removed required field or a changed key meaning; the old and new topics run side by side for a period.	Recurring duplication of storage, producers and consumer migration work, plus a retirement plan for the old topic.
Deprecation and retirement policy	Needed when old topics or schema versions must be removed on a known schedule for storage, access or audit reasons.	Recurring review effort; the policy is cheap to write but needs someone to enforce it.

How to set up contracts on an existing topic set

This sequence moves an existing Kafka topic set from informal agreements to enforced data contracts, one owned topic at a time. It ends with a pilot domain where producers, consumers and the breaking-change procedure have been tested in CI and in production-like traffic.

Inventory topics, producers and consumers
Platform engineering pulls the topic list from the cluster and maps each topic to the applications that publish to it and the groups that consume it. Use the broker's admin tooling and your client configuration inventory, then reconcile the result with the teams who own those services, since consumer groups often outlive the applications that created them. The step is done when every topic has a recorded producer list, a consumer list and a flag for topics with no known owner. Unowned topics are the ones that will cause the most trouble later, so mark them clearly rather than guessing.
Assign a named owner to each topic
For each topic, the owning domain team names one accountable person or team, not a shared mailbox, who approves schema changes and answers for consumer impact. The platform team should not own business payloads by default. Record the owner in the inventory and confirm that each owner has accepted the role in writing. The check is simple: pick any topic at random and you should be able to name who approves a change to its schema within a minute.
Choose the registry and compatibility rule per subject
Decide which schema registry will hold the contracts and which serialization formats teams may use, then set a compatibility rule for each subject. Backward or full compatibility modes are common starting points for consumers that read older records, and transitive variants suit consumers that lag several versions. Confirm the registry's API compatibility with the serializers your teams already run, and confirm the current license terms with the vendor or project before you commit. The output is a per-subject table of format, registry and compatibility mode, reviewed by the owners from the previous step.
Store ownership metadata in version control
Write each contract as a file in a Git repository: the schema, the compatibility mode, the owner, the consumers that depend on it and the retention and classification of the data. Changes to these files go through the same review as code. Registry entries are then generated or synchronized from the repository, so the repository, not the registry UI, is the record of what was agreed. Done when a fresh clone of the repository describes every topic in the inventory and a change to an owner field shows up in a pull request.
Add CI validation for schema changes
Add a pipeline step that runs on every pull request touching a contract. It checks that the schema parses, that the proposed version passes the registry's compatibility test against the latest registered version, and that the owner approval is present. The registry's compatibility check endpoint can be called from CI, so the rule you chose in the earlier step is enforced rather than remembered. The check to run is a deliberate failure: submit a field removal or a type change that breaks the chosen mode, and confirm the pipeline fails and names the subject. Then submit a compatible change and confirm it passes.
Define the breaking-change procedure
Write down what happens when a change cannot be made compatible. The owner creates a new topic with a new name or version suffix, producers publish to both topics during a migration window agreed with the consumers, and each consumer group moves over and confirms it has finished reading the old topic. The old topic is retired only after the owners sign off. Agree the length of the window with the consumers, since no supplied fact sets one. Test the procedure once on a non-production topic and record the steps, including how offsets are checked and who signs off the retirement.
Pilot the contracts on one domain
Choose one domain with a cooperative owner, a moderate number of topics and at least one consumer that is not controlled by the same team. Apply every previous step to that domain only, then run a real schema change through the full path, including one deliberate breaking change that goes through the new-topic procedure. Collect the friction points from producer and consumer teams, adjust the templates and CI checks, and then decide whether to extend the pattern to the next domain. The pilot is complete when the team can show a passing compatible change, a failing breaking change and a finished migration with recorded sign-off.

Handling breaking changes with new topics instead of silent edits

A breaking schema change is a new contract, so it needs a new topic rather than an edit to the old one. The cards below cover the parallel work that makes that move safe: naming, dual publishing, consumer cutover, retirement of the old topic and communication to every owner involved.

Name new topics with an explicit version

Give the replacement topic a version suffix or generation marker in its name, such as orders.v2, and record in the contract metadata which version supersedes which. The name should make the break visible in dashboards, ACL rules and consumer configuration. Check that your naming convention and any topic creation policy allow the new name before producers are changed, and that the old and new topics can be owned and monitored separately.

Publish to both topics during migration

For a defined window, producers write each event to the old topic and the new topic, so consumers can move at their own pace. The dual publish window must have a stated end date agreed by the producer and every registered consumer. Confirm that the producer can handle partial failure, for example a successful write to one topic and a failed write to the other, and decide which topic is authoritative when the two disagree.

Track each consumer through cutover

Each consumer group should move to the new topic on its own schedule, and the platform team should be able to see which groups still read the old topic. Use consumer group lag and offset data on both topics to confirm that a group has caught up and is no longer reading. Treat a group that stays on the old topic after the deadline as an open issue for its owner, not as a reason to extend the window indefinitely.

Retire the old topic against retention

Before deleting the old topic, confirm that no consumer, replay job or audit process still needs its history, and that any data that must be kept has been archived elsewhere. Retention settings decide how long old records remain readable, so ask each owner whether they rely on replay from the old topic. Remove ACLs and connector configurations that reference the old topic only after the deletion is complete and verified.

Tell owners of every change

Announce the new topic, the dual publish window, the cutover deadline and the retirement date to producer and consumer owners in writing, and record the same dates in the data contract. Silent edits break consumers that deserialize or filter on fields they expect, often without an error that points to the cause. Ask each owner to confirm receipt, name the person responsible for their cutover, and report any blockers before the window closes.

Questions teams ask before hiring help with data contracts

What drives the cost of rolling out data contracts across many teams?

The main driver is how many producers and topics need contracts, and how many of them already publish data that consumers depend on. Each topic needs a schema, a compatibility rule, an owner and a review step, so effort grows with the number of teams that must agree on terms. Other drivers are the state of existing schemas (undocumented payloads take longer to describe), the serialization formats in use, and whether CI pipelines already build and test producer code. One-time costs cover the design of the contract model, the CI checks and the first pass over existing topics. Recurring costs are the review workload and maintenance of the registry and checks. Ask for a written scope that lists the topics in the first wave, the formats, the CI systems involved and the teams that will review changes.

How long does a data contracts pilot take?

A pilot's length depends mainly on the number of topics, the number of producing teams and the tooling already in place. A pilot on a handful of topics with one schema registry and existing CI is far shorter than one that spans several teams, mixed formats and new pipeline stages. Confirm the timeline in the written scope rather than relying on a general figure. Ask for the agreed topic list, the acceptance criteria for the pilot, the milestones for owner sign-off and the point at which CI enforcement becomes blocking. A good scope also states what happens if a team is not ready to name an owner, since that is often the constraint, not the code.

What does the client need to provide for a data contracts engagement?

The work depends on access and named people more than on any single tool. Teams typically need to provide:

A named business owner and a technical owner for each topic in scope
Read access to existing schemas, producer code and sample messages
Access to the CI system where schema checks will run, with permission to add pipeline steps
A running schema registry or agreement on where schemas will be stored
Time from consumer teams to review proposed compatibility rules

If any of these are missing, the schedule usually slips. Ask the provider to list which items each party supplies before work starts.

Can an internal platform team set up data contracts without outside help?

Yes, many internal teams can, provided they have engineers who know their Kafka cluster, their serialization formats and their CI system, and who can secure time from producing and consuming teams. The harder parts are rarely technical. They include getting producers to accept compatibility rules, agreeing on what counts as a breaking change, and keeping ownership metadata current after the first rollout. An outside partner may help when several teams need to be brought along quickly, when the team lacks experience with registry compatibility modes, or when the platform team needs an independent review of its design. Compare the internal effort with a written scope before deciding.

How do data contracts relate to compliance evidence?

Data contracts do not make a system compliant by themselves, but they can produce evidence that compliance assessments ask for, such as documented data owners, controlled change review and a record of how schema changes were approved. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies make those determinations. Ask the provider which controls the contract process supports, which records it keeps (for example, review history and owner records in version control), and which parts remain the responsibility of your compliance team.

What should a written data contracts scope include?

A useful written scope names the topics covered, the formats and schema registry in use, the compatibility mode chosen for each subject, and the CI checks that will run on each change. It should list the ownership metadata captured for each topic, the process for handling a breaking change (usually a new topic with a migration period rather than a silent edit), and the deliverables such as the contract template, review checklist and runbook. It should also state who approves each stage, what is excluded, and how changes to scope are priced and scheduled. Ask for these items in writing before comparing quotes.

What goes wrong most often when teams adopt data contracts?

The most common failures are predictable. Teams add contracts without a named owner, so nobody approves changes. Compatibility rules are set without checking the deploy order of producers and consumers, which breaks consumers that lag several versions. CI checks run but are not blocking, so breaking changes still reach production. Existing topics are left out of the first wave, creating two standards. Ask the provider how it will test compatibility rules against real consumer upgrade patterns and how it will handle exceptions for legacy topics before the first production change.

Which Kafka version should data contracts work against?

The contract process itself is largely independent of the Kafka broker version, but the cluster and client libraries you run must be supported. The Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as the supported releases, and Kafka 4.0 supports only KRaft mode. Confirm which release your cluster runs and which client and serializer versions your producers and consumers use before designing compatibility checks. Ask the provider to record these versions in the scope so the contract tests match what runs in production.

What changes for producers and consumers once contracts are in place

For producer teams, the most visible change is that a schema change now goes through review before it reaches a topic. A producer that wants to add a field, rename one or change a type opens a change against the contract, and the CI checks run the compatibility rule set for that subject before anything merges. Producers should expect this review to take longer than an edit that ships on its own, especially for changes that touch several consumers. The work moves rather than disappears: the time once spent on after-the-fact fixes and rollbacks shifts to writing the change, documenting its effect on consumers and waiting for the owning team's sign-off. Producer leads should agree in advance which changes are routine and which need a consumer conversation, so that the review queue does not become the bottleneck.

For consumer teams, the contract turns an unannounced breaking change into a visible event. Consumers can rely on the compatibility mode and the owner metadata to know who to contact, and they can plan upgrades on their own schedule instead of discovering a problem in production. The cost is that consumers now have to read contract changes, test against new versions and sometimes move to a successor topic when a breaking change is published as a new topic rather than an in-place edit. Consumer leads should confirm how long they need to adopt a new version or topic and record that window in the ownership metadata. Teams that lag several versions behind need the transitive compatibility modes described elsewhere on this page, so their workload is planned, not surprise rework.

For platform owners, the change is in where the hard questions land. When a topic misbehaves, the owner field and the registry history tell the on-call engineer which team published the schema version in question and which version was live at the time, so the investigation starts with a named owner rather than a search across channels. Platform owners carry the rules themselves: they maintain the CI checks, the default compatibility settings and the process for creating successor topics, and they must keep the review path fast enough that teams keep using it. Ask whether the registry and CI tooling can report which producers and consumers are bound to each subject, because that report decides how quickly a breaking change can be scoped. The platform team's workload increases in governance and tooling and falls in incident triage.

How DeployOpen scopes and delivers data contracts for Kafka

DeployOpen treats each Kafka data contract as an agreement between the producing and consuming teams, written down before any topic changes. We scope the agreement, build the enforcement in your own CI and cluster, and hand over the rules, ownership records and runbooks your teams will use after the engagement ends.

How we scope

Inventory the topics and producers in scope and name the owning team for each one.
Agree the serialization format and compatibility mode per subject with producer and consumer leads.
Record the scope, deliverables and timeline in a written, fixed-scope agreement before work starts.

How we build

Define the contract template covering schema, ownership, classification and breaking-change process.
Configure schema compatibility checks and review steps in your existing CI pipeline.
Test producer and consumer behavior against proposed changes in a non-production Kafka cluster you control.

How we hand over

Deliver the contract repository, review workflow, and ownership metadata in formats your teams can maintain.
Document the process for introducing a new topic when a change would break existing consumers.
Walk platform, security and data teams through the runbooks and evidence they will need to review changes.

How we support

Ongoing support, rule updates and periodic reviews of contracts are available as a retainer.
Confirm with your own teams which compatibility rules and registry features your chosen versions provide.
Ask for a written quote that lists each deliverable, the responsible party, and any recurring work.

Data contracts for Kafka topics