Choosing schema compatibility modes for Kafka topics
Start each subject with backward compatibility and choose the mode per subject, not per cluster, because each topic has its own producers, its own consumers and its own tolerance for change. A single cluster often carries topics whose consumers upgrade on very different schedules, so one global rule either blocks harmless changes on a fast-moving topic or lets breaking changes through on a slow one. Set the rule on the subject, record the reason next to it, and revisit it when the set of consumers changes.
Teams need this decision once producers and consumers ship on different release trains. A producer team may publish a new field on Monday while a consumer service running an older build still reads the topic, or a long-lived consumer may lag several schema versions behind the latest. The registry checks each new schema against the previous versions under the subject's mode and rejects changes that fail. Compatibility modes differ in which direction they protect: backward modes protect consumers that read new data with old schemas, forward modes protect consumers that read new data with old schemas in reverse, and full modes require both. Transitive variants apply the check against every prior version rather than only the latest, which matters when consumers lag several versions. Confirm in your registry's documentation exactly which mode names and transitive behavior it implements, because the names are not standardized across products.
The registry enforces the rule but does not sequence your deployments. A backward-compatible change still breaks a consumer if the team deploys the producer before the consumer has been prepared for the new field, so each subject needs a written deploy order: consumers first for backward changes, producers first for forward changes, and a documented path for changes that need both. Ask each team which consumers read the subject, how far behind they can run, and who owns the rollout. Those answers decide whether a transitive mode is needed and whether a breaking change belongs in a new topic instead.
Which compatibility mode fits which consumer pattern
This table compares backward, forward and full compatibility, with their transitive variants, by the changes each mode accepts, which side deploys first and the consumer pattern it suits. Read each row against your deploy order and how far behind consumers may fall. The main takeaway: transitive modes check a new schema against every earlier version, which protects consumers that lag several versions.
| Compatibility mode | Changes accepted | Upgrade first | Best fit |
|---|---|---|---|
| Backward | Deleting fields and adding optional fields with defaults | Consumers | Consumers are updated before producers and only the latest version matters |
| Backward transitive | Same as backward, checked against every earlier version | Consumers | Consumers may read records written under any earlier schema, such as replayed topics |
| Forward | Adding fields and deleting optional fields | Producers | Producers ship first and existing consumers keep running until they are updated |
| Forward transitive | Same as forward, checked against every earlier version | Producers | Old consumers may still read records written under any earlier schema |
| Full | Only changes valid in both directions, such as optional fields with defaults | Either side | Producers and consumers deploy independently and only the latest version is checked |
| Full transitive | Same as full, checked against every earlier version | Either side | Independent deploys on topics with long retention or many active schema versions |
| Consumers lagging several versions | Backward transitive or full transitive | Consumers or either side | Consumers on an old version can still read records from any version, so a change cannot break a version two releases back |
| Producers lagging several versions | Forward transitive or full transitive | Producers or either side | Older producer builds can still write records that current consumers read |
| None | Any change, with no compatibility check | Coordinated cutover | Only for a planned, coordinated switch where every client moves together or on a new topic |
Setting compatibility per subject step by step
This sequence takes a schema registry from an unmanaged set of subjects to a documented compatibility mode for each one, with each mode tested against a compatible and an incompatible change before it reaches production. The result is a decision record that producer and consumer teams can point to when a schema changes.
Inventory subjects and name their owners
List every subject in the non-production registry, which usually maps to a topic's key or value schema, and record the producing team, the consuming teams and one accountable owner for each. Ask the platform team to export the list from the registry rather than relying on memory, since subjects created by old services are often the ones nobody remembers. The step is done when every subject has an owner and a named consumer list, and any orphaned subject is flagged for review before its mode is chosen.
Map consumer upgrade lag per subject
For each subject, ask the consuming teams how many schema versions they can be behind when they deploy, how often they redeploy, and whether any consumer runs from a long-lived batch job or an archived reprocessing path. Consumers that lag several versions need a transitive mode, because a non-transitive check only compares a new version with the one before it. The output is a short note per subject stating the maximum observed lag, which feeds the mode choice in the next step.
Choose a mode for each subject
Pick backward, forward or full, and decide whether each one should be transitive, using the lag notes and the direction of the risk. Backward modes protect consumers that upgrade before producers, forward modes protect consumers that upgrade after producers, and full modes protect both at the cost of tighter changes. Record the chosen mode, the reason and the owner in a table with one row per subject. Confirm the exact mode names and their behavior against the registry's own documentation before you write them down.
Apply the mode to each subject in non-production
Set the compatibility mode on each subject in a non-production registry, one subject at a time, and read the setting back from the registry to confirm it was stored. Do not rely on a global default to cover subjects you have not reviewed, because a default can silently apply to a subject that needs a stricter rule. The step closes when a read-back for every subject matches the decision table, and any mismatch has been corrected and logged.
Test a compatible change against each subject
Register a new schema version that adds an optional field with a default, or otherwise follows the rules of the chosen mode, and confirm the registry accepts it. Run this against a sample producer and a sample consumer of the previous version to confirm that records serialized with the new schema still deserialize. The check is passed when the registry returns a successful registration and the sample consumer reads the new records without errors.
Test an incompatible change is refused
Attempt to register a change the mode should reject, such as removing a required field or changing a field type in a way the rules do not allow, and confirm the registry refuses it with a compatibility error. This is the check that proves the mode is doing its job, so run it for every subject, not just one representative. If a rejected change is accepted, stop, fix the mode or the subject setting, and repeat the compatible and incompatible tests before continuing.
Record the decision and the test evidence
Write down, for each subject, the chosen mode, the owner, the observed consumer lag, the date of the tests, and the results of the compatible and incompatible change checks. Store the record in the same repository as the platform configuration so changes to modes go through review. The step is done when each subject's row links to its test results and a second person has reviewed the table, which becomes the input for the production rollout and for later data contract reviews.
Deploy order for producers and consumers under each mode
Under BACKWARD, a new schema version can read data written with the previous version. Consumers therefore move first. Once every consumer understands the new schema, producers can start writing records that use it. The order protects consumers because they never receive a record they cannot decode.
- Upgrade order: Deploy consumer code that reads the new schema before any producer emits records with it.
- Producer sequencing: Hold producer rollouts until consumer version reports show the fleet has moved.
- Producer rollback: Rolling producers back is safe for consumers already upgraded, because they can still read the older records.
- Consumer rollback: Rolling consumers back after producers have started writing the new schema can leave them unable to read new records, so treat it as a stop-and-fix event.
Before each promotion, check that every consumer group reports the new schema version and that no producer has been moved ahead of the last consumer.
Under FORWARD, data written with a new schema version can be read by consumers using the previous version. Producers therefore move first. Consumers keep working through the rollout, so consumer teams can schedule their upgrades on their own timeline without a coordinated freeze.
- Upgrade order: Roll out producers that emit the new schema, then upgrade consumers at their own pace.
- Consumer window: Consumers still on the old code keep reading new records, which gives each team time to test.
- Producer rollback: Safe while consumers still run the previous code, because they read records from the new schema by design.
- Mixed-version reads: Consumers already upgraded may still meet records written by an older producer during a rollback, so confirm they decode both formats before relying on forward compatibility alone.
Before promoting producers, confirm which consumer groups are still on the previous version and that their owners have a dated upgrade plan.
Under FULL, a new schema version is compatible in both directions with the version before it. Either producers or consumers may move first, because each side can read data written by the other across one version step. This removes the strict ordering that BACKWARD and FORWARD require, at the cost of a narrower compatibility guarantee.
- Either order: Producers or consumers may move first, provided each change is compatible with the immediately previous version.
- Multi-step changes: A change that skips a version or stacks several edits needs an agreed order, so write that order into the release plan.
- Rollback: Rolling back one version step is covered in either direction, but a rollback across several steps falls outside the rule and needs its own check.
- Lagging consumers: A check against the latest version does not prove that a consumer two versions behind can read current records.
Before each release, confirm the change passes the full check against the latest registered version and record which side moved first.
The transitive variants, BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE and FULL_TRANSITIVE, check each new version against all retained versions instead of only the latest. The deploy direction stays the same as the base mode, but the rule no longer depends on the consumer fleet moving in lockstep. Any consumer version still present in the registry history can read data produced under the new schema.
- Direction: Backward variants keep consumers first, and forward variants keep producers first.
- Lagging consumers: Use BACKWARD_TRANSITIVE when some consumers may run several versions behind, because each new schema must read every retained version.
- Rollback window: Rollback to any retained version is covered, but only while those versions stay registered.
- Retention settings: Confirm with your registry vendor or documentation how deleted or soft-deleted versions are treated, because the guarantee applies to versions the registry still evaluates.
Before relying on a transitive mode, confirm which versions remain registered and how many consumer versions are running today.
Governance and access risks around schema changes
A compatibility setting is only as strong as the people allowed to change it and the access paths that reach the registry. The cards below cover the decisions that connect rule changes, review, and Kafka access, so a weakened rule does not reach production unnoticed.
Who may change a compatibility mode
Changing a subject's compatibility mode is a platform decision, not a developer convenience. Decide which role can set or relax a mode, and keep that role separate from the teams that publish schemas to the subject. Illustrative case: a producer team sets NONE to clear a failing build, and consumers on older versions start failing a week later. Ask who holds registry admin rights today and whether that list matches your change policy.
Review compatibility changes like code
Put every compatibility change through the same review path as schema changes: a pull request or change ticket that names the subject, the old and new mode, the reason, and the consumers affected. Automated checks should flag any loosening of a mode before a human approves it. Confirm that your registry or CI tooling records who made each change and when, so the history can be audited later.
Silent weakening of a rule
The most common governance failure is a quiet move from BACKWARD_TRANSITIVE to BACKWARD or to NONE, often made to unblock one release. The change looks harmless in the registry UI but removes the guarantee that lagging consumers can read new data. Schedule periodic exports of subject settings and compare them with the approved baseline, so drift is detected rather than discovered during an incident.
Kafka ACLs in KRaft
In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Confirm that this setting is false in your clusters and that producers and consumers of schema-bound topics have explicit grants. Review which principals can write to the topics that carry schema IDs, since they determine what gets published.
Authentication for Kafka clients
Kafka supports SSL and several SASL mechanisms, including SCRAM-SHA-512 and OAUTHBEARER, and security is optional, so a cluster can accept unauthenticated listeners beside secured ones. Identify each principal that registers or reads schemas, and make sure no service runs on an unsecured listener. Tie each service account to one owner so a compromised credential can be revoked without affecting other teams.
Registry access is a separate control
Kafka ACLs protect topics, not the schema registry's own API. Whether the registry enforces its own authentication, role-based authorization, and per-subject permissions must be confirmed for the product you run, and it is not implied by Kafka's configuration. Test that a non-admin identity cannot change a subject's compatibility mode, and record the result as evidence before go-live.
Questions teams ask before changing compatibility rules
Does one compatibility mode fit every Kafka topic?
No. A single mode rarely fits all subjects because each topic has a different set of producers and consumers with different upgrade speeds. A topic read by one internal service that the same team deploys can usually tolerate a stricter rule than a topic read by many independent teams or external partners. Choose the mode per subject, record the reason next to the subject, and review it when the set of consumers changes. Confirm the modes your chosen registry supports, since the names and transitive behavior can differ between implementations.
How far behind can a lagging consumer fall before schema changes break it?
It depends on the compatibility mode. Non-transitive modes check each new version only against the version immediately before it, so a consumer several versions behind may read data written under a schema it never saw and can fail. Transitive modes check each new version against all earlier versions, which is why they suit consumers that lag several versions. Your practical limit is the set of versions you have agreed to keep compatible, so write that limit down, test a consumer pinned to the oldest supported version, and confirm the exact behavior with your registry's documentation before relying on it.
Does a breaking schema change always need a new topic?
Not always, but it is the safer choice when a change cannot be made compatible under the agreed rules. A new topic lets old and new consumers run side by side, and it avoids changing the meaning of records already stored in the old topic. Adding a field with a default value or removing an optional field may stay compatible in the same subject. Renaming a field, changing a type or changing the meaning of a value usually does not. The cost is that every consumer must move to the new topic on its own schedule, and the team must plan how the old topic is retired.
What does the team need to own before changing compatibility rules?
The team needs a named owner for each subject, a written list of consumers, and an agreed policy for how far back consumers may lag. It also needs a place where schema changes are reviewed, usually a pull request check that runs the registry's compatibility test before a new version is registered. Someone must also own the deploy order, so producers and consumers are updated in the sequence the mode allows. Without these, compatibility rules are set once and then ignored, which is the most common cause of silent breakage.
Can our own engineers change compatibility modes without an outside partner?
Often yes, if the team already operates Kafka and the registry and can run tests against a staging cluster. Changing a mode is a configuration action, but the real work is the inventory of consumers, the test plan for old versions, and the coordination across teams. Teams that lack a clear owner for shared topics, or that cannot reproduce production consumer versions in staging, usually benefit from outside help to set the policy and test harness. DeployOpen plans and implements this work in your infrastructure under a written, fixed scope if you want that help.
What makes this work take more or less effort?
The main drivers are the number of subjects, the number of independent consumer teams, and how many consumers run versions older than the current release. Effort also rises when producers and consumers ship from different repositories, when the registry is shared across many teams, or when there is no automated compatibility check. Effort falls when consumers are inventoried, schemas are in version control, and deploy order is already documented. Cost follows the same drivers: internal staff time for an in-house effort, or the written scope and fees for a partner engagement. Ask any quote to list the subjects, consumer teams and test environments it covers, and to state which parts are one-time work and which are ongoing review.
Do Kafka platform releases change how compatibility rules behave?
Kafka itself does not define schema compatibility, so the rules come from the schema registry you run, not from the broker version. Upgrading Kafka does not change your subject settings, but it does change which clients and tools you must run. The Kafka upgrade notes list Java requirements that apply to clients and Kafka Streams applications, and those upgrades can coincide with a schema change. Plan the two in separate change windows and confirm the registry version you run is supported alongside your target Kafka release.
Limits to confirm in your own registry and environment
Compatibility modes only protect consumers if the registry enforces them the way you expect. The cards below cover the registry rules, format behavior, version history and runtime versions that determine whether a chosen mode holds in your environment.
Registry rule support per mode
Not every registry implements every compatibility mode, and some expose transitive variants under different names or only through their own API. Confirm that the registry you run accepts BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE and FULL_TRANSITIVE per subject, and that the setting can be changed per subject as well as globally. Test each mode with a deliberately breaking schema in a non-production subject before you rely on it.
Format-specific compatibility behavior
Compatibility checks depend on the schema format. Avro, Protobuf and JSON Schema each define different rules for defaults, optional fields, field numbers and unknown properties, so the same mode can accept a change in one format and reject it in another. Run your real change set through the registry for every format your teams use, and record which checks are enforced by the registry and which depend on review.
Retained version history
A transitive check compares a new schema with every version the registry still holds for that subject, not only the latest one. If your registry prunes old versions, deletes soft-deleted versions or has a retention policy, the check covers less history than the mode name suggests. Ask how many versions are kept, whether deleted versions still count, and whether the history survives a registry backup and restore.
Kafka version in use
Compatibility rules are a registry feature, but the Kafka brokers hold the topics your consumers read, and the release line you run sets the upgrade path. Confirm which Kafka release your cluster runs and whether it is on the supported list. Rolling upgrades are supported from each release in the past year to the latest version, so an old cluster may need intermediate steps before schema tooling changes.
Java runtime requirements
Brokers, Kafka Connect and tools require Java 17 from Kafka 4.0 onward, while clients and Kafka Streams applications need at least Java 11. Serializer libraries and registry clients bundled into producers and consumers inherit that floor. Inventory the JVM version of every service that serializes or deserializes records, because a consumer stuck on an older runtime can block the upgrade of the rest of the chain.
Registry supported version
Schema registry projects publish their own release lines, and the compatibility API and serializer behavior change between them. Confirm the exact registry version you plan to deploy, its supported status and its client libraries before writing deploy-order runbooks. Ask the vendor or project for its release policy and the security patch window for the version you select.
How DeployOpen scopes compatibility rules for your teams
Most engagements on compatibility rules are fixed-scope, so the subject list, owners and modes are agreed in writing before any registry setting changes. In practice that means a single table that names each subject (usually <topic>-key and <topic>-value under the default TopicNameStrategy), the team that owns its schema, the mode chosen for it (BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE, FULL_TRANSITIVE or a non-transitive equivalent), and the deploy order that goes with it. Owners decide which consumers must read old data, which producers must write data that old readers accept, and which subjects are allowed to break compatibility at all. Those decisions are the expensive part of the work, and they are cheaper to settle in a workshop than after a producer has already shipped a field rename.
Implementation then covers the registry configuration itself, applied per subject and, where your policy requires it, at the global default. DeployOpen wires the rules into the registry you run, adds a CI check that calls the registry's compatibility endpoint before a schema merges, and writes negative tests with schemas that should fail, such as a removed required field or a changed type. Each test is run against the registry version you deploy, because the compatibility API and the rule names can differ between open-source registries. Confirm which registry and version your environment uses before scoping, since that choice sets the exact configuration keys and endpoints. The handover includes a runbook for changing a mode, a record of who approved each exception, and the deploy-order checklist for each subject.
Ongoing support is available as a retainer for teams that add subjects, change owners, or need periodic review as consumers move between versions. A retainer covers reviewing new subjects against the agreed table and keeping the CI checks working after registry upgrades. Ask any partner, including DeployOpen, to put the subject list, the test cases, the handover items and the support boundaries into the written scope, so you can judge the work against what was agreed rather than against a general description of the tool.
How DeployOpen works on schema compatibility for Kafka
DeployOpen plans, builds, documents and supports self-hosted open-source platforms inside your own infrastructure, including the schema rules that govern Kafka topics. We start from the producers and consumers that depend on each subject, then write the compatibility decisions down before any registry setting changes.
How we scope
- Inventory every subject, its producers, its consumers and the consumer release lag your teams actually run
- Agree a compatibility mode per subject in a written scope, with the reasons recorded next to each choice
- Define which subjects stay on transitive modes and which may use non-transitive rules, with owners named
- Confirm the registry, serialization formats and authentication model that the rules must be enforced through
How we build
- Configure subject-level and global compatibility settings in the registry you run, then verify them against test schemas
- Build CI checks that run compatibility tests on each proposed schema change before it reaches the registry
- Write deploy-order runbooks for producers and consumers, covering rollouts that add optional fields and rollouts that remove them
- Test the rules against sample evolution scenarios, including a consumer several versions behind the producer
How we hand over
- Deliver a documented compatibility matrix listing each subject, its mode, its owner and its approved change types
- Provide the CI configuration, registry settings export and runbooks in your repositories, with a walkthrough for your teams
- Record the decision log so future changes to a mode go through the same review as the original choice
- Document which checks remain manual, so your reviewers know what the tooling does not cover
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts
- Ongoing support, registry upgrades and continuous review of new subjects are available as a retainer
- Revisit compatibility modes when Kafka or registry versions change, using the release notes your team confirms
- Confirm support terms, response expectations and handover ownership with DeployOpen in the written scope
