Avro vs Protobuf vs JSON Schema for Kafka

Avro vs Protobuf vs JSON Schema for Kafka messages: which format fits your topics

When several teams publish to one shared, self-hosted Kafka cluster without an agreed format, the format decision sets how schemas can change over time, how large each message is on the wire, and which schema registries and client languages you can support. Kafka itself does not inspect or enforce a payload format. Brokers store and move bytes, so the choice of Avro, Protobuf or JSON Schema lives in the serializers your producers and consumers use and in the schema registry that stores and checks the schemas. The cluster's security and authorization are configured separately, through the controls in the Kafka security overview, so the format choice does not change who may publish or read a topic. Kafka itself is released under the Apache License 2.0 and the supported 4.x releases are listed on the Apache Kafka downloads page, but neither fact tells you which format to use.

You need this decision now in three situations: a new platform where no team has published yet, producers in several programming languages that must agree on one schema language, or a registry migration in which schema IDs and serializer settings must carry over unchanged. In each case, confirm with your engineering leads which languages each team uses, which registry your platform will run (Apicurio Registry, Karapace or Confluent Schema Registry), and whether any consumers are in systems that cannot upgrade quickly. Check that the candidate serializers and the chosen registry both support the format and the compatibility modes you plan to enforce. Verify these support statements against each project's current documentation, since registry feature coverage changes between releases.

The decision can wait when you have one team, one language and a short list of topics, provided you still record the format per topic so later teams do not guess. Changing format after many consumers depend on a topic is expensive, because every consumer and the registry subject must move together. Ask any partner for a written comparison that covers your languages, your registry and a sample of real message sizes measured in your environment, and treat the result as the input to your schema evolution rules rather than a final answer.

How Avro, Protobuf and JSON Schema differ on evolution, size and tooling

Avro is a schema-first binary format that Kafka teams often pair with a schema registry. Compatibility depends on the reader and writer schemas both being available when data is decoded, so evolution is governed by rules about adding and removing fields with defaults. Records are compact because field names are not repeated in each payload, and the schema is needed to decode them. Avro has long been the default in Confluent-style Kafka stacks, so its serializers are widely available.

Evolution: Adding a field works only with a default value, and removing a field needs a default on the reader side for consumers that still expect it.
Payload: Binary records carry no field names, so small messages with many fields are usually compact. Whether the schema ID travels with each record depends on the serializer and registry setup.
Tooling: Java support is mature. Support in other languages varies by library, so confirm the library and its version for each consumer language.
Pitfalls: Teams often add a field without a default, then find that old consumers or old data cannot be decoded.
Registry: Confirm which registry APIs your serializers call before you commit to a registry.

Before adopting Avro, check that every producer and consumer language has a maintained Avro library and that your registry stores Avro subjects in the way your serializers expect.

Which criteria should decide the format for your organization

No single format wins on every criterion, so the choice depends on how your teams change schemas, how much data each message carries, and what your registry and tooling support. Each card names one question to answer and what a team usually learns from it before committing to a format.

Evolution needs of slow consumers

Ask how far behind your slowest consumers run when a producer changes a schema. Consumers that lag several versions need rules that hold across the whole history, not just the previous version. Check how each format handles added, removed and renamed fields, and whether your registry enforces transitive compatibility. Teams often learn this matters more than payload size once a consumer is redeployed rarely.

Payload size and throughput

Ask how many messages per second each topic carries and how large each record is after serialization. Binary formats such as Avro and Protobuf usually produce smaller records than JSON, because field names are not repeated in every message. Measure with your own sample records, since the saving depends on field count, nesting and whether the schema travels by ID or inline.

Language coverage across teams

List every language that producers and consumers use, including batch jobs, stream processors and analytics tools. Confirm that each serializer library is maintained for those languages and that generated code or runtime reflection is available where your teams need it. A format that is well supported in Java may need more work in a smaller language ecosystem, so check each one before standardizing.

Registry support in your chosen registry

Ask which formats your selected registry stores, validates and checks for compatibility. Support differs across Apicurio Registry, Karapace and Confluent Schema Registry, and some features such as rule types or reference handling may vary by format. Confirm the exact format and rule support against each project's current documentation in a test environment before you commit topics to a format.

Code generation and CI checks

Ask whether your pipeline should generate classes from schemas and block merges that break compatibility. Protobuf and Avro both have mature code generators, while JSON Schema validation is often done at runtime or with separate tooling. Decide which schema checks run in CI, who owns the generated code, and how regenerated artifacts reach producers and consumers.

Existing team skills

Ask which formats your engineers already read, write and debug in production. A format that the team can inspect quickly during an incident can outweigh a small gain in size or speed. Plan training and examples for the format you choose, and document how to decode a sample record from each topic so that on-call engineers are not blocked.

Which registries support each format across Apicurio, Karapace and Confluent

Format support differs by registry and by version, so the first step is to confirm, for each candidate registry, which of Avro, Protobuf and JSON Schema it accepts in the release you would deploy. Do not carry a support matrix from a blog post, a conference talk or an older version of this page into a decision. Read the current documentation of Apicurio Registry, Karapace and Confluent Schema Registry, note the exact release number you tested, and record whether each format is supported for registration, compatibility checks, reference handling and serializer integration. A registry that stores Protobuf schemas but cannot enforce compatibility for them is a different outcome from one that does both, and your data contract process will depend on which applies.

The Confluent-compatible API changes the serializer decision more than the registry feature list does. Many existing producers and consumers call the Confluent serializer libraries, which expect the Confluent wire format and a schema ID that the registry can resolve. If your candidate registry implements that API, you can often keep those clients and switch only the endpoint and credentials. If it implements only part of the API, or handles schema IDs, subject naming or compatibility responses differently, every client that uses those paths needs testing before cutover. Ask each vendor or project which endpoints, serializer versions and format-specific behaviors are covered, then run your highest-volume Avro, Protobuf and JSON Schema producers and their consumers against a non-production instance. Verify that deserialization succeeds for old and new schema IDs, not only for new writes.

Format options and when each one fits

This table compares the three schema formats on evolution rules, payload and tooling, language and registry support, and typical fit for Kafka topics. Confirm language and registry support against the exact client libraries and registry versions you plan to run, because support varies by tool and release.

Format	Schema evolution strengths	Payload and tooling considerations	Language support to confirm	Registry support to confirm	Best fit
Avro	Well suited to compatibility modes where readers and writers resolve fields by name and default values, so adding fields with defaults is the common path.	Compact binary encoding with schema shipped by registry reference; tooling is mature in the JVM ecosystem and common in Kafka Connect and stream processing.	Confirm the Avro library and generated-class support for each language your producers and consumers use.	Confirm which registries accept Avro subjects and which compatibility modes they enforce for your setup.	Data pipelines and JVM-heavy estates where schemas change often and consumers include Connect and Streams applications.
Protobuf	Field numbers, not names, identify data on the wire, so renaming fields is safe while reusing or changing field numbers breaks readers; reserved fields guard against reuse.	Compact binary encoding with strong code generation; generated types are common in service-oriented teams, though nested or well-known types can complicate tooling.	Confirm generated code and runtime support for each language, including the version of the protobuf compiler your teams use.	Confirm registry support for Protobuf subjects, including how imports and multiple message types per file are resolved.	Polyglot service teams that already build APIs with Protobuf and want one contract definition across gRPC and Kafka.
JSON Schema	Flexible validation rules and optional fields make it easy to describe loosely structured events, but compatibility depends on keyword semantics that differ between drafts.	Text payloads are larger and slower to parse than binary formats, but are readable in tools and easy to debug without special decoders.	Most languages have JSON parsers; confirm validator library support and the JSON Schema draft each library implements.	Confirm registry support for JSON Schema subjects and whether compatibility checks cover the rules your schemas use.	Event streams consumed by web, analytics or non-JVM teams, where human readability and broad tooling matter more than payload size.
Mixed formats across topic families	Each topic family keeps one format for its lifetime, so evolution rules stay consistent for the producers and consumers of that family.	Mixing formats adds more client libraries, more registry subject types and more serializer configuration for platform teams to maintain.	Confirm that every team's clients can use the formats assigned to their topics without custom serializers.	Confirm that the registry you choose applies compatibility rules per subject for each format in use.	A sensible choice when one domain has a strong existing format, such as Protobuf services feeding Avro analytics topics, and the platform team documents the boundary.

How to pick and roll out a format on a self-hosted cluster

This sequence turns a format choice into a tested standard: each step inventories the teams involved, sets evolution rules, proves the registry and clients work, and ends with CI checks and a rollout plan that other teams can follow.

Inventory producers, consumers and their languages
List every service that produces or consumes the topics in scope, the language and client library each uses, and the Kafka client versions in place. Include consumers you do not own, such as analytics jobs and partner pipelines, because they set the constraints on format choice. Record which teams can change code in a release and which cannot. The step is complete when each topic has a named producer team, a named list of consumer teams and their languages, and an agreed owner who can answer questions about the payloads. Confirm the language support of each candidate format against your client libraries before you rely on it, since support varies by language and by serializer package.
Define evolution rules per subject
For each subject, decide which compatibility mode the slowest consumer needs. Consumers that lag several versions behind producers usually need transitive modes, while tightly coupled pairs may accept simpler rules. Write down which kinds of change are allowed (adding optional fields, renaming, changing types, removing fields) for each format, because the rules differ between Avro, Protobuf and JSON Schema. Agree how a breaking change is handled, including when it requires a new topic instead of an in-place change. The output is a short table of subject, format, compatibility mode and owner, reviewed by at least one consumer team. Verify it by checking one sample change per format against the chosen mode in a test registry before the table is approved.
Pick a registry and confirm format support
Choose the registry that will hold the schemas, such as Apicurio Registry, Karapace or Confluent Schema Registry, and confirm which formats it stores and validates, which API it exposes to existing serializers, and how it authenticates clients. Confirm licensing terms for your use before deployment, because the Confluent Community License and open-source alternatives carry different conditions. Ask the vendor or project to confirm any format or compatibility feature you depend on, since registry support for Protobuf and JSON Schema differs by product and version. The step is complete when a written support matrix lists each format, each registry feature you need and the source that confirmed it.
Run a pilot topic and measure payload size
Create one pilot topic with representative production messages, encode the same records in each candidate format and measure the serialized size, including the schema ID overhead the registry wire format adds. Also measure encode and decode time in your main languages on a sample of realistic volume. Keep the test records copied from real data or a faithful copy of it, with sensitive fields masked. The step closes with a comparison table of size and CPU per format, per topic, with the methodology written down so the numbers can be repeated. Verify the pilot by replaying the same records through a consumer in each language and confirming the decoded values match the originals field by field.
Write the serializers and the CI schema checks
Implement the producer and consumer serializers in each language using the chosen format and registry client, and keep the schema files in version control next to the code that owns them. Add a CI job that registers each changed schema against the registry in compatibility-check mode, so a change that breaks the agreed rule fails the build before merge. Include a test where a deliberately incompatible change is submitted and must be rejected, and a test where a compatible change passes. The step is complete when both tests run in CI for each owning repository and the pipeline output is archived with the build. Confirm that the CI user has only the registry permissions it needs.
Publish the standard and the rollout plan
Write a short standard that states the approved format per use case, the registry and its owners, the compatibility mode per subject type, the naming convention for subjects, and the process for exceptions. Attach the rollout plan: which topics move first, which consumers must upgrade before producers switch, how old and new formats coexist during migration, and how the schema IDs in existing records are kept readable. Name the owner for each approval. The sequence is complete when the standard is published to the teams that reviewed it, the first production topic has passed its rollout checkpoint, and a consumer in each language has decoded a live sample record in the new format.

Mixing formats across teams and where it goes wrong

Most format problems on a shared Kafka platform do not come from choosing the wrong format. They come from several teams choosing different formats, or the same team drifting over time without a shared rule. The cards below cover the risks that appear once more than one format is in use, and the practice that prevents each one.

Format drift on one topic

A topic that starts as Avro can receive Protobuf or raw JSON from a producer added later, usually because a new team writes its own serializer. Consumers then fail on records they cannot parse, and the failure often appears only after a deployment. Prevention: set one format per topic in the topic metadata, and have the producer's registry subject and serializer checked in CI before a release.

Consumers that cannot decode

A consumer built for one format cannot read records in another, even when the business fields are identical. Teams that share a topic across languages may find that a library for the second format is missing or pinned to an old version. Prevention: list every consumer's supported formats in the catalog, and test each one against a sample record from every producer before a new format is allowed on a shared topic.

Duplicated schema definitions diverging

When teams copy a schema file into their own repository instead of importing it, the copies drift. A field is renamed in one place, a default changes in another, and both still pass local tests. Prevention: keep shared definitions in one schema repository with an owner per domain, publish versioned artifacts, and have consuming services reference a version rather than a copy.

Tooling gaps in one language

Protobuf code generation, Avro record classes and JSON Schema validators differ in maturity across languages, and a team writing in a less common language may find generated code is stale or hard to regenerate. Prevention: before standardizing, check generator and serializer support for each language in use, pin generator versions, and record the approved toolchain per language in the platform standards.

Migration cost when a format changes

Moving a topic or domain from one format to another later means new schemas, new serializers in every producer, dual-publishing during the transition, and consumer changes on each side. The cost grows with the number of consumers and the retention of the topic, since old records remain in the log. Prevention: decide the format per domain before the first producer goes live, and use a documented exception process with a review date for any deviation.

Exceptions without a process

Teams often need a format exception for a legitimate reason, such as an existing partner feed in JSON. When exceptions are granted informally, the platform ends up with many special cases that no one owns. Prevention: write a short exception request that names the topic, the reason, the consumers affected, an owner and an end date, and keep the approved list visible to all platform teams.

Questions teams ask before choosing a Kafka message format

Which message format is best for a new Kafka platform?

No format wins on every axis, so the best choice depends on which languages your teams use, how often schemas will change, and which registry you plan to run. Avro is common in Kafka pipelines that feed data warehouses and Connect connectors, Protobuf suits teams already generating code from .proto files across gRPC services, and JSON Schema fits teams whose payloads are already JSON and whose consumers want readable messages. Start by inventorying producers and consumers, then test one representative topic in each candidate format against your registry and client libraries before standardizing.

How much effort do the decision and rollout take, and what drives it?

Effort scales with the number of producers, consumers, languages and topics that must change, not with the format decision itself. The main drivers are how many services already serialize messages, whether schemas exist only in code or are already written down, how much consumer-side deployment order you can control, and whether you must support old and new formats during a transition. Ask any partner to break the quote into discovery, format and registry design, pilot, and rollout waves, and to state which parts are one-time and which recur as schemas evolve.

Can an internal team run the format evaluation alone?

Often yes, provided one or two engineers have hands-on experience with each serialization library and someone owns the registry. The evaluation needs real sample messages, a compatibility test that runs a new schema against old readers and writers, and a measured view of payload size for your typical records. Internal teams usually stall on consumer coordination across departments rather than on the technical comparison, so decide early who approves breaking changes and who will run the rollout.

How do format changes affect existing topics?

Changing the format of an existing topic does not rewrite messages already stored in it, so consumers must be able to read both old and new records during the transition. The safer pattern is usually a new topic for the new format, a period where producers write to both or consumers read from both, and a planned cutoff. Schema compatibility rules determine which changes are safe in place, so review them per subject before any producer switches formats.

Is it acceptable to mix Avro, Protobuf and JSON Schema across teams?

Mixing formats is workable when each topic uses one format consistently and the registry can store all of them. Problems appear when one topic carries several formats or when platform tooling, such as connectors, stream processors and data catalogs, supports only some of them. Write a short policy that states which formats are allowed, which team owns each subject, and how a team proposes a new format. Confirm with each registry and connector project which formats it supports before you approve a mix.

What should a partner need from us before scoping the work?

A partner needs a language and framework inventory, a set of representative sample messages from real topics (with sensitive fields masked), a list of current consumers and their deploy cadence, and your registry choice or the constraints that narrow it, such as licensing, hosting or authentication requirements. It also helps to know which compatibility rules your teams expect and who signs off on breaking changes. Missing inputs usually extend discovery, so provide them in writing before the scope is agreed.

How long does a pilot take, and what determines its length?

Pilot length depends on the number of topics, the number of languages in use, whether a registry already exists, and how many consumers must be exercised before you trust the compatibility checks. A narrow pilot covering one format, one registry and two or three topics is much shorter than a multi-format trial across several teams. Ask for the pilot's exit criteria in the scope, including which compatibility tests must pass and which consumers must be verified, rather than a fixed number of weeks.

What does a fixed-scope quote cover?

A fixed-scope quote covers the agreed deliverables written into the statement of work, such as the format decision record, registry deployment, compatibility rules per subject, the pilot topics, documentation and handover. It should name what is excluded, including changes to application code owned by your teams, and how change requests are priced. DeployOpen does not publish a price list, so a quote follows from the written scope. Ask for each deliverable to be listed with its acceptance criteria.

How DeployOpen scopes and delivers a Kafka format decision

DeployOpen approaches the Avro, Protobuf and JSON Schema choice as a scoped design decision tied to your topics, consumers and registry, not as a general preference. We write down what is decided, what is tested and what your teams own after handover, in infrastructure you control.

How we scope

Engagements are fixed-scope, with the format decision, deliverables and timeline agreed in writing before work starts.
The scope lists the topics, producer and consumer languages, and registry candidates the format decision must cover.
We confirm which registry compatibility modes and serializer libraries each team already uses before choosing a format.
Ask for the scope to name every topic that will keep, change or migrate its format, so the quote matches the work.

How we build

We build a comparison of your sample messages in each candidate format, measuring payload size and serializer behavior for your data.
We test schema evolution with real producer and consumer version pairs under the compatibility mode you choose.
We verify registry behavior for each format on the registry you run, including how schema IDs and references are handled.
We set up CI checks that reject incompatible schema changes before they reach a shared topic.

How we hand over

You receive a written format decision record covering each topic, the rationale, and the rules that apply to it.
Schema files, registry configuration, compatibility settings and CI pipeline definitions are delivered to your repositories.
Runbooks explain how to add a topic, evolve a schema and handle a breaking change through a new topic.
Your own engineers run a walkthrough and a supervised schema change before the engagement closes.

How we support

Ongoing support, registry upgrades and continuous testing are available as retainers after handover.
Retainers can cover reviews of new schemas from teams that adopt Kafka later, keeping the format rules consistent.
Re-testing is scoped when you upgrade Kafka, change registry software or add a new client language.
Confirm with the vendor or project documentation for each registry version before accepting it into scope.

Avro vs Protobuf vs JSON Schema for Kafka messages