Karapace as a Confluent Schema Registry replacement: what it does on your Kafka
Karapace is a schema registry and REST access layer you can run yourself for the schemas that Kafka producers and consumers use. If your services already use Confluent-style serializers and deserializers, the question is whether a self-operated registry can answer the same API calls those clients make, without changing application code. Karapace is considered when that API compatibility matters more than adopting a different registry model, and when you want the registry to sit inside the infrastructure you already control. It stores schemas in a Kafka topic on the cluster it serves, so the registry's durability depends on the Kafka cluster's replication, retention and backup design, which you should review before go-live rather than treat as a separate concern.
Compatibility is the part to verify first. Confirm the Karapace release you intend to run against the client library versions, serializer classes and schema formats your teams use, and against the Karapace documentation for each endpoint those clients call, including subject, version and compatibility operations. No vendor-verified compatibility matrix is covered here, so do not assume feature parity beyond what your own test run demonstrates. Build a short test in a non-production environment that registers, fetches and evolves a schema for each format and client language in use, and check that schema IDs embedded in serialized records resolve as expected. Record the results against the exact versions in your change record.
Plan the decision as a trade-off, not a replacement by default. Karapace suits teams whose main requirement is a drop-in registry for existing Confluent-style clients. If your schema governance needs depend on features you have not confirmed in Karapace, or if you are starting without legacy clients, compare it with other open-source options before committing. Confirm the scope of operational support, upgrade procedures and who owns the topic that holds the schema data, because those duties recur for as long as the registry runs.
What you need in place before Karapace can run against your cluster
Karapace stores schemas in a Kafka topic, so the cluster, the identity it uses and the network path to the brokers must be settled before the registry is installed. The cards below cover the Kafka baseline, the credentials and certificates that connect the registry to it, and the sizing questions to answer first.
KRaft cluster on a supported release
Karapace needs a running Kafka cluster to hold its schema storage topic. Run the cluster in KRaft mode, since Kafka 4.0 supports only KRaft and ZooKeeper-based clusters must migrate first. Pick a release listed as supported on the Kafka downloads page, currently 4.3.1, 4.2.2 and 4.1.2, and confirm the version you choose is still in support on the day you deploy. Use KRaft mode only as the baseline for every new registry deployment.
Java 17 on brokers and tools
Since Kafka 4.0, brokers, Kafka Connect and command-line tools require Java 17, while clients need at least Java 11. Check the JVM on every broker and on any admin host you use to create topics or ACLs. The registry itself has its own runtime requirements, so confirm those in Karapace's documentation instead of assuming they match the broker JVM.
Broker listener reachable from the registry host
Karapace connects to the brokers as a Kafka client, so the registry host must reach a listener that the brokers advertise to clients. Verify the advertised listener addresses, not only the bootstrap address, because a host that resolves the bootstrap name may still fail on the advertised one. Confirm firewall rules, DNS and any load balancer in front of the brokers before installation.
Dedicated principal and ACLs
Give Karapace its own Kafka principal rather than a shared administrator account. Grant that principal only the operations it needs on the schema storage topic, and no broad rights on application topics. In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set. Leave that setting at its default unless you have a reviewed reason to change it.
TLS certificates for broker and API
Plan two certificate paths: one from the registry to the Kafka brokers, and one for the registry's HTTPS API that producers and consumers call. Kafka supports SSL and SASL authentication and TLS encryption, and the mechanism you choose determines which certificates and credentials you issue. Decide who issues the certificates, how they rotate, and where the trust stores live before the registry is configured.
Network and capacity planning
Schema traffic is small compared with message traffic, but every serializer call depends on the registry being available, so plan for redundancy and placement. Decide how many registry instances you will run, where they sit relative to the brokers, and how they fail over. Confirm the storage topic's replication factor and retention settings with your Kafka operators, and ask the Karapace project for its sizing guidance for your expected number of subjects and clients.
How to configure Karapace against a KRaft Kafka cluster
This sequence takes Karapace from a Kafka principal with narrow permissions to two registry instances that share one schema storage topic, with a test serializer confirming that schemas register and fetch correctly. Each step ends with a check you can run before moving on.
Create the registry principal and its ACLs
Your Kafka administrator creates a dedicated principal for Karapace, separate from application and admin accounts. In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log, and any resource without an ACL is limited to super users unless allow.everyone.if.no.acl.found is set to true, so the registry principal needs explicit grants on the schema storage topic and its consumer group. Confirm the grants with the Kafka ACL listing tool, then confirm that a client using the principal can describe the storage topic but cannot read or write other topics. Exact ACL operations for the schema topic should follow the Karapace documentation for your chosen version.
Configure the Kafka connection with TLS and SASL
Set the registry's bootstrap servers, TLS truststore and SASL settings in the Karapace configuration file, choosing one mechanism from the list Kafka documents: GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 or OAUTHBEARER. Kafka security is optional, so an unsecured listener would accept clients without these settings, which is why you should confirm the broker listener requires SASL and TLS before you start the registry. Take every configuration key name from the Karapace documentation for the version you run, not from older examples. Done means the registry starts without a Kafka authentication error in its log.
Set the storage topic replication and retention
Create the schema storage topic yourself rather than letting it be auto-created with broker defaults. Set a replication factor that matches your broker count and a retention policy that never expires schema records, because deleting this topic or letting records age out removes the schemas that serialized messages depend on. Confirm the replication factor, partition count and cleanup policy with the describe command, and record them in your runbook. Ask the Karapace maintainers or your vendor support which topic configuration they recommend for the version you chose.
Start one instance and confirm topic writes
Start a single Karapace instance with the verified configuration and register one test schema through its REST endpoint. Then consume the storage topic from the beginning using the registry principal or an admin client and check that a record for the schema appears. If nothing appears, the registry is not writing to the storage topic it was configured to use, so check the topic name in the configuration before continuing.
Add a second instance on the same topic
Start a second Karapace instance pointing at the same storage topic and the same Kafka principal, with its own host name and HTTP listener. Both instances read the same storage topic, so a schema registered through one should be fetchable through the other. Stop the first instance, then query the second instance for the test schema to confirm that it serves the stored record. Restart the first instance before continuing so both are running for the next step.
Point a test serializer at the REST endpoint
Configure a test producer that uses a Confluent-compatible serializer and set its schema registry URL to the load balancer or address in front of your Karapace instances. Produce a message to a non-production topic so the serializer registers its schema through the REST API. Confirm the registration succeeded by listing the subject and version in the registry, then fetch the schema by its ID. Confirm that the registry rejects requests from a client without credentials, which proves that access control applies at the REST layer as well as the Kafka layer.
Verify the compatibility behavior before production use
Attempt to register a schema change that breaks your chosen compatibility rule and confirm the registry rejects it with an error. Then register a compatible change and confirm it is accepted under the same subject. Record both results with the Karapace version, the compatibility setting and the test subject name. These results give your team a baseline to repeat after each upgrade or configuration change.
How Kafka principals, ACLs and schema subjects map to each other
This table maps each Kafka principal in a Karapace deployment to the storage topic and schema subjects it touches, the operation it performs and the access to grant or deny. Rows marked with a dash in the registry authorization column depend on confirming Karapace's own authorization model, so check those with the Karapace documentation before relying on them. Because Kafka denies resources without ACLs to everyone except super users by default, every principal needs an explicit row.
| Principal | Resources needed | Operation and access to grant or deny |
|---|---|---|
| Registry admin service account | Storage topic for schemas, the consumer group Karapace uses, and all subjects | Grant read, write and describe on the storage topic and its consumer group. Deny direct producer writes to the storage topic for all other principals. Confirm registry-level admin rights in the Karapace documentation. |
| Producer service account | Subjects for the topics it publishes to, plus schema lookup by ID | Grant read on the subjects it registers or looks up. Deny subject deletion and compatibility changes. Confirm whether registration permissions are enforced by Karapace or only by Kafka ACLs. |
| Consumer service account | Subjects for topics it reads, plus schema lookup by ID | Grant read on the subjects it deserializes. Deny registration and any compatibility change. Confirm whether the registry restricts reads per subject or per instance. |
| Read-only auditor | Subject listing and version metadata, with no topic data access | Grant read-only metadata access to subjects and versions. Deny all writes and deny Kafka data topic reads unless audit scope requires them. Confirm read-only role support in the Karapace documentation. |
| Migration or CI pipeline account | Subjects under test and the staging registry instance | Grant write only in the staging instance and read in production for compatibility checks. Deny production registration. Remove the account once cutover is complete. |
| Any principal without a row | Anything not explicitly granted | Deny by design. Under Kafka's default behavior, resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so keep that setting false and document every grant. |
Security controls for the registry and its Kafka storage
Karapace keeps its schemas in a Kafka topic, so the registry's security depends on both the REST layer and the cluster beneath it. The controls below cover transport, identity, authorization, storage protection and API access, and each one needs an explicit decision because Kafka security is optional by default.
TLS on the REST API and Kafka
Encrypt client traffic to the Karapace REST endpoint and encrypt the registry's connections to the Kafka brokers. Kafka security is optional, so a plaintext listener will work unless the team removes it. Disable plaintext listeners on the path the registry uses, and confirm in Karapace's configuration reference how TLS is enabled for its HTTP listener and its Kafka client settings. Check certificate expiry and rotation before go-live.
SASL identity for the registry
Give the registry its own Kafka principal rather than sharing an administrator or application account. Kafka supports the GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER mechanisms. Pick one that your identity platform can operate and rotate, and confirm which mechanisms the Karapace release you deploy can use with your cluster.
Least-privilege ACLs on storage
In KRaft clusters, the StandardAuthorizer keeps ACLs in the cluster metadata log, and resources without an ACL are limited to super users unless allow.everyone.if.no.acl.found is set to true. Grant the registry principal read and write access to its storage topic and the consumer group it needs, and nothing broader. Keep that flag off so unlisted resources stay closed.
Protecting the storage topic
The schema storage topic is the only copy of registry state, so an accidental delete or a configuration change to cleanup policy can remove schema history that serialized records depend on. Restrict topic deletion and configuration changes to a small operator group, and back up the topic contents to storage the platform team controls. Confirm with Karapace which topic configuration it requires before you change any setting.
Access control on the REST API
Kafka ACLs protect the cluster, but they do not decide which clients may register or delete schemas through Karapace. Determine whether the Karapace release you run provides authentication or authorization on its REST endpoints, or whether an upstream gateway or reverse proxy must enforce it. Separate read-only lookups from write and delete operations, and test that a consumer cannot change compatibility settings.
Why Karapace registrations fail and how to fix them
This table pairs each common Karapace registration failure with its likely cause and a concrete fix. Read it when a registration, client connection or schema lookup fails after deployment or migration. The main takeaway is that most failures trace to storage topic permissions, TLS trust, ID carry-over, compatibility settings or normalization, and each has a specific check you can run.
| Symptom | Likely cause | Fix |
|---|---|---|
| Unauthorized errors when Karapace starts or registers a schema | The Karapace service principal lacks ACLs on the storage topic (for example, _schemas), so it cannot read or write registry state | Grant the principal read, write and describe access on the storage topic and the consumer group it uses. In KRaft clusters, ACLs live in the metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true. Check the Kafka authorization documentation before changing that setting. |
| TLS handshake failures between Karapace and the brokers or clients | The certificate chain is incomplete or the CA is not in the truststore, or the broker hostname does not match the certificate's subject or SAN entries | Confirm the full chain is presented, import the issuing CA into the truststore Karapace uses, and make sure the advertised broker hostnames appear in the certificate SAN list. Test with a TLS client from the same host before changing Karapace. |
| Schema IDs differ from IDs stored in existing records after migration | Schemas were re-registered in a new order or with new IDs instead of preserving the original ID mapping, so serialized records point at IDs that no longer resolve | Export subjects and versions with their IDs from the source registry and restore them in the same order before switching producers. Verify a sample of existing record IDs against the target registry before cutover. |
| Registration rejected because a compatibility rule blocks the change | The subject's compatibility mode (backward, forward, full or a transitive variant) rejects the new schema because a field was removed, a default is missing or a type changed | Check the subject's current mode, then either adjust the schema to meet the rule (add defaults, keep removed fields optional) or change the mode deliberately with an agreed review. Do not disable compatibility checks globally to get a release through. |
| An identical schema registers as a new version | Whitespace, field ordering or other formatting differences change the normalized form, or normalization is applied on one side only (for example, the producer's client versus the registry) | Compare the canonical form of both schemas and confirm both sides use the same normalization setting. Confirm which normalization Karapace applies for each format in its documentation before relying on identical-schema detection. |
| Registry state is lost after a restart | The storage topic has short retention, was deleted or was recreated, so the stored schemas and IDs disappeared | Set retention on the storage topic to keep all schema records (use a cleanup policy and retention that match the registry's needs), protect the topic from deletion with ACLs, and back it up with the rest of the cluster. Do not recreate the topic on restart without a restore plan. |
When Karapace fits and when Apicurio fits better
Many teams already run producers and consumers with Confluent serializers that call a schema registry over its REST API. If that is your situation, the registry is often the component you can least afford to change, because client configuration, deployment pipelines and schema ID references all depend on it. The question is whether a candidate keeps that contract stable for the serializer versions and tooling you already operate.
- Serializer inventory: list each language client, serializer library version and subject naming pattern in use today.
- Endpoint coverage: compare the REST paths your clients call, including subject lookup, registration and compatibility checks, against each project's API documentation.
- Schema ID behavior: confirm that IDs returned after migration match the IDs that serialized records already carry.
- Staging tests: run your existing serializers against a staging copy before any cutover.
Ask each project which API version it targets and ask it to demonstrate a passing run of your serializer test suite. Apicurio's options are covered on its own sibling page.
Karapace stores schemas in a Kafka topic, which keeps the registry close to the cluster it serves and avoids a separate database in many setups. Some organizations need schemas held in a relational database, reviewed through a Git-based workflow, or governed by a store with its own backup and replication policy. Those requirements often decide the registry more than any API detail.
- Backup ownership: decide whether schemas are restored with the Kafka cluster or from a separate backup.
- Topic durability: confirm the replication factor, retention and cleanup settings for the internal schema topic on your cluster.
- Alternative stores: ask each project which backing stores it supports today and which are planned, without assuming either answer.
- Recovery drills: restore schemas into a second environment, because missing schemas break deserialization for every consumer.
Check how schemas are recovered after a cluster rebuild and who approves that procedure. Apicurio's storage options are covered on its own sibling page.
Teams that mix Avro, Protobuf and JSON Schema, or that want validation beyond compatibility checks, need to verify each format against their own serializers. Format support and rule enforcement can differ between clients and registries, and normalization affects how equivalent schemas are matched. Treat each capability as something to test with your own schemas.
- Format list: confirm support for Avro, Protobuf and JSON Schema separately, including the schema versions your teams use.
- Normalization: submit equivalent schemas that differ in whitespace, field order or references, then check whether they receive the same ID and record any limits you find.
- Content rules: write the rules you need, such as required metadata or naming conventions, and check whether the candidate enforces them at registration.
- Reference handling: test schemas that import other schemas, since cross-subject references are a common failure point.
Run a representative sample from each team through the candidate before deciding. Apicurio's format and rule coverage is covered on its own sibling page.
A strong need for registry-side controls, such as per-team access to subjects, audit trails or integration with a central identity provider, can change the evaluation more than the API does. The key question is what the candidate enforces itself and what must be handled by a reverse proxy or gateway placed in front of it.
- Authentication methods: list the methods your identity provider supports, such as OIDC or mutual TLS, and ask which the candidate accepts directly.
- Authorization scope: confirm whether access can be limited per subject, per team or only per registry instance.
- Audit records: check which actions are logged, including registration, deletion and compatibility changes, and where those logs are sent.
- Gateway dependency: if controls live in a proxy, record who owns that component and how it is patched.
Ask each project for a written authentication and authorization model and test it with your identity provider in staging. Apicurio's authentication features are covered on its own sibling page.
Questions teams ask before they choose Karapace
What drives the cost of a Karapace deployment?
Cluster readiness, the number of clients to validate, the migration of existing subjects and the high availability design drive most of the cost. A Kafka cluster that already has TLS and ACLs in place needs less groundwork than one that does not. Each client language and serializer version adds validation work, and every subject with a version history adds migration checks, especially where existing schema IDs must still resolve. An HA design adds instances, storage and failover testing. Build and migration are one-time costs, while upgrades, monitoring and client changes recur. Ask any quote to itemize these components, state what is excluded and name the test evidence that closes the work.
Can our team deploy Karapace without outside help?
Yes, if the team already has the skills the work depends on. A team that has run Kafka ACLs and TLS in production, understands how serializers look up schemas and can run a compatibility test suite against its own clients is well placed to do the work itself. Teams without that experience should expect a learning period and more review cycles. Before deciding, check three things: whether the team can write and test ACLs for the registry's service identity, whether it can issue and rotate the certificates the registry needs, and whether it has a repeatable way to prove every producer and consumer still works after a switch. Outside help can cover gaps in those areas under a written scope.
How long does a Karapace migration take?
The supporting material for this page does not give a duration, so treat any number in a proposal as a question to challenge. Duration depends on how many subjects and schema versions exist, how many clients read and write them, how many languages those clients use, how much test coverage already exists and how often change windows open. Cluster readiness and HA design can run in parallel with the client inventory, but cutover for each client moves at the pace of its owning team. Ask for a plan that lists milestones, the test evidence required to pass each one and the dependencies on teams outside the platform group. Confirm your change calendar before the plan is written.
What does our side need to provide for the work?
Expect to provide access, inventory and decisions. Engineers need access to the Kafka brokers and to the environments where the registry will run, including the ability to create service accounts and ACLs. Certificates for the registry and its clients must come from your certificate authority or from a process you approve. A client inventory is essential, listing each producer and consumer with its language, serializer version and owner. You also need named change windows, a rollback owner for each cutover and a decision-maker who can accept or reject compatibility results. Missing items usually appear as schedule slips, so list them in the written scope before work starts.
How do we confirm Karapace license terms before deploying?
Confirm the license directly with the Karapace project and read the terms in its source repository before you plan the rollout. Summaries, blog posts and third-party pages can be out of date, so the license file and any notices in the source tree are the sources to check. Ask your legal or procurement team to review how your planned use fits those terms, especially if you intend to offer the registry to other organizations or bundle it into a product. Record the version and date you reviewed, because license terms can change between releases. DeployOpen can help document your deployment, but the licensing decision stays with your organization and its counsel.
Can DeployOpen take on Karapace work under a fixed scope?
Yes. Most DeployOpen engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. A quote follows from that written scope, so a useful first step is a scoping discussion covering the Kafka cluster, the client inventory, the serialization formats in use and the compatibility rules you need to keep. Work that continues after go-live, such as upgrades, monitoring reviews and onboarding of new clients, can be handled under a retainer. Ask any partner to state in writing what is included, what is excluded and how acceptance will be checked.
How DeployOpen scopes and delivers a Karapace implementation
DeployOpen plans, implements, integrates, documents and supports self-hosted open-source platforms inside your own infrastructure, including Karapace as a schema registry for Kafka topics. Engagements start with a written scope, so the compatibility, storage and access decisions are agreed before any cluster or client changes.
How we scope
- Inventory every subject, schema format and serializer your producers and consumers use today.
- Agree which compatibility modes apply per subject and who approves changes to them.
- Set the written scope, deliverables and timeline for the Karapace work before build starts.
How we build
- Deploy Karapace in the infrastructure you control, with its storage topic and Kafka access configured.
- Test client compatibility against your existing serializers before switching any production traffic.
- Configure authentication and access controls for the REST API and document each setting used.
How we hand over
- Deliver runbooks for deployment, schema export and import, rollback and routine maintenance.
- Provide test evidence showing schema IDs and compatibility checks behave as agreed.
- Walk your platform, security and data teams through operations and ownership of the registry.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- Review Karapace release notes and planned changes against your environment before upgrading.
- Confirm with your team which support scope, response expectations and upgrade cadence apply.
