What Apicurio Registry does for your Kafka producers and consumers
Apicurio Registry stores and versions the schemas for your Kafka messages and serves them to serializers and deserializers. Its Confluent-compatible API is designed so that existing Avro, Protobuf and JSON Schema serializers can point at it with minimal client change, which matters if your producers and consumers already run on Confluent serializer libraries. Today, many teams keep those schemas in application code, a wiki or a registry that no single team owns. Moving them into a registry you operate gives platform and data teams one place to see which subject a topic uses, which version a producer wrote, and which compatibility rule applies before a change ships.
In the usual integration, each producer and consumer calls the registry over HTTP when it first needs a schema, then caches the result. Serialized records carry a reference to the schema, so the registry must stay reachable to clients that are starting up or that see a new schema ID. The registry runs in your own cloud account, data center or restricted network, so schema content and access logs stay under your control. Before you commit to a release, confirm the exact API paths, the supported formats, and which client library versions work with that release in the Apicurio documentation. No Apicurio version, endpoint or client compatibility details have been verified for this page, so treat any version you read elsewhere as unconfirmed until you check it against the release you plan to run.
What Apicurio Registry needs from your Kafka cluster and platform
Each card below is a question you must answer before Apicurio Registry setup begins. The answers decide where schemas are stored, how the registry is reached and secured, and which clients can talk to it. Settle them with your platform, security and data teams together.
Which Kafka release runs KRaft?
Confirm that your cluster runs in KRaft mode and pick the release you will operate. The Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported, and earlier versions are archived. Clusters still on ZooKeeper must migrate to KRaft before any move to 4.x. Check the Java runtime the chosen Kafka release needs, since brokers require Java 17 from 4.0 onward. If you are on an older KRaft release, plan the upgrade path first.
Where do schema records live?
Decide the storage backend before installing anything. Apicurio Registry can keep schemas in a database or in a Kafka topic, and each choice changes backup, failover and who must be paged when the registry fails. Ask which database engine your platform team already runs and supports with backups, and whether the Kafka cluster should also hold registry state. Write down the owner of the storage layer.
Where will the registry run?
Choose the runtime that your operations team can patch, scale and monitor. Options usually include Kubernetes with a Helm chart or operator, or a virtual machine managed by your existing configuration tooling. Confirm the number of replicas needed for high availability and where they sit relative to the Kafka brokers. The registry should be placed in the same network zone as the Kafka and identity services it depends on.
Which certificate authority issues TLS?
Registry endpoints, the storage connection and the Kafka listeners all need certificates. Decide which internal certificate authority signs them, how renewal happens, and who holds the private keys. Clients will reject a registry whose certificate chain they do not trust, so distribute the CA bundle to every serializer host before testing. Document the rotation procedure, since expired certificates are a common cause of registry outages.
Who signs in to the registry?
Pick the identity provider that authenticates both people editing schemas and applications that register or fetch them. Decide how service accounts are created and rotated, and which groups may change compatibility rules or delete artifacts. Confirm that your provider can issue tokens the registry can validate. Keep the authorization model simple at first, because changes to who may publish a schema are the hardest to audit later.
Which serializers must reach the registry?
Inventory every producer and consumer library, including language, version and whether it uses the Confluent-compatible serializers. Teams often run different versions, and older clients may only support the Confluent-style API. Map each application to the network path it uses from its own subnet or cluster to the registry and to the brokers. Record which teams own each topic, because they will decide compatibility rules for their subjects.
Which Java runtime does the release need?
Apicurio Registry is a Java application, so the Java version depends on the release you choose. Confirm the required JDK in the release notes for that Apicurio version and check it against your base images and security baseline. Also confirm the runtime version used by the Kafka clients in your serializer inventory, since mismatches there cause failures that look like registry problems. Record the version in your change request.
How the storage backend choice changes operations
The storage backend decides how Apicurio Registry is backed up, restored, scaled and recovered when something fails, so it should be chosen with operations in mind, not only for the first install. Start by listing the storage options that the Apicurio Registry release you plan to run supports, and confirm each one against the vendor documentation for that exact version. Do not assume that a backend listed in an older guide or blog post is still supported, and check whether a backend is marked for production use or only for development and testing.
Next, decide who operates the store. If the backend is a relational database, your database team may already run backups, point-in-time recovery and failover for similar systems, and the registry becomes one more application on that platform. If the backend is a Kafka topic, schemas are stored as records, so the Kafka team must own replication, retention settings, topic configuration and recovery of the topic, and the registry's availability depends on the health of that cluster. Ask who gets paged when schema reads fail and who can restore schemas after accidental deletion.
Finally, check how the chosen store stays consistent across registry replicas. Confirm with the vendor documentation how concurrent writes of the same schema are handled, whether replicas read from a shared store or cache data locally, and how quickly changes are visible to all replicas. Test a restore into a separate environment and verify that schema IDs and versions match what serialized records reference. Match the choice to skills your team already has, because a backend your staff can restore under pressure is usually a safer choice than one that is technically better but unfamiliar.
Setting up Apicurio Registry in sequence
This sequence takes a team from confirmed Kafka and runtime versions to a working Apicurio Registry that serves schemas to a test producer and consumer. Each step closes with a check, so the team can stop and fix problems before moving on to the next layer.
Confirm Kafka and runtime versions
The platform engineering lead records the Kafka release the cluster runs, the Java runtime on the hosts, and the Apicurio Registry release the team intends to deploy. Kafka 4.x brokers and tools require Java 17, and clients need at least Java 11, so the check covers both sides. Confirm that the chosen Apicurio release supports the Kafka and Java versions in its own compatibility notes, since those are not covered by the Kafka documentation. Write the versions into the design record. The step is done when every component has a pinned version and the Java requirement matches each host.
Provision the storage backend
The infrastructure team chooses where registry data lives, such as a supported relational database or Kafka storage, and provisions it in the environment you control. Confirm the storage options and their setup in the Apicurio documentation for your release, including the required configuration keys and credentials. Keep the database on its own backup schedule and restrict network access to the registry service. The step is done when the storage endpoint accepts a connection from the registry host and a test write and read succeed with the intended service account.
Deploy a single registry instance in non-production
The platform team deploys one Apicurio Registry instance in a non-production namespace or host group, using the storage settings from the previous step. Use the deployment method and configuration keys in the Apicurio documentation for your release, and keep the configuration in version control. Avoid exposing the instance beyond the test network at this stage. The step is done when the health endpoint documented for your release returns a healthy status and the registry logs show a clean startup with no storage errors.
Enable TLS and authentication
The security engineer places the registry behind TLS using certificates from your internal authority, then configures authentication with the identity mechanism your organization uses. Confirm which authentication options your release supports and the exact property names in the Apicurio security documentation, and check how the Kafka clients will present credentials. Test access with a valid identity and with an invalid one. The step is done when unauthenticated requests are refused, authenticated requests succeed over HTTPS, and the certificate chain validates from a client host.
Configure the Confluent-compatible API and test one serializer
The integration engineer enables the Confluent-compatible API so existing serializers can use the registry URL without code changes. Confirm the endpoint path and the compatibility options in the Apicurio documentation for your release. Point one test producer at the registry with its serializer configuration and register a sample schema through it. The step is done when the producer registers the schema, the registry lists it under the expected subject, and the serializer reports a schema ID returned by the registry.
Set content rules on a test subject
The schema owner applies a compatibility or validity rule to a test subject, choosing the rule type from the options in the Apicurio documentation for your release. Then the owner submits a deliberately incompatible schema change to confirm the registry rejects it. Record which rule mode fits your consumer upgrade order before applying it to any production subject. The step is done when the registry refuses the incompatible version with a clear error and accepts a compatible version under the same rule.
Run a produce and consume test with schema lookup
The team runs a produce and consume test on a non-production Kafka topic. The producer serializes a record using the registry, and a separate consumer, configured with the same registry URL and credentials, looks up the schema by ID and decodes the record. Include at least one record that uses the rule-controlled subject. Check that the consumer's deserializer returns the expected fields and that registry request logs show lookups by ID. The step is done when the consumer decodes the test record correctly, and the team has a documented baseline before any production client is pointed at the registry.
Mapping Kafka clients, schema identifiers and rules to registry objects
This table shows where each Kafka-side concept lives in Apicurio Registry, who owns it and what to confirm in the release documentation before building. Read it as a checklist for the mapping design: the general model is consistent, but every exact name, header and configuration key must be checked against the version you deploy.
| Item | Where it lives | Who owns it | What to confirm |
|---|---|---|---|
| Topic to subject or artifact naming | Topic names in Kafka map to artifact IDs or subject names in the registry, usually by a naming strategy set in the serializer | Platform team sets the convention, data owners apply it per topic | Default topic-to-artifact strategy, and whether the Confluent-compatible API uses subject names the same way |
| Schema ID referenced by serialized records | Numeric global ID embedded in each record or its headers, resolved against the registry's global ID space | Platform team keeps IDs stable; consumers depend on them | Which ID format the serializers write, and whether the wire format matches the Confluent layout your consumers expect |
| Compatibility rule scope | Set globally for the registry or overridden per artifact or subject | Platform team sets defaults, data owners request exceptions | Rule types supported (backward, forward, full and transitive variants) and the precedence between global and per-artifact rules |
| Serializer client settings for registry URL and credentials | Registry endpoint and authentication settings in each producer and consumer configuration | Application teams, with credentials issued by the platform team | Exact configuration keys for the URL, auth method and TLS trust store in your release |
| Service accounts for producers and consumers | Identity provider or registry-side authentication, with roles that separate register, read and admin rights | Security team issues accounts, application owners request them | Role names, whether write and read permissions can be split, and how service account tokens are rotated |
| Group or artifact ownership metadata | Labels, properties or group structure stored with artifacts in the registry | Data owners for each domain, reviewed by governance | Which metadata fields are searchable and whether ownership can be enforced by policy or only recorded |
Using the Confluent-compatible API without rewriting every client
Teams that already run Kafka producers and consumers with Confluent serializers usually want a compatibility API so they do not have to change serializer classes, converter settings or client configuration across dozens of services. The Confluent-compatible API in Apicurio Registry lets those clients keep pointing at a schema registry URL and use their existing serializer and deserializer configuration, while the schemas themselves are stored and governed in Apicurio. The platform team's job is to prove that this holds for the clients you actually run, not to assume it. Start with an inventory of every serializer library and version in use, including the Java, Python, Go and .NET clients that some teams build with their own wrappers. Record which versions you tested against Apicurio, and treat any version not on that list as untested until a run proves otherwise.
The two checks that matter most are schema identity and compatibility behavior. Records already on topics carry schema IDs, and consumers resolve those IDs against the registry. Before cutover, confirm with the Apicurio documentation for your release how IDs are assigned and whether existing IDs can be preserved on import, and confirm it against your own data by deserializing a sample of historical records from each topic. For compatibility, check that the modes and content rules configured for each subject reject the same breaking changes your producers expect to be rejected, and accept the changes they expect to pass. Compatibility semantics are where client-side behavior and server-side enforcement most often diverge, so test each rule with a deliberately breaking and a deliberately safe schema change.
Run the testing as a parallel run in a non-production cluster that mirrors your topic layout, client versions and registry configuration. Point a copy of each service at the Apicurio compatibility endpoint, replay representative traffic, and compare what is registered, what IDs are returned and what consumers deserialize. Do not claim full parity with Confluent Schema Registry or any other registry. Check each endpoint your clients call, including subject listing, version lookup, schema lookup by content and compatibility checks, against the Apicurio compatibility documentation, and record every difference as either a fix, an accepted gap or a client change. Ask Apicurio support or the project maintainers to confirm any endpoint your clients depend on that is not clearly documented for your release, and keep the parallel run results as the evidence for the cutover decision.
Securing the registry, its storage and its Kafka access
The registry sits between producers, consumers and the storage that holds every schema version, so a weakness in any one link can change what data is accepted on a topic. The cards below cover the controls that connect those links: transport encryption, identity, Kafka permissions, credential separation, change auditing and recovery. Confirm each one against the Apicurio release you deploy, since the controls available depend on the version and storage backend you choose.
Encrypt every registry connection
Use TLS between serializers and the registry, between the registry and its storage backend, and between the registry and Kafka where the topics it uses require it. Kafka supports SSL/TLS for clients, brokers and tools, and security is optional, so an unsecured listener can remain open by accident. Check that no plaintext listener is exposed, and confirm which certificate authority and rotation process your team will operate for the registry's endpoints.
Authenticate people and services
Route human sign-in for the registry UI and admin API through your identity provider using OIDC or OAuth2 where the registry supports it, so access follows your existing group and offboarding process. Service accounts for producers and consumers need their own identities. Confirm which authentication methods the registry version supports and whether role mapping can be driven by identity provider groups.
Scope Kafka ACLs to registry topics
If the registry stores its data in a Kafka topic, grant its principal only the topic operations it needs and nothing cluster-wide. In KRaft clusters, the StandardAuthorizer keeps ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Keep that setting false, and review the registry's ACLs whenever its topics or consumer groups change.
Use one credential per client
Give each producer application and each consumer group its own principal and credential rather than sharing one service account across teams. Kafka supports SCRAM-SHA-512 and OAUTHBEARER among its SASL mechanisms, and separate principals let you rotate or revoke a single application without disrupting others. Confirm which SASL mechanisms the Apicurio serializer libraries in your stack support before you choose one.
Audit every schema change
Record who created, updated, deleted or changed the compatibility rule of a schema, and when. Schema changes alter what producers may publish, so the audit trail belongs in your security review and incident process. Ask which audit events the registry emits, where they are written, and whether they can be forwarded to your log platform before you rely on them for evidence.
Test restores of the storage backend
Back up the storage backend that holds schema versions and their IDs, because serialized records reference those IDs and a lost schema can make old data unreadable. Schedule backups, and run a restore into a separate environment on a regular basis to confirm the registry starts and returns the expected subjects and versions. Record the restore steps and the time they took so the recovery plan reflects real performance.
Common Apicurio Registry failures on Kafka and how to diagnose them
This table pairs common symptoms with their likely causes and the checks that confirm each one. Read it top to bottom when a serializer, consumer or registry instance misbehaves, and start with the cheapest verification before changing configuration. The main takeaway is that most schema incidents trace back to compatibility settings, client endpoint configuration or ID handling, and can be confirmed without changing production data.
| Symptom | Likely cause | Fix and verification |
|---|---|---|
| Schema registration rejected by a compatibility rule | The new schema breaks the backward, forward or full rule set on that artifact or group, often because a field was removed or a default was dropped | Read the artifact's rules and the global rule, then test the new schema against the latest version in a non-production registry before changing the schema or the rule |
| Consumers fail to decode records after a schema change | Consumers run an older schema version, or the compatibility mode allows a change that older readers cannot handle | Check which schema version each consumer resolves, confirm the compatibility mode matches the deploy order, and roll the consumer forward before the producer |
| Serializers cannot reach the registry (TLS or DNS) | The registry hostname does not resolve from the client network, or the client does not trust the certificate chain used by the registry endpoint | Resolve the registry hostname from a client pod or host, then run an HTTPS request against the registry's API root and confirm the certificate chain is trusted by the client truststore |
| Schema ID mismatch after a migration | Records were serialized with IDs from the source registry, while the target registry assigned different IDs, or clients cache old IDs | Compare the ID of a sample record's schema in both registries, confirm the compatibility API mode used by the clients, and restart clients to clear cached IDs before rerunning the test |
| Authorization errors on Kafka topics used by the platform | The registry's Kafka client principal lacks read or write permission on its storage topic or on related topics, or ACLs were applied to a different principal name | List the ACLs for the registry principal on the storage topic, confirm the principal name matches the TLS or SASL identity in the registry configuration, and add only the operations the storage topic requires |
| Registry instances disagree after a storage failover | One instance is serving from a stale replica or cache, or instances are pointed at different storage backends | Query the same artifact and version on each instance, confirm all instances share the same storage configuration, and check replication lag on the storage layer before allowing writes again |
Questions buyers ask before self-hosting Apicurio Registry
What drives the effort and cost of self-hosting Apicurio Registry?
The main drivers are the number of serializers and client languages that must switch to the registry, the storage backend you choose, how registry users authenticate, whether you need high availability, and whether you are migrating schemas from an existing registry. Each serializer and language adds compatibility testing, and a migration from another registry adds export, ID preservation and client cutover work. Costs fall into two groups: the one-time build (design, deployment, migration and testing) and the recurring work (upgrades, backups and rule governance). Ask any partner to price each driver separately in the written scope so you can see which items change if your inventory grows.
How long does setting up Apicurio Registry usually take?
No duration figure was supplied for this page, so treat any timeline as unconfirmed until it is agreed in writing. The schedule depends on the number of producer and consumer applications, whether schema IDs from an existing registry must be preserved, how many identity and network integrations are needed, and how quickly your teams can approve change windows. A single-team deployment with a small serializer inventory is usually scoped very differently from a multi-team cutover. Ask for milestones tied to named deliverables, and confirm which of them depend on your teams rather than on the implementation work.
What do we need to provide for a Apicurio Registry engagement?
The work needs a few named owners and concrete access before it starts. Assign one owner for the Kafka platform, one for identity and access, and one for each team whose serializers will change. Provide access to test Kafka clusters and a non-production registry environment, a complete inventory of producers and consumers with their language and serializer libraries, and an agreed list of change windows for cutover. Without the serializer inventory, compatibility testing cannot be scoped, so this is usually the first item to finalize.
Can our in-house team self-host Apicurio Registry without a partner?
Yes, a team with working Kafka operations skills and identity integration experience can deploy and run Apicurio Registry. The main gap is usually compatibility testing: confirming that every existing serializer, in every language your teams use, reads and writes the registry's Confluent-compatible API correctly under the compatibility rules you choose. Teams that have not done this before often underestimate the number of client versions in production. A partner is most useful when you need that testing scoped and evidenced quickly, or when you want a second review of storage, authentication and high availability choices.
Which parts of a Apicurio Registry deployment are one-time and which recur?
The one-time costs are design, initial deployment, the migration of schemas and client cutover, and the first round of compatibility testing. The recurring costs are upgrades, which you should plan against the Kafka and registry release cadence, backups of the registry's storage, and ongoing governance of content rules, reviews of schema changes and access reviews. Recurring work can be handled by your team or covered by a support retainer, which is available from DeployOpen. Ask for the recurring items to be listed separately in the quote, with the frequency each one is expected to run.
Does DeployOpen issue compliance attestations for a Apicurio Registry setup?
No. DeployOpen does not issue audit reports, attestations or certifications. It prepares systems, controls and evidence so that an independent auditor or certification body can assess them. For a schema registry, that typically means documenting who may register or change schemas, how compatibility rules are enforced, how access is authenticated and reviewed, how backups are tested, and how changes are recorded. Confirm with your auditor which evidence they expect before the build starts, because the expected format varies by framework and by organization.
What should we check before choosing a storage backend and authentication method?
Start with your existing operational standards. Check whether your platform team already runs a database service you can support for registry storage, and confirm the backup and recovery approach that service provides. For authentication, check which identity provider your organization uses and whether Apicurio Registry's supported options match your access model, since this must be confirmed against the current documentation and your own test environment. Also decide who may register schemas in production, because that decision shapes the authorization design more than the storage choice does.
How DeployOpen scopes and delivers a registry for your Kafka platform
DeployOpen plans and builds Apicurio Registry inside your own infrastructure, sized to the Kafka clusters and serializers your teams already run. Work starts with a written scope that fixes deliverables, the storage design, compatibility rules and the handover material before any build begins.
How we scope
- Inventory the Kafka clusters, serializers and client languages that will use the registry and which topics need schemas.
- Agree the storage backend, deployment target and whether existing Confluent-compatible clients will point at the registry.
- Fix the written scope covering authentication, content rules, high availability, deliverables and timelines before work starts.
How we build
- Deploy Apicurio Registry in your environment and wire it to your chosen storage and identity provider.
- Configure content rules and compatibility settings per subject, and test them against sample producer and consumer schema changes.
- Verify the Confluent-compatible API with the serializers your teams use, including schema ID behavior for existing records.
How we hand over
- Document the architecture, access model, rule configuration and backup and restore steps for the registry and its storage.
- Run a handover session with the platform, security and development teams covering how to add subjects and change rules.
- Deliver runbooks for upgrades, failover and rule review so your staff can operate the registry after the engagement ends.
How we support
- Ongoing support, upgrades and continuous testing of the registry are available as a retainer after handover.
- Retainer work can cover version upgrades, rule and access changes, and periodic verification of client compatibility.
- Ask for the support scope, response arrangements and upgrade approach in the written quote before committing.
