Migrate from Azure Event Hubs to self-hosted Apache Kafka: the short answer on effort
If your topics live in Azure Event Hubs and you want Kafka in infrastructure you control, the short answer is that MirrorMaker 2 can replicate your topics through the Event Hubs Kafka endpoint on port 9093 using SASL_SSL with the PLAIN mechanism. Microsoft documents this path because Event Hubs is compatible with the Apache Kafka protocol, and it states that replication can run in both directions. That means the real work is not moving bytes. It is replication setup, identity and credentials, consumer offsets, and the cutover order for producers and consumers.
Event Hubs is not Kafka in every respect, so do not assume a like-for-like swap. Before you plan, inventory the namespace: which event hubs are in use, their consumer groups, retention settings, partition counts, authorization rules and any Azure-side features that your applications depend on, such as capture or managed identity. Each of these needs a decision on the Kafka side. Confirm with Microsoft's documentation which Event Hubs behaviors are exposed through the Kafka protocol and which are not, since your replication design depends on that boundary.
Effort scales with the size of that inventory: the number of topics, consumer groups, client applications and the Azure-specific integrations around them. Once the inventory is complete and agreed, the work can be written as a fixed-scope engagement with deliverables and timelines set in writing before work starts. Ask for the inventory method, the list of Azure-dependent features, the replication and offset plan, and the cutover runbook to be part of any quote. Start with the Microsoft Learn MirrorMaker 2 tutorial to confirm the connection pattern against your own namespace.
What maps cleanly from Event Hubs to Kafka and what does not
This table pairs each Event Hubs concept with its closest Kafka equivalent and marks where the match is partial or missing. Read the rows marked as gaps first, because they decide which application code, Azure integrations and access processes must be rebuilt rather than moved. The main takeaway is that the streaming model carries over, while identity, capture and access control need explicit design on the Kafka side.
| Event Hubs concept | Kafka equivalent | What to confirm before cutover |
|---|---|---|
| Namespace | Kafka cluster (KRaft mode) with its listeners and bootstrap servers | Namespace is an Azure resource boundary, while a cluster is a set of brokers and controllers you size, patch and back up yourselves |
| Event hub | Topic | Retention, cleanup policy and partition count are set per topic in Kafka, so copy the Event Hubs settings explicitly |
| Partitions | Partitions | Partition count is fixed by you and affects ordering and parallelism, so confirm keys and count with consumers before replicating |
| Consumer groups | Consumer groups with committed offsets | Offsets are stored in Kafka, and MirrorMaker 2 can replicate group offsets, but consumers must still be pointed at the new cluster |
| SAS policies and connection strings | SASL users with SCRAM or PLAIN credentials, plus ACLs | Map each SAS policy to a principal and explicit ACLs; Kafka has no direct equivalent of SAS policy scopes, so confirm the permission model per team |
| Azure-native identity (Microsoft Entra ID, managed identities) | No direct equivalent in Kafka; OAUTHBEARER or Kerberos require an identity provider you run | Gap: decide which identity provider issues tokens or credentials for clients, and confirm how workloads will authenticate without Azure-managed identities |
| Event Hubs Capture to storage | No built-in equivalent; Kafka Connect sink connectors or custom consumers write to storage | Gap: Capture has no direct Kafka equivalent, so confirm the sink connector, file format and retention for archived data before cutover |
| Retention | Log retention and size-based limits per topic | Confirm the retention window and storage sizing against your replay needs, since storage now sits on infrastructure you control |
| ACL default behavior in KRaft | StandardAuthorizer stores ACLs in the metadata log; resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is true | Gap: create ACLs for every topic and group before switching clients, and keep this default unless you have a deliberate reason to change it |
| TLS and credentials | SASL_SSL with PLAIN sends credentials in a username and password exchange | PLAIN must run over TLS in every path; confirm certificate trust for all clients and rotate any credentials copied from connection strings |
Documented tools for moving Event Hubs data into Kafka
The migration path rests on a small set of documented tools: MirrorMaker 2 for data and metadata, Kafka clients for the cutover, and the admin tooling for topics and ACLs. Each one touches a different part of the move, so plan them together. Confirm early which Event Hubs features sit outside the Kafka protocol path.
MirrorMaker 2 as the replication path
MirrorMaker 2 is built on Kafka Connect and replicates topics with their configuration, consumer groups with committed offsets, and ACLs between clusters while preserving partitioning. For this migration it is the main tool that copies events from the Event Hubs Kafka endpoint into your cluster. Check that your topic-to-partition mapping on the target matches what consumers expect before you start the first mirror.
Microsoft's documented MirrorMaker 2 setup
Microsoft documents running MirrorMaker 2 against Event Hubs because the service speaks the Apache Kafka protocol. Connections go to the namespace endpoint on port 9093 using SASL_SSL with the PLAIN mechanism, and the same guidance states that replication can run in both directions. Use it as the reference for connection settings, then adapt the secrets handling to your own secret store.
Kafka client configuration for cutover
Producers and consumers move to your cluster by changing bootstrap servers, security protocol, and SASL settings. Event Hubs connection strings use the PLAIN mechanism, while your cluster may use SCRAM or another mechanism, so each application needs a tested client configuration. Confirm producer idempotence, consumer group IDs, and offset reset behavior before switching any workload.
Admin tooling for topics and ACLs
Topics, partition counts, and retention settings are created on the target cluster with the Kafka admin tooling, and access is granted through ACLs stored in the KRaft metadata log. MirrorMaker 2 can carry ACLs across, but review them rather than trusting the copy, because Event Hubs authorization is scoped to shared access policies. Without an explicit ACL or the allow.everyone.if.no.acl.found setting, resources are limited to super users.
Event Hubs features outside this path
The Kafka protocol compatibility covers produce and consume traffic, but Event Hubs has features that the Kafka endpoint does not replace. Capture, which writes events to storage, is one to check, along with any Event Hubs-specific features your applications call through Azure SDKs. Ask your Azure team to list every feature in use, then decide for each one whether it needs a replacement on your side.
Offset and cutover checks
Consumer offsets replicated by MirrorMaker 2 let consumers resume on the new cluster, but only if group IDs and topic names match. Before cutover, compare consumer lag and offsets on both sides, and plan a window where producers pause or dual-write. Record the offset state for each consumer group so you can roll back to the Event Hubs side if needed.
How the migration sequence runs: inventory, pilot, parallel run and cutover
The sequence moves Event Hubs traffic to a self-hosted Kafka cluster in stages, with each stage producing evidence that the next one can start.
Inventory namespaces, topics and clients
Your platform or integration team lists every Event Hubs namespace, event hub (topic), partition count, retention setting, consumer group, producer application and shared access signature policy. Each producer and consumer is tagged with its owner and its cutover window. The step is done when the inventory is reviewed by the application owners and every client that connects through the Kafka endpoint or the Event Hubs SDK is accounted for. Ask your Azure administrators to confirm which policies grant send and listen rights, because those rights must be recreated as Kafka credentials and ACLs.
Build the target cluster with KRaft and TLS
The platform team deploys the self-hosted Kafka cluster in KRaft mode with TLS on client listeners, authentication enabled and the StandardAuthorizer configured for ACLs. Topics are created with partition counts and retention matched to the inventory, not copied blindly, because partition counts affect ordering and parallelism. The step is done when a test client can produce and consume over the secured listener and the ACLs deny access by default to any identity not granted it.
Pilot one low-risk topic
Choose a topic with a single producer, a single consumer group and tolerance for brief interruption. Run MirrorMaker 2 from the Event Hubs namespace to the new cluster, using the Event Hubs Kafka endpoint on port 9093 with SASL_SSL and the PLAIN mechanism as Microsoft documents, and check that records, topic configuration and consumer offsets arrive as expected. The step is done when message counts, key-to-partition placement and consumer group offsets match between the two sides for a full business cycle agreed in advance.
Run both platforms in parallel
Expand replication to the remaining topics in waves. Consumers on the new cluster read replicated data and compare outputs against consumers still on Event Hubs, while the offset translation for each consumer group is checked before anyone relies on it. The step is done when output reconciliation shows no unexplained gaps, duplicates or ordering differences, and the owners of each downstream system sign off on the comparison.
Switch consumers before producers
Move consumers to the Kafka cluster one group at a time, starting from the committed offsets that MirrorMaker 2 has translated. Keep each group's previous configuration ready so it can be restored quickly. The step is done when each consumer group reports steady lag on the new cluster and no application is still reading from Event Hubs for the topics it owns.
Switch producers in stages
Change producer applications to write to the self-hosted cluster one service at a time, verifying that records appear on the new side and that downstream consumers continue to process them. Because Microsoft documents replication in both directions, MirrorMaker 2 can run from the new cluster back to Event Hubs during this period, which gives a reverse path if a producer must return to the old endpoint. The step is done when every producer writes to the new cluster and the reverse replication has been tested for at least one producer.
Hold a rollback window and then decommission
Keep Event Hubs namespaces and the reverse replication available for a rollback window that the business owners set in writing before cutover. Confirm that no traffic remains on the old endpoints, that retained data needed for audit or replay has been exported, and that access policies are revoked. The step is done when the rollback window closes without a rollback, the old namespaces are deleted or archived under your retention rules, and the inventory is updated to show the final state.
How consumer offsets and in-flight messages move between clusters
MirrorMaker 2 replicates consumer groups along with their committed offsets, so a consumer group that has read from an Event Hubs consumer group can resume on the Kafka cluster at a mapped position instead of starting from the beginning or the latest offset. The mapping is not a one-to-one copy of numbers, because the source and target partitions do not keep identical offsets. Confirm in the MirrorMaker 2 geo-replication documentation how the offset translation and checkpoint topics work for your MirrorMaker version and group settings. Then test each consumer group on its own: start it on the target from the translated position in a non-production copy, compare what it reads against what the source delivered, and record the order in which groups switch. Groups that feed each other, or that share a downstream table, should move together in one planned window.
Before any switch, pause or stop producers on Event Hubs, or move them to the new cluster in a controlled order, and then drain. Drain means waiting until the committed offset of each consumer group on the source equals the log end offset of every partition it reads, so lag is zero. Check lag per group and per partition on both sides, not only the total, because one idle partition can hide a stalled consumer. Only then start consumers on the target. Record the translated offsets you used, so you can explain any gap or repeat later.
Kafka delivers at least once by default, so a consumer that processed messages but failed before committing its offset will see them again after a switch. Expect some duplicates at the boundary, and design handlers to tolerate them through idempotent writes, deduplication keys or a short replay window that your team has tested. Check which of your consumers rely on exactly-once behavior, and confirm with your application owners how they handle reprocessing before the cutover date.
What teams ask before moving off Event Hubs to Kafka
What drives the cost and effort of moving off Event Hubs to Kafka?
The biggest drivers are the number of Event Hubs namespaces, the count of event hubs (topics) and their partition counts, how much identity and access work is needed, and how many producers and consumers must be repointed. Each namespace and event hub needs an inventory entry, a target topic with matching partitioning, and a decision on how its consumer groups and offsets move. Identity rework is often the largest unknown, because Event Hubs access is usually built on Azure identities or connection strings that must be replaced with Kafka authentication and ACLs on your cluster. Costs fall into two groups: one-time work (inventory, design, build, cutover, testing) and recurring work (running the cluster, patching, monitoring, and the infrastructure you pay for in your own accounts). Ask any partner for a quote built from a written inventory, listing each component, its owner and its test plan.
How long does a migration from Event Hubs to self-hosted Kafka take?
The duration depends on your inventory and on how long both platforms must run in parallel, so no fixed timeline applies to every estate. A small set of event hubs with a few consumers can cut over in a short, contained window, while many namespaces, strict replay requirements or consumers you do not own usually extend the schedule. The parallel run is the main variable: MirrorMaker 2 can replicate topics, consumer group offsets and ACLs between clusters, which lets you move consumers in stages, but each stage needs validation. Your team should confirm the inventory, the cutover order and the rollback window before anyone commits to a date. A written scope from DeployOpen will set the timeline for your specific estate.
What must our team provide before a migration starts?
Your team needs to provide a complete inventory and clear owners. The inventory should list every namespace, event hub, consumer group, producer application, and any downstream system that reads from Event Hubs. Owners must be named for each producer and consumer, because cutover decisions depend on them. You also need access to the Azure subscription that holds the namespaces, with rights to read configuration and, where needed, to create the replication path, and access to the Kafka cluster you will run. Test traffic is essential: representative messages and a non-production replay window let the team confirm ordering, throughput, and offset behavior before production cutover. Finally, decide who signs off on each stage.
Can our own team do this migration without a partner?
Yes, a team with working Kafka skills can plan and run this migration in-house, provided it has time for the inventory, the identity redesign and the parallel run. The Event Hubs Kafka endpoint, MirrorMaker 2 and Kafka's own security model are documented, so the required steps are known. A partner adds value when the team lacks recent Kafka operations experience, when the estate is large, or when the migration must fit a compliance evidence trail. DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms in your own infrastructure, and most engagements are fixed-scope with agreed deliverables. Whichever path you choose, your team keeps ownership of the running cluster.
Does Apache Kafka have licensing costs?
Apache Kafka is an Apache Software Foundation project released under the Apache License 2.0 (SPDX identifier Apache-2.0), and the project itself offers no paid edition. Your software cost for the broker and the tools that ship with it is therefore zero in licensing terms. Costs that remain are the infrastructure you run it on, the staff who operate it, and any third-party components you choose to add, such as connectors or a schema registry, which carry their own licenses. Check those licenses separately before adopting them. Note that this page does not cover Event Hubs pricing, so compare your current Azure bill with your own infrastructure and staffing estimates.
How do consumers keep their position when they switch clusters?
Consumers keep their position through consumer group offsets, which MirrorMaker 2 can replicate along with the topics. The replicated offsets are translated for the target cluster, so a consumer group can resume near where it stopped. The exact behavior depends on how far replication has progressed when you stop producing to the old cluster, so plan a drain period and verify lag before switching. Confirm with your test environment that offsets land where you expect before relying on them in production. Keep the old namespace readable during the rollback window so you can return if a consumer misbehaves.
What access controls must we rebuild on the Kafka side?
Event Hubs access is tied to Azure identities and shared access policies, so those must be replaced with Kafka's own controls. Kafka supports SSL and SASL authentication, with GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER as the documented SASL mechanisms. In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Map each Event Hubs policy to a Kafka principal and a narrow ACL set, then test every producer and consumer against the new cluster. Confirm which mechanism your identity provider can support before you choose one.
What usually goes wrong in an Event Hubs to Kafka move
Most failed moves off Event Hubs are not caused by Kafka itself. They come from credentials, access rules, offsets, partition layouts, versions and rollback plans that were assumed rather than tested. Each card below names one of these risks and the check that prevents it.
Credentials sent without TLS
Event Hubs connection strings are used as SASL PLAIN passwords over port 9093 with SASL_SSL. On a self-hosted cluster, PLAIN credentials are only protected if every listener that carries them uses TLS, so an unencrypted internal listener can expose them on the network. Confirm that each client and MirrorMaker 2 listener uses SASL_SSL, that no plaintext listener accepts PLAIN, and that connection strings are moved into a secrets store rather than config files.
ACLs left at default restriction
In KRaft clusters, the StandardAuthorizer restricts resources without ACLs to super users unless allow.everyone.if.no.acl.found is set to true. Teams that cut over expecting Event Hubs-style access find consumers and producers failing on their first connection. Before cutover, write ACLs for every topic, consumer group and transactional ID, and test each application identity against the new cluster rather than relying on a broad default.
Consumer offsets not verified
MirrorMaker 2 replicates consumer groups and their offsets, but offsets are only useful if they line up with the messages that were actually copied. A consumer that resumes from a stale or mismatched offset can skip or reprocess events. Compare committed offsets and lag for each group on both sides before the switch, and agree with each consuming team on how duplicates will be handled during the first hours after cutover.
Partition mismatch changes ordering
Event Hubs partitions and self-hosted Kafka partitions are sized separately, and MirrorMaker 2 preserves partitioning from source to target. If the target topic has a different partition count, or producers use a different key or partitioner, messages with the same key can land on different partitions and lose their order. Record the source partition count and key scheme for every topic, and confirm the target layout matches before any producer switches.
Kafka and Java requirements missed
Since Kafka 4.0, brokers, Kafka Connect and tools require Java 17, while clients and Kafka Streams applications need at least Java 11. Older application runtimes, or a Connect worker on Java 11, will fail after the platform is built. Inventory the JVM version of every producer, consumer, Streams job and Connect worker, upgrade what is below the minimum, and confirm the Kafka release you deploy is one the upgrade notes support for your starting version.
No tested rollback
Cutover plans often assume that switching producers back to Event Hubs is always possible, but once producers write only to the new cluster, events written there are not in Event Hubs. A rollback needs replication running in the reverse direction, which Microsoft documents as possible for MirrorMaker 2 with Event Hubs, and a rehearsed switch-back procedure. Rehearse the full rollback in a non-production environment, time it against your agreed outage tolerance, and name the owner who decides to roll back.
What drives the cost and duration of the move
This table lists the workstreams that set the cost and length of an Event Hubs to self-hosted Kafka move, who is paid for each, and whether the cost is one-time or recurring. Use it to build a quote request that compares like with like, and to see which items a partner should price separately from the platform build.
| Driver | Why it changes cost or effort | One-time or recurring, and what to ask for in a quote |
|---|---|---|
| Namespaces and event hubs in scope | Each namespace and hub adds topics, producer and consumer inventory, and cutover steps to plan and verify. | One-time. Ask for the inventory method and a per-namespace breakdown of the work. |
| Partitions and throughput per topic | Partition counts and peak rates set broker sizing, replication settings and how long mirroring takes to catch up. | Mostly one-time for sizing, recurring for the infrastructure it requires. Ask how sizing assumptions are documented and tested. |
| Consumer groups and offset state | Each group needs its offsets carried over or reset deliberately, and each reset needs an application owner to agree. | One-time. Ask how groups are mapped, who signs off on offset handling and how replay is tested. |
| Identity and SAS policy rework | Shared access signature policies and connection strings must be replaced with Kafka users, SASL credentials and ACLs on the new cluster. | One-time build, with recurring credential rotation. Ask how secrets are issued, stored and rotated and who owns each identity. |
| Kafka platform build (KRaft, TLS, ACLs) | Controllers, brokers, listeners, certificates and authorization must be built and hardened before any data moves. | One-time build, recurring operation. Ask for the deployment method, the hardening baseline and the operations handover. |
| Kafka Connect and downstream integrations | Connectors, sinks and stream processors that read from Event Hubs must be repointed, rebuilt or retired. | One-time for rework, recurring for connector hosting and licensing. Ask which connectors exist and which versions they require. |
| Testing and parallel run length | Both systems run side by side while producers, consumers and replication are checked, and the parallel period sets the duration. | One-time, but it extends calendar time and the cost of running two platforms. Ask for the exit criteria and the length of the parallel run. |
| Compliance evidence | Access logs, retention settings, encryption configuration and change records must be collected for assessors, and the platform must match the controls it claims. | One-time to prepare, recurring to maintain. Ask which frameworks apply and what evidence format the assessor expects. |
| Azure cost tail during cutover | Event Hubs stays billed until producers, consumers and replication are all moved and the namespaces are retired. | Recurring until retirement. Confirm the retirement date with your own Azure billing owner, since it depends on your cutover plan. |
What your team provides and what DeployOpen delivers
The migration starts with a complete inventory of what Event Hubs carries today. Your team should list every event hub in each namespace, its partition count, retention setting, capture configuration, and each producer and consumer application with its consumer group and checkpoint store. Name an owner for every topic who can answer questions about message schemas, ordering needs and acceptable duplicate handling. Without those owners, the cutover plan has no one to approve it. Also provide access to the namespace with rights to read configuration and to create or rotate shared access signature policies, since the Kafka endpoint authenticates with SAS credentials. Confirm which Event Hubs features your applications rely on, because features outside the Kafka protocol will need separate replacements on the target side.
On the target side, your team provides the infrastructure where the self-hosted Kafka cluster will run, along with network paths from that cluster to the Event Hubs namespace on port 9093 for the MirrorMaker 2 phase. Confirm firewall rules, DNS resolution for the namespace, and where secrets for the replication connectors will be stored. Name the people who approve test windows, the production cutover date and the rollback decision, and make sure they are available for the cutover period. Agree on how long Event Hubs and the new cluster will run in parallel, since that period drives both your cloud spend and the coordination effort.
DeployOpen plans, builds and documents the work inside your environment, and most engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. The written scope should name the replication design, the topic and consumer group mapping, the offset translation and switchover runbook, the test evidence and the handover material. Ask for the scope to state which systems are in and out, which producer and consumer changes DeployOpen will make and which your teams will make, and what happens if an approver is unavailable on the planned date. Ongoing support, upgrades and continuous testing are available as separate retainers, so ask for those to be quoted separately from the migration itself.
How DeployOpen scopes, delivers and supports the migration
DeployOpen plans, builds, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For this migration, we start from a written scope covering your Event Hubs namespaces, clients and cutover window, then deliver the Kafka side and the MirrorMaker 2 replication under that agreement.
How we scope
- Inventory every Event Hubs namespace, event hub, consumer group and producer that must move.
- Agree in writing which topics, partition counts, retention settings and access rules the new cluster will carry.
- Confirm whether replication must run in one direction for cutover or in both directions during a rollback window.
How we build
- Build a KRaft-mode Kafka cluster in your infrastructure with TLS listeners and SASL authentication configured.
- Configure MirrorMaker 2 with SASL_SSL and PLAIN against the Event Hubs Kafka endpoint, then test replicated topics and offsets.
- Test producer and consumer switchover in a staging environment before any production client is moved.
How we hand over
- Deliver runbooks for cutover, offset checkpoints, rollback steps and credential rotation for the new cluster.
- Document topic configuration, ACLs and replication settings so your team can review and change them later.
- Walk your platform and application owners through the cutover results and the open items before sign-off.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after the migration closes.
- Plan Kafka version upgrades against the supported release line and the upgrade notes for your cluster.
- Review access, replication health and monitoring findings with your team on the agreed retainer schedule.
