Migrate from IBM MQ to Apache Kafka: what the move involves and what drives the effort
A team that runs IBM MQ queues and wants Kafka for streaming or event distribution usually moves in stages, with MQ kept running while applications shift one group at a time. The IBM MQ source and sink connectors bridge the two during cutover: the source connector moves messages from MQ queues into Kafka topics, and the sink connector moves Kafka records back into MQ for consumers that have not yet been rewritten. Running both directions is normal during the transition, so the plan should treat MQ as a live dependency until the last consumer has moved, not as a system to switch off on a set date.
The effort depends on three things you can measure before any build starts. The first is queue count and ownership: each queue needs a decision about whether it maps to a topic, is drained by a bridge, or is retired, and each consuming application needs a named owner for its cutover. The second is message format. The connector converters handle binary, text, JSON and XML payloads, but a format that depends on MQ-specific headers, message properties or a custom serializer needs its own mapping and test cases. The third is the delivery guarantee each flow requires. Exactly-once source mode is off by default and needs Kafka Connect 3.3.0 or later in distributed mode, exactly.once.source.support enabled, a dedicated MQ state queue and tasks.max set to 1, so a flow that needs it carries extra capacity and ordering constraints. Confirm the connector version against the connector repository and your MQ version before sizing.
Which IBM MQ capabilities map cleanly to Kafka and which do not
This table compares core IBM MQ concepts with their closest Kafka equivalents and marks where the mapping is direct, partial or needs redesign. Read each row as a design question to settle with your application owners before the connector mapping is fixed. The main takeaway is that ordering, acknowledgment and selector behavior need explicit decisions, not a one-to-one translation.
| Capability | IBM MQ | Apache Kafka |
|---|---|---|
| Destinations | Queues for point-to-point delivery and topics for publish-subscribe, each with its own ownership and attributes | Topics only, with consumer groups deciding whether consumers share work or each receive every record |
| Consumption model | Each message is delivered to one consumer on a queue, and a consumer that gets it removes it from the queue | Each consumer group reads a partition in order and tracks its own offset, so records remain in the log after being read |
| Message ordering | Ordering is generally preserved on a single queue for one consumer, but parallel consumers on one queue can reorder | Ordering is guaranteed only within a partition, so the record key and partition count must be chosen to keep related messages together |
| Selectors | Consumers can filter with message selectors on properties or headers | No broker-side selector equivalent in core Kafka, so filtering moves to consumers, sink-side logic or separate topics. Confirm connector filter options in the connector documentation |
| Request-reply | Reply-to queues and correlation IDs are a common pattern in MQ applications | Works with a reply topic and correlation key, but needs explicit timeout, cleanup and consumer ownership design |
| Acknowledgment | Messages are removed on get within a syncpoint unit of work or after a confirmed commit | Consumers commit offsets, and at-least-once replay is the default unless the pipeline uses transactions or idempotent processing |
| Persistence and retention | Messages persist until consumed or expired, with queue depth limits set per queue | Records persist for a configured retention period or size, independent of how many consumer groups have read them |
| Message formats | Text, binary and structured payloads with MQ message descriptors carrying headers and properties | Keys and values are bytes, so converters and header mapping must be defined for binary, text, JSON and XML. Confirm supported converters in the connector documentation |
| Transactions and exactly-once | Units of work can span several queue operations under syncpoint | Idempotent producers and transactions cover Kafka writes, while MQ source exactly-once needs a dedicated state queue and Connect settings |
Documented connectors and converters for the MQ bridge
The IBM MQ bridge is built from open-source connectors maintained in the ibm-messaging project, so the connector choice and converter choice are the first design decisions. Each card below covers one part of that choice, from licensing and version support to payload converters and the separately published sink.
Source connector licensing and versions
The IBM MQ source connector is licensed under Apache 2.0, so the connector itself carries no license fee. It supports IBM MQ v8 or later and Apache Kafka 2.0.0 or later. Check these version floors against your own queue managers and your Kafka Connect runtime before you commit to the bridge, and confirm the current support statement in the repository, because the bridge will run on that combination for the length of the migration.
Separately published sink connector
A sink connector for IBM MQ is published as a separate artifact from the source connector, in the same ibm-messaging project. Teams that need to write Kafka topics back into queues during a staged cutover must install and version it independently. Confirm its supported MQ and Kafka versions in the repository rather than assuming they match the source connector.
Binary payload handling
Binary messages, such as packed or compressed structures, should not pass through a text converter, because the bytes can be altered or rejected. Decide whether the payload is stored as opaque bytes for consumers that already understand the format, or whether a schema-aware consumer will decode it. Test a sample message from each binary format end to end, including its length and trailing bytes.
Text and JSON converter choices
Text payloads need an explicit character encoding decision, usually UTF-8, agreed with the producing applications before the bridge starts. JSON payloads can be treated as strings to preserve exact bytes or parsed into structured records for downstream transforms. Each choice changes how consumers see fields and whitespace, so record the decision per queue and verify it with representative messages.
XML payload handling
XML messages often carry namespaces, attributes and ordering that a parser may normalize. Decide whether to keep the document as text to preserve its exact form, which is usually safer for signed or legacy payloads, or to parse it for transforms. Validate that the converter output still matches the schema your consumers expect, and confirm the behavior against the repository before relying on it.
Confirming converter behavior
Converter behavior is set by configuration keys and the connector version, so assumptions from other projects can be wrong. Pin the connector version, run a test queue through each converter, and compare record bytes, headers and keys in the target topic. Ask your DeployOpen contact to write these checks into the cutover plan so the converter choice is evidenced before any application moves.
What exactly-once source mode requires before you turn it on
Exactly-once delivery in the IBM MQ source connector is off by default, so it is a deliberate choice that needs several pieces in place before it works. The connector needs Kafka Connect 3.3.0 or later running in distributed mode, with exactly.once.source.support enabled on the workers. Standalone mode does not meet the requirement, so the platform team should confirm the Connect runtime, worker configuration and rollout order before any MQ queue is attached. Because the set of supported Connect versions changes over time, check the exact requirements against the ibm-messaging/kafka-connect-mq-source repository at the version you plan to deploy.
The connector also needs a dedicated MQ state queue, separate from the queues carrying business messages, and tasks.max set to 1. The single task is the main trade-off. One task means one consumer of the source queue, so throughput is capped by what that task can read, convert and produce, and adding partitions in Kafka does not change it. Estimate the message rate and peak burst the migration must absorb before you commit to exactly-once, and compare that with the at-least-once alternative, which lets you run more tasks but requires idempotent consumers downstream. Ask IBM or your MQ administrator to confirm the message sizes and peak rates for each queue in scope.
Treat the state queue as an operational component, not a configuration detail. Give it a named owner, a capacity and depth alert, and a documented procedure for what happens if it is purged, lost or restored from an older backup, because the connector's delivery guarantee depends on the state it records there. Include the state queue in your MQ backup scope and in your cutover runbook, and test a connector restart and a Connect worker failure in a non-production environment before the first production queue is moved.
How a staged cutover from IBM MQ to Kafka runs
The sequence moves IBM MQ traffic to Kafka one flow at a time, with reconciliation checkpoints and a rollback path at each stage so that MQ is retired only after the Kafka side has been verified against it.
Inventory queues, channels and applications
Build a register of every queue manager, queue, channel, listener and application that produces or consumes messages, including message formats, persistence settings, expected volumes and ordering needs. Owners of each application confirm their consumers and any dependencies on transactions spanning several queues. The step is done when every flow is assigned a migration wave and an owner, and the team can state which flows are candidates for the source connector and which need a different design. Ask the platform team to confirm the MQ version and the connector support matrix before the register is signed off.
Build a pilot topic with the source connector
Choose one low-risk, non-transactional queue and deploy the IBM MQ source connector on the Kafka Connect cluster, with the converter settings chosen for that message format. Create the target topic with planned partition count, retention and access control, and set the connector to copy messages from the queue into it. The step is done when test messages arrive in the topic with the expected key, headers and payload, and a consumer reads them correctly. Record the MQ queue depth and the topic record count for the same test window so the first reconciliation has a baseline.
Decide delivery semantics for each flow
For each pilot flow, decide whether at-least-once delivery with idempotent consumers is acceptable or whether exactly-once source mode is required. Exactly-once mode is off by default and needs Kafka Connect 3.3.0 or later in distributed mode, with exactly.once.source.support enabled, a dedicated MQ state queue and tasks.max set to 1. Architects and application owners sign the decision in the register, and the state queue is provisioned and tested before any production flow depends on it. Confirm the throughput limit that a single task imposes on the flow before accepting that setting.
Run a parallel period with reconciliation
Run MQ and Kafka side by side, with the connector copying production messages into Kafka while existing consumers stay on MQ. At agreed intervals, compare message counts between the source queue, the topic and any downstream sink, and check offsets in the connector state against the message identifiers. Investigate every difference, including duplicates, gaps and converter errors, before the flow advances. The checkpoint closes when counts match within the tolerance the application owners accepted and the error log is empty for the agreed window. Keep the connector ready to be paused so that the parallel run can be stopped without affecting MQ.
Cut over consumers one application at a time
Move each consuming application to read from the Kafka topic, starting with the least critical one. Run the new consumer in a shadow mode first if the application allows it, comparing its output with the output of the MQ consumer. Switch the live consumer only after the comparison matches, and keep the MQ subscription in place for the rollback window. The cutover is complete for that application when it processes live traffic from Kafka, its consumer group lag stays within the limits the owner set, and its business outputs reconcile with the previous period.
Define and test rollback for each stage
Document how to return each flow to MQ at every stage: pausing the connector, re-enabling the MQ consumer, and handling messages already in Kafka that have not been consumed. Rehearse the rollback in a non-production environment and record the steps the operations team followed and any gaps they found. Confirm that messages copied to Kafka during a rollback will not be processed twice by consumers that have been switched back to MQ. The step produces a signed rollback runbook for each application and a tested result for each stage.
Stop MQ traffic and decommission queues
After every consumer for a flow has run on Kafka through the agreed verification period, stop the producers that write to the MQ queue and confirm the queue stays empty. Keep the queue manager and its channels available until the retention and audit needs of the business are met, then remove the queues, channels and listeners in a change-controlled window. The step is done when the register shows no remaining consumers or producers for the flow, the final reconciliation report is archived, and the removal is recorded in the change log.
Preparing the Kafka platform before connectors go live
Before any MQ connector is deployed, the target Kafka cluster must already be running in KRaft mode, because Apache Kafka 4.0 removed ZooKeeper mode entirely. Choose one of the supported 4.x releases listed on the Kafka downloads page, currently 4.3.1, 4.2.2 and 4.1.2, and confirm the version against that page on the day you build. If any existing cluster in your estate still depends on ZooKeeper, that migration has to finish first, and the upgrade notes recommend moving KRaft clusters older than 3.3.x to 3.9.x before going to 4.x. Confirm the Java runtime too: brokers, Kafka Connect and tools need Java 17 from Kafka 4.0 onward, so the Connect worker hosts and the broker images should be checked for it before anything else is installed.
Decide the listener and authentication model next. Kafka supports SSL and SASL, with SASL mechanisms GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER, and the Connect worker needs its own principal for both the MQ-side work and the Kafka-side producer. Plan TLS on every listener that carries MQ-derived data, since security is optional in Kafka and a mixed secured and unsecured listener setup is easy to leave behind by accident. Authorization in KRaft is handled by the StandardAuthorizer, which stores ACLs in the metadata log. The critical day-one risk is that resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Create the Connect principal, its topic and group ACLs, and the consumer ACLs for downstream applications before the first connector starts, and test each one with a non-super user account.
What drives the cost and duration of the migration
This table lists the factors that move the effort of an IBM MQ to Kafka migration, who carries each cost, and whether it is one-time or recurring. Use it to check that a written quote covers each driver before you compare options.
| Driver | Why it changes effort | One-time or recurring | What to ask for in a quote |
|---|---|---|---|
| Queues, channels and topic mapping | Each queue or channel that is bridged needs a source connector task, a topic mapping, ACLs and monitoring. Many small queues cost more than a few large ones. | One-time build, then recurring if new queues are added | A per-queue inventory with the mapping to topics, owners and a list of excluded queues |
| Message formats and converters | Binary, text, JSON and XML payloads need different converter and transform work. XML and mixed formats usually take the most testing. | One-time build and test | Which converters are configured for each queue, and who validates sample messages |
| Exactly-once requirements | Exactly-once source mode needs Kafka Connect 3.3.0 or later in distributed mode, exactly.once.source.support enabled, a dedicated MQ state queue and tasks.max set to 1. Each of these shapes the design and throughput. | One-time design and test, with ongoing capacity for the single task | Written statement of which flows need exactly-once, the state queue design and the throughput limit accepted |
| Consumer count and rewrites | Each MQ consumer that moves to Kafka may need new offset handling, retry logic and idempotent processing. Applications that rely on MQ transactions or message selectors need the most rework. | One-time application work, owned by the application teams | A consumer list with the change type for each (no change, configuration, code rewrite) |
| Throughput and ordering | Ordering in MQ is per queue. Kafka orders within a partition, so the partition key and count decide whether order is kept. Higher peak rates need more connector tasks and broker capacity. | One-time design and load test, recurring broker capacity | Peak message rates per queue, the ordering rule for each flow and the load test scope |
| Security and certificates | Connector connections to queue managers need TLS and channel authentication settings. Kafka side needs SSL or SASL and ACLs in the metadata log. Certificate rotation adds recurring work. | One-time setup, recurring certificate renewal and access reviews | The auth model for MQ and Kafka, certificate sources, rotation schedule and who approves ACL changes |
| Connector and Kafka operations | Connectors, Kafka Connect workers, brokers and monitoring must be patched, upgraded and watched after cutover. Operating cost continues for as long as the bridge runs. | Recurring, based on the chosen operating model | Who patches Connect and brokers, the alerting scope and the support hours included |
| Length of the parallel run | Running MQ and Kafka side by side keeps both platforms, licenses and runbooks alive. Each extra week of overlap adds cost, and rollback needs a tested path. | Recurring for the overlap period | The exit criteria for each stage, the planned overlap length and the rollback test scope |
What usually goes wrong when moving IBM MQ traffic to Kafka
Most IBM MQ to Kafka migrations fail in the handoff rather than in the connectors themselves. The risks below cluster around delivery guarantees, message ordering, payload handling, access control and the decision to retire MQ. Each card names the check to run before the next cutover stage.
Duplicate or lost messages at cutover
During a staged cutover, the same message can be consumed from MQ and from Kafka, or a message can be removed from MQ before its Kafka record is durably written. Both outcomes depend on how the source connector acknowledges messages and whether exactly-once mode is enabled. Define in writing which system is authoritative for each application before any traffic moves, and test a duplicate injection and a broker restart during the bridge period. Give each consuming application an idempotency key so replays are harmless.
Assuming MQ ordering across queues
MQ guarantees ordering within a queue for a single consumer, but applications often assume an order across several queues that feed one business process. A Kafka topic with partitions keeps order only within a partition, so messages that were once sequenced by queue can arrive interleaved. Inventory the ordering assumptions per flow, then choose a partition key that matches each business entity. Document any flow where cross-queue order cannot be preserved and handle it in the consumer.
Underestimating single-task exactly-once throughput
Exactly-once delivery in the MQ source connector is off by default and runs with tasks.max set to 1, so one task carries the full MQ throughput for that connector. Load that is comfortable with several parallel tasks may saturate a single task during peaks. Measure sustained message rate and message size against a representative queue before committing to exactly-once for high-volume queues. Where throughput is insufficient, split the traffic across queues with separate connectors or accept at-least-once with idempotent consumers.
Converter mismatches for XML and binary payloads
The source connector maps MQ message bodies to Kafka records through converters, and a mismatch shows up as corrupted, truncated or unreadable payloads. Binary messages need a byte-preserving path, text needs an explicit character set, and XML or JSON may be altered if a transform reserializes them. Include a sample of each payload type from production in the test set, compare byte-level output at the sink, and set the converter explicitly per topic rather than relying on defaults.
Missing ACLs blocking connectors
When the cluster uses a StandardAuthorizer in KRaft mode, resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. A connector principal without topic, group and cluster grants will fail on startup or when writing to its internal topics, which can look like a connector bug. Create the connector principal, grant write access to the target topics and the Connect internal topics, and test authorization with each connector before the first cutover stage. Keep the permissive default off in production.
Retiring MQ before reconciliation
Teams often switch off a queue manager once the Kafka path looks healthy, before message counts and business totals have been compared. Missing or duplicated messages then surface days later with no source to replay from. Keep MQ channels and queues available until reconciliation confirms matching counts, sampled payloads and closed business transactions for each flow. Set the retirement gate in the cutover plan with a named owner who signs off on the reconciliation evidence.
Questions buyers ask before migrating IBM MQ to Kafka
What drives the cost of moving IBM MQ traffic to Kafka?
The cost depends on how many queues, topics, applications and message formats are in scope, and on how much of the existing MQ estate must keep working during the move. Kafka and the IBM MQ connectors are open source, so the main costs are people and infrastructure. Your team or a partner pays for discovery, connector configuration, converter work for binary, text, JSON and XML payloads, testing, and cutover. Infrastructure is a recurring cost for the Kafka brokers, Kafka Connect workers, monitoring and backups. Ask any quote to split one-time migration work from recurring operations, and to state which queues and applications it covers, which formats need custom converters, and whether exactly-once delivery is in scope. The Apache Kafka project licenses its software under Apache 2.0 with no paid edition from the project itself, so licensing is not usually the driver.
How long does an IBM MQ to Kafka migration take?
Duration depends on the number of queues and applications, the complexity of message formats, the number of consumers that must be rewritten or re-pointed, and how many cutover stages your business accepts. Discovery and inventory usually take longer than teams expect because queue ownership and consumer behavior are often undocumented. Exactly-once requirements, custom converters and security integration add time. Ask for a schedule with milestones tied to written scope rather than a single end date, and ask which phases depend on your change windows and approvals. No duration is given here because the work varies widely between estates, so a written scope is the only reliable basis for a timeline.
What must our team supply before the work starts?
The team needs to supply a queue inventory and an application map before design can begin. The inventory should list each queue and topic, its queue manager, its message rate and size, its persistence settings, and its owner. The application map should show which producers and consumers use each queue, how they connect, and what they expect from message order and redelivery. Message samples are essential for choosing converters. Provide representative samples of binary, text, JSON and XML payloads, including headers and character encodings. You also need access to a non-production MQ environment, a named contact for each consuming application, and an agreed set of acceptance tests. Missing samples are one of the most common reasons converter design slips.
Can our internal team do the migration without a partner?
An internal team can do the migration if it has experience with IBM MQ administration, Kafka operations, Kafka Connect and the applications involved. The hardest parts are usually the ones that touch production behavior: deciding how each queue maps to topics and partitions, configuring exactly-once source mode, securing the connectors, and proving that each consumer behaves correctly during the parallel run. Ask yourself whether the team has capacity to run discovery, build, testing and cutover alongside existing duties, and whether it can staff a rollback plan. A partner can take on any subset of these tasks under a fixed scope, or support your team through a single phase. Use the written scope to decide which parts to keep in-house.
Can IBM MQ stay in service during the parallel run?
Yes, the usual approach is to keep IBM MQ running while Kafka receives a copy of selected traffic and applications move over one stage at a time. The IBM MQ source connector reads messages from MQ queues into Kafka topics, so producers and consumers can be switched in stages while both platforms run. The connector can be configured to read from queues that remain in use, but you must confirm the delivery semantics for each queue, because duplicate or out-of-order delivery during overlap can affect downstream systems. Plan for a dedicated MQ state queue when exactly-once mode is used, and test the duplicate-handling behavior of each consumer before any stage goes live. Each cutover stage should have a documented rollback path back to MQ.
Does exactly-once delivery need special setup for the MQ source connector?
Yes. Exactly-once delivery in the IBM MQ source connector is off by default. To enable it, you need Kafka Connect 3.3.0 or later running in distributed mode, with exactly.once.source.support enabled, a dedicated MQ state queue, and tasks.max set to 1. Those requirements affect throughput, because a single task limits parallelism for that connector, so test the rate you need before committing to exactly-once for high-volume queues. Confirm the requirements against the connector repository before you design the deployment, since connector behavior and supported versions can change. Decide per queue whether exactly-once is necessary or whether consumers can tolerate duplicates.
What security controls must the migration design cover?
The design must cover authentication and encryption for every connection: from the connector to IBM MQ, from Kafka Connect to the brokers, and from consumers to Kafka. Kafka supports SSL and SASL, and documented SASL mechanisms include GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER, so choose the mechanism your identity platform supports. In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Build ACLs for each topic and connector before cutover, and check that MQ channel authentication and credentials are reviewed as part of the same plan. Confirm how secrets for connector configuration are stored and rotated in your environment.
How DeployOpen scopes and delivers an IBM MQ to Kafka migration
DeployOpen approaches an IBM MQ to Apache Kafka migration as a written, staged project inside your own infrastructure. We map which queues and message formats move, build the Kafka Connect pipeline and cutover plan, and hand over runbooks your team can operate. Engagements are fixed-scope, with ongoing support available as a retainer.
How we scope
- The written scope lists which MQ queues, channels and applications are offloaded to Kafka in each stage.
- Message formats are catalogued per queue: binary, text, JSON or XML, with the converter choice recorded for each.
- The scope states whether exactly-once source mode is required and names the dedicated MQ state queue it will use.
How we build
- We deploy Kafka Connect in distributed mode with the IBM MQ source connector configured against your queue managers.
- Exactly-once source settings, including exactly.once.source.support and tasks.max set to 1, are tested before any production traffic flows.
- Sink connectors and consumer applications are built and tested against topics that mirror the queues being offloaded.
How we hand over
- Deliverables include connector configurations, topic and ACL definitions, and the staged cutover runbook for each application.
- Rollback steps for every cutover stage are documented, with the checks that confirm message counts and ordering match.
- Your operators receive a walkthrough of connector status, offset monitoring and restart procedures on your own systems.
How we support
- Post-cutover support covers connector health, version upgrades and changes to queues or message formats.
- Ongoing support, upgrades and continuous testing are available as a retainer, sized to the written scope.
- Requests for changes to the migration scope are documented and agreed in writing before the work starts.
