What moving from Pub/Sub to Kafka involves and how much effort it takes
If your team runs Pub/Sub topics and subscriptions today and wants them on Kafka in infrastructure you control, the move is a redesign of how messages are keyed, partitioned, ordered and consumed, followed by a staged cutover. It is not a bulk copy of a backlog. The effort sits mostly in design work: deciding which Pub/Sub ordering keys become Kafka message keys, how partitions are counted and assigned, which consumers move first, and how long both systems run side by side. Kafka itself is licensed under Apache 2.0 and the Apache Software Foundation offers no paid edition, so the software cost is zero. What you pay for is infrastructure, operations and the people or partner who design and test the change.
Start by inventorying every topic and subscription, then record for each one its ordering requirements, expected message rate, retention needs, and the consumers that depend on it. Google's Pub/Sub Group Kafka Connector, licensed under Apache 2.0, includes a source connector that reads from a Pub/Sub subscription and writes to a Kafka topic, so draining existing subscriptions into Kafka during cutover is part of the plan. Two controls in that connector shape the design: kafka.key.attribute sets the Kafka record key, and kafka.partition.scheme controls partition assignment. Confirm the connector version and its supported options against the connector repository before you write the design down.
Plan for the Pub/Sub message size limit of 10 MB, which the connector documentation notes. Any producer that already stays under that limit can keep its payloads unchanged, but payloads near the limit need a review of broker and client settings on the Kafka side. Ask your partner to quote the work as a written scope covering topic and partition design, connector deployment, consumer rewrites, a dual-running period and the verification plan. Effort rises with the number of subscriptions, the number of consumers that need ordering guarantees, and the number of teams that must change their code. Confirm the exact effort with your own environment and inventory rather than relying on a general estimate.
Which Pub/Sub capabilities map to Kafka and which need redesign
This table compares Pub/Sub concepts with their Kafka equivalents and marks where the connector maps data directly and where consumer design changes. Read each row as a question to answer for your own topics before cutover. The main takeaway is that subscription behavior and message keys need explicit decisions, not defaults.
| Pub/Sub concept | Kafka equivalent | What to confirm or redesign |
|---|---|---|
| Topic | Topic | Map one Pub/Sub topic to one Kafka topic, then confirm partition count and replication before the connector starts. |
| Subscription | Consumer group | Subscriptions acknowledge messages individually, while consumer groups track offsets per partition, so each consumer must be redesigned around offset commits. |
| Ack deadline and redelivery | Offset commit and reprocessing | Kafka does not redeliver on a missed ack deadline; consumers must handle retries and replays themselves. |
| Ordering key | Record key with partitioning | Use kafka.partition.scheme and kafka.key.attribute to keep related messages on one partition. Confirm this is what ordering needs. |
| Message attributes | Record headers or key | Attributes are not automatically carried as keys; set kafka.key.attribute to the attribute that defines ordering or partitioning. |
| Message size limit | Broker and producer size settings | Pub/Sub messages cannot exceed 10 MB. Confirm your Kafka size limits accept the same payloads before migrating producers. |
| Retention | Topic retention settings | Pub/Sub retention is per subscription, while Kafka retention is per topic. Set retention to cover the replay window you need during cutover. |
Documented tools for draining Pub/Sub into Kafka
The migration path rests on one documented tool, Google's Pub/Sub Group Kafka Connector, which runs inside Kafka Connect. The cards below cover what the connector needs, where it runs, and which versions and settings you must confirm before the cutover plan is written.
Pub/Sub Group Kafka Connector source
The source connector reads from Pub/Sub and writes to a Kafka topic. Each connector instance needs a Pub/Sub subscription, the Google Cloud project that owns it, and the target Kafka topic. Draining starts with the subscription you already use, so confirm which subscriptions feed production consumers and which are idle before you point a connector at them. Messages are consumed from the subscription, so a live consumer and the connector compete for the same messages.
Pub/Sub Group Kafka Connector sink
The connector also includes a sink that writes Kafka records back to Pub/Sub. Use it only when a consumer still on Pub/Sub must receive traffic produced to Kafka during a staged cutover. It is not required for a one-way drain. If you plan a sink, record which topics it writes to, how it maps keys, and which consumers will see duplicates during the overlap window.
Kafka Connect as the runtime
The connector does not run on its own. It is deployed as a plugin in Kafka Connect, either in distributed mode on your own servers or on Kubernetes. Plan the Connect cluster as a separate workload with its own sizing, configuration topics and failure handling. Decide who owns the Connect cluster after cutover, because the connector outlives the migration project.
Java 17 runtime check
Since Kafka 4.0, brokers, Kafka Connect and tools require Java 17. Confirm that the Connect workers you build run on Java 17 or later, and that the connector version you select supports the same runtime. Do not assume compatibility from the connector name or from an older Kafka 3.x deployment. Ask your platform team to record the JVM version in the Connect image so the check is repeatable.
Connector version compatibility
Confirm the connector release against the Kafka Connect version in your environment, using the connector's own repository and release notes. Do not rely on a compatibility matrix from a blog post or an older project page. Record the connector version, Kafka version and JVM version together in your change plan, and retest after any upgrade to either side.
How to sequence inventory, bridging, parallel run and cutover
This sequence moves a Pub/Sub workload to self-hosted Kafka in stages, so each subscription, producer and consumer group is proven on Kafka before its traffic is switched and Pub/Sub is retired.
Inventory topics, subscriptions and publishers
Your team lists every Pub/Sub topic and subscription, the publishers and subscribers attached to each, the message sizes in use, and whether producers set ordering keys or attributes that drive partitioning. Map each subscription to the consumer group that reads it and to the downstream systems it feeds. Confirm which messages approach the 10 MB Pub/Sub limit, since they affect Kafka broker and client size settings. The step is done when every subscription has a named owner, a target Kafka topic and a written note on ordering and duplicate tolerance.
Stand up a KRaft Kafka cluster with security planned
Deploy a Kafka cluster in KRaft mode in your own infrastructure, using a current 4.x release from the Kafka downloads page. Plan TLS, SASL mechanism selection and StandardAuthorizer ACLs before any topic is created, because resources without ACLs are restricted to super users unless you change that default. Decide now how topics will be named, partitioned and retained so the Connect configuration does not need rework later. The step is done when a test producer and consumer can authenticate, authorized access is confirmed for a dedicated service principal, and topic settings match the inventory.
Configure the source connector per subscription
Deploy Google's Pub/Sub Group Kafka Connector source connector on Kafka Connect, one connector configuration per subscription that will be drained. Each configuration needs the subscription, project and target Kafka topic. Set kafka.key.attribute and kafka.partition.scheme deliberately, because they control Kafka keys and partition assignment and therefore per-key ordering on the Kafka side. Confirm the connector license terms and the Connect runtime requirements in the connector repository. The step is done when each connector reports running, and a test message published to Pub/Sub appears on the target topic with the expected key and partition.
Validate ordering and duplicate handling
Before any consumer moves, test how messages arrive on Kafka compared with Pub/Sub. Publish controlled sequences with ordering keys and confirm that per-key order holds on the target partitions. Force connector restarts and redelivery, then count duplicates and confirm that every consumer can tolerate them or that idempotent handling is in place. Record the results per subscription. The step is done when each subscription has a documented pass or a documented gap with an owner, and no switch is scheduled for a subscription with an open gap.
Run both systems with dual consumption and reconciliation
Keep existing Pub/Sub consumers running while the source connector feeds Kafka, and bring up the new Kafka consumers in shadow mode that write to a separate sink or a non-production store. Reconcile message counts, keys and sampled payloads between the two paths over an agreed observation window set by your team. Track consumer lag, connector errors and dead-letter volumes on both sides. The step is done when reconciliation shows no unexplained differences and the operations team has signed off the dashboards and alerts.
Move producers and consumers group by group
Switch one consumer group at a time to read from Kafka, starting with the lowest-risk subscription, and publish from the new producers to Kafka topics only when their consumers are ready. Keep a rollback path that returns a group to its Pub/Sub subscription until the cutover is accepted. Coordinate publisher changes with the connector so no message is produced to both paths in a way that a consumer would process twice. The step is done when each group has run on Kafka through its agreed observation window with the reconciliation checks still passing.
Decommission Pub/Sub after a verified window
Once all groups have run on Kafka for a verified window, stop the source connectors, remove remaining Pub/Sub subscriptions and retire the publisher credentials that no longer have a use. Archive the inventory, ACL records, connector configurations and reconciliation evidence so the decision can be reviewed later. The step is done when no Pub/Sub subscription is active for the migrated workload, Kafka is the only path in production, and the archived evidence has been accepted by the service owner.
How keys, partition schemes and ordering keys carry over
The Pub/Sub connector decides per-entity ordering on the Kafka side through two settings. The kafka.key.attribute setting names the message attribute whose value becomes the Kafka record key, and kafka.partition.scheme controls how the connector assigns that key to a partition. Kafka only guarantees order within a single partition, so the record key and the partition scheme together determine whether messages for one customer, order or device stay in sequence after the move. Before cutover, confirm which attribute your publishers already set, whether every message carries it, and which partition scheme the connector release you deploy supports. The connector documentation in the Pub/Sub Group Kafka Connector repository is the place to check the exact option values.
Pub/Sub ordering keys do not map automatically to Kafka. An ordering key in Pub/Sub only orders delivery to a subscriber that is using ordering, so the Kafka equivalent is a record key that is stable for each entity. Plan the mapping so that one entity always produces the same key and therefore lands on the same partition. Then test it: publish several messages for one entity, drain them through the connector, and confirm they arrive in the expected order on one partition. Messages without an ordering key need a deliberate decision, either a null or random key with no ordering promise, or a fallback attribute that your consumers can rely on. Consumers that depended on Pub/Sub ordering should be reviewed as part of the same plan.
Partition count is the main risk to this mapping. Kafka assigns a key to a partition by hashing it against the current partition count, so adding partitions later sends the same key to a different partition. Messages for one entity can then be split across old and new partitions during a transition, and ordering is lost for that window. Choose the partition count before the connector starts writing, size it for expected growth, and treat any later increase as a planned event with consumer draining rather than a routine change. Ask your implementation partner to document the chosen count, the key format and the test results so that the decision is auditable by your own team.
What drives the cost and duration of the migration
This table compares the factors that raise or lower migration effort, row by row, for a Pub/Sub to Kafka move. Read each row against your own inventory of subscriptions, publishers and subscribers to see which factors dominate your estate. The main takeaway is that ordering needs, key design and the length of the parallel run usually shape effort more than topic count alone.
| Driver | What raises effort | What lowers effort |
|---|---|---|
| Topics and subscriptions | Many subscriptions with separate consumers, dead-letter settings and filters that each need a mapped consumer group and a drain plan | A small set of subscriptions consolidated per domain, with an inventory that lists owner, consumer and backlog for each |
| Ordering requirements | Ordering keys that must hold across the cutover, consumers that rely on per-key sequence, or topics where replay order matters | Ordering needs confirmed per subscription before design, with keys mapped to Kafka keys and partition schemes tested on sample traffic |
| Publishers and subscribers that must change | Services that publish to Pub/Sub through many client libraries or that consume through pull and streaming pull code spread across teams | Publishers routed through a shared producer library, and consumers that can switch with a configuration change behind a feature flag |
| Schema and attribute complexity | Messages where business meaning sits in attributes, mixed payload formats, or fields that must map into Kafka keys and headers | Documented message schemas, a single attribute convention, and a short list of attributes that drive kafka.key.attribute |
| Message size limits | Payloads that approach the 10 MB Pub/Sub limit, which may need a redesign such as storing large content externally before crossing into Kafka | Payloads already well under the limit, with a check of the largest messages in the last retention window |
| Security model | Kafka SASL mechanism choice, ACLs stored in the metadata log that must be written for every new principal, and TLS certificate issuance across sites | Reuse of the identity provider and an agreed mechanism per client type, with ACLs generated from one reviewed source |
| Retention and replay needs | Consumers that need history from before cutover, or replay windows that require the new topics to hold data for longer than the source did | Clear agreement on how much history must move, with replay served from the Kafka retention period you configure |
| Cross-site replication | Replication across regions or data centers that must keep offsets and ACLs aligned, requiring MirrorMaker 2 design and failover testing | A single site for the first cutover, with replication added once the consumer model is stable |
| Length of the parallel run | Dual publishing or bridged consumption kept for many weeks, with reconciliation of duplicates and gaps at each consumer | Defined exit criteria for each subscription, and a retirement date agreed with each consumer owner before the parallel run starts |
What usually goes wrong during a Pub/Sub to Kafka cutover
Most failed Pub/Sub to Kafka cutovers trace back to a few decisions made early: how keys and partitions are chosen, who can read which topics, and how long both systems run side by side. The cards below cover each risk, what causes it, and the check that prevents it before traffic moves.
Duplicate processing during dual consumption
While the Pub/Sub subscription and the Kafka topic are both live, a consumer may process the same business event twice, once from each path. This is most likely when a service is moved before its downstream writes are idempotent. Before cutover, confirm each consumer deduplicates on a stable business identifier, and decide which system is the single writer for each event type during the overlap window. Name one writer per event type before any consumer is pointed at both paths.
Ordering breaks from key choice
Kafka only orders records within a partition, so the key you map to must carry the ordering guarantee your consumers depend on. The connector's kafka.key.attribute and kafka.partition.scheme settings decide the key and partition assignment, and a key drawn from a field that varies per message can split related events across partitions. Test with a sample of real ordered sequences and check that each sequence lands on one partition before enabling the connector in production.
Consumers without ACLs
In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. A new consumer group or topic that lacks explicit ACLs will fail on first connect, which can look like a connector or network fault. Create topic, group and connector principal ACLs as part of the build, and test each consumer identity against the cluster before cutover.
Unplanned partition count changes
Partition count sets how many consumers can read in parallel and how keys map to partitions. Increasing partitions later changes the key-to-partition mapping for new records, which can break per-key ordering across the change point. Decide the partition count from expected throughput and consumer parallelism before the topic is created, and treat any later increase as a planned change with a consumer drain and an ordering review.
Connector failures that stall draining
A source connector that stops on a bad record, a rejected credential or a network path issue leaves messages sitting in the Pub/Sub subscription while consumers wait. Because Pub/Sub messages cannot exceed 10 MB, oversized payloads are a predictable failure source. Monitor connector task state and the subscription backlog together, set alerts on both, and define who acts on a stall and how far the cutover window can slip before rollback is considered.
Ack-deadline assumptions in consumer code
Pub/Sub consumers acknowledge messages within an ack deadline, and code written around redelivery timing may assume that behavior holds. Kafka consumers instead track offsets per consumer group and commit them, so a consumer that is slow or that commits before finishing work can skip or repeat records differently. Review each consumer for assumptions about redelivery and dead-letter behavior, and set offset commit timing explicitly during the rewrite.
How to run Pub/Sub and Kafka side by side without double processing
The safe pattern is to keep every existing Pub/Sub consumer running while the Pub/Sub Group Kafka Connector feeds the same messages into Kafka. Nothing downstream changes on day one. You then move consumers one group at a time: stop a Pub/Sub consumer group, confirm the matching Kafka consumer group has caught up, start the Kafka consumer, and watch its lag and output. The connector's source side reads from a Pub/Sub subscription, so each subscription you keep draining is an active consumer of that data. Plan which subscriptions feed which Kafka topics before you start, because two consumers that both act on the same message are the cause of double processing.
Double processing is not prevented by the connector or by Kafka. Kafka consumer groups track offsets per group, so two groups reading one topic will each process every message, which is correct for fan-out but wrong if both groups perform the same side effect. Pub/Sub delivery is at least once, and the connector does not promise that a message is written to Kafka exactly once, so a retry can produce a duplicate record. Your consumers therefore need an agreed idempotency rule: a stable message identifier, usually taken from the Pub/Sub message ID or an attribute your publishers set, stored with each side effect so a repeated message is recognized and skipped. Confirm with your application owners which key they will use before the first consumer moves.
Reconciliation closes the gap that idempotency alone cannot see. During each overlap window, compare record counts and message identifiers between the Pub/Sub subscription and the Kafka topic, and check the connector's throughput and consumer lag on both sides. Investigate any identifier that appears in one path and not the other, and any consumer group whose lag keeps growing after it starts. Keep the Pub/Sub consumer until the reconciliation for that group is clean and its side effects have been checked against the business records. Only then remove the old consumer. The exact checks depend on your message volume and retention settings, which you should confirm with your own team and the connector documentation.
Questions teams ask before moving off Pub/Sub
How much effort does a Pub/Sub to Kafka migration take, and what drives it?
Effort depends on how many topics, subscriptions and consumers you have, and on how much of your code assumes Pub/Sub behavior. The connector moves messages, but each subscription still needs a decision about its Kafka topic, key and partition scheme, and each consumer needs to be rewritten or repointed. Teams with few subscriptions and simple consumers usually have a small job. Teams with many services, custom acknowledgment logic or cross-project subscriptions have a large one. Ask any partner for a fixed-scope estimate built from a subscription inventory, and ask what is one-time (design, connector setup, cutover) and what recurs (running the Kafka cluster and connectors after go-live).
How long does the parallel run need to be?
The parallel run lasts until every consumer has been verified against Kafka and no traffic still depends on Pub/Sub. What sets its length is the slowest consumer to validate, the retention you need for replay, and how long your teams need to compare outputs. Pub/Sub retention and Kafka retention are configured separately, so set Kafka topic retention to cover the rollback window you agree on. The connector can drain Pub/Sub subscriptions into Kafka while producers still publish to Pub/Sub, which makes a staged cutover practical. Confirm the exact window in the written scope rather than relying on a general figure.
What does the migration need from our team?
You need a named owner for each topic and subscription, people who can confirm what each consumer does with its messages, and access to the publishers and subscribers you are moving. You also need access to the cloud project that holds the Pub/Sub resources, a service account that can read the subscriptions, and a Kafka cluster in infrastructure you control, with network paths between them. Expect your platform team to own cluster sizing, TLS and authentication, and monitoring. Ask a partner to list these needs in the scope so ownership gaps show up before work starts.
Can our in-house team run this migration alone?
Often yes, if the team already operates Kafka or has someone who has run Kafka clusters in production, and if the subscription count is modest. The risks come from less visible work: partition planning, authentication and ACL design, consumer offset handling, and testing that consumers behave the same under Kafka's delivery model. If none of these skills exist in-house, the usual approach is a fixed-scope engagement for design and cutover, followed by internal ownership. DeployOpen implements, hardens, documents and supports self-hosted open-source platforms in your infrastructure, so it can carry the build alongside your team.
Is the Pub/Sub Group Kafka Connector free, and under what license?
The Pub/Sub Group Kafka Connector is licensed under the Apache License 2.0, and its repository includes source and sink connectors for Cloud Pub/Sub and Pub/Sub Lite. The license covers the connector code, not your cloud usage. Pub/Sub itself is billed by Google, so you will pay for Pub/Sub traffic during the parallel run, and you will pay for the infrastructure that hosts Kafka and the connector. Check the license terms of any other connector or tool in the pipeline, since their licenses can differ. Confirm the license of the version you deploy against the repository before you build.
Does Kafka ordering match Pub/Sub ordering keys?
Kafka does not guarantee order across a topic, only within a partition. The connector's kafka.key.attribute and kafka.partition.scheme settings control which Kafka key is written and how partitions are assigned, so messages that share a Pub/Sub ordering key can be sent to the same partition if the key is carried over correctly. Ordering then holds for each key within a partition, provided producers keep writing to that partition and the partition count does not change. Pub/Sub's ordering behavior, redelivery and acknowledgment differ from Kafka's offset model, so test each consumer with replayed traffic. Also note that the connector documentation states Pub/Sub messages cannot exceed 10 MB, which affects how you size Kafka messages.
How DeployOpen scopes and delivers a Pub/Sub to Kafka migration
DeployOpen plans, implements, tests, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For this migration we start from a written scope that names the subscriptions, topics, ordering requirements and cutover window, then build and verify the connector path in your environment.
How we scope
- We inventory every Pub/Sub topic and subscription that the Kafka cluster will replace or mirror.
- We record which subscriptions need draining, and which consumers depend on ordering keys.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before any work begins.
How we build
- We deploy the Pub/Sub Group Kafka Connector source path and set key and partition scheme settings per topic.
- We test payloads near the 10 MB Pub/Sub message limit and confirm how each is handled.
- We run old and new paths side by side during cutover and verify consumers read the same messages.
How we hand over
- We document connector configuration, key mapping, partition decisions and the cutover runbook.
- We deliver a rollback procedure for each consumer group, with the checks to run before switching.
- We train your platform and application teams on operating the connector and the new Kafka topics.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- A retainer can cover connector version updates and Kafka upgrades on the release line you choose.
- Ask for a quote built from the written scope, which sets the support boundary for your environment.
