Producing and consuming Kafka over HTTP

An HTTP proxy creates a consumer instance on the server side when a client registers, and that instance holds group membership, partition assignment and fetch position on the client's behalf. A native client keeps this state in its own process and handles rebalances itself. Over HTTP, your client holds only an instance name or URL and perhaps a token, so every instance is a resource the proxy must track until it is deleted or expires. The exact registration, subscription and fetch calls depend on the proxy. The Strimzi HTTP Bridge and Karapace expose different endpoints and lifecycle behavior, so confirm the sequence against the documentation for the version you deploy before you write client code.

Offsets move forward only when a commit succeeds, so the commit mode configured in the proxy and in your client decides whether progress advances without your code acknowledging the work. Check whether the proxy auto-commits, how often it does so, and whether a commit covers everything fetched or only what your client has named. If you commit manually, make the order explicit in your code: process the records, write the results to their destination, then commit. A commit issued before processing finishes can skip records on failure, while a commit issued after processing can replay records. Document the chosen order with the team that owns the consumer.

Both apply. Committing after processing gives at-least-once delivery: a crash between processing and committing replays records on restart. Committing before processing gives at-most-once delivery: a crash after the commit but before processing loses those records. HTTP makes the gap harder to reason about because fetch, processing and commit happen in separate requests. If a commit reaches the proxy but the response is lost to a network fault, your client cannot tell whether the offset moved. Design downstream writes to be idempotent, keyed on topic, partition and offset or on a business key, so that replays do not create duplicates in the systems you feed.

An instance that stays idle longer than the proxy's configured timeout is usually removed, and any fetched but uncommitted records are delivered again to the next instance that joins the group. Committed offsets stay in Kafka, so progress is tied to commits rather than to the instance itself. Your client must notice the expiry, typically from a not-found response on its next call, and create a new instance before it resumes. Confirm the idle timeout, the cleanup interval and the exact error returned on expiry in the proxy documentation. Those three settings determine how much work can repeat after a client pauses longer than expected.

Instance state held only in proxy memory is lost on restart, but committed offsets in Kafka survive, so consumers resume from their last commit. Clients that still hold old instance URLs will get errors and must re-create their instances. Whether instance metadata persists across restarts depends on the proxy and its version, so confirm it before you rely on it. If you run several proxy replicas behind a load balancer, requests for a given instance must reach the replica that holds it, which means session affinity or a shared metadata store. Test a restart during a live consume loop and measure how many records are replayed.

Yes. Delete the instance when the client shuts down cleanly, record the instance name so a restarted client can clean up its predecessor, and re-create the instance when a not-found response appears. Instances that are never deleted consume proxy resources and can keep partition assignments until they time out, which delays rebalances for the rest of the group. Confirm whether the proxy caps the number of instances per group or per user, because a client that creates an instance on every poll will reach that cap quickly. Track instance counts in your monitoring so leaks show up before they affect other teams.

Each poll and each commit is an HTTP round trip, so throughput depends on request latency, batch size and how many requests your client can keep in flight. Native clients fetch large batches over persistent connections and pipeline requests internally, so they usually sustain higher rates for the same hardware. The proxy adds its own serialization and connection handling on top. Plan for the gap by testing with your message sizes and your network path, not by reading generic figures. Set explicit timeouts and retry policies in your client, because a slow proxy response and a lost one look the same from the client side.

Confirm these items against the version you deploy:

• The idle timeout after which an instance is removed, and the cleanup interval
• The error returned for an expired or deleted instance
• The default auto-commit behavior and whether it can be turned off
• Whether instances and their offsets survive a proxy restart
• Any limit on instances per group or per user
• How authentication tokens expire and how clients refresh them

Record the answers in your runbook and test each one in a non-production environment. The Karapace repository is a useful starting point for its REST behavior, but the values that matter for your deployment are the ones your team measures.

Cost follows four drivers: the number of HTTP endpoints and topics you expose, the authentication model placed in front of the bridge, the number of concurrent consumers, and the compliance evidence you must produce. A small internal bridge with one shared credential costs far less to build and run than a multi-tenant gateway with per-client identity, rate limits and audit logging. Costs split into one-time work (design, build, testing, documentation) and recurring work (patching the bridge, rotating credentials, upgrading the proxy alongside Kafka, and monitoring). Ask any partner for a quote that separates these two and lists the endpoints, identity provider and consumer count it assumes.

Duration depends on scope rather than on Kafka itself. The main variables are how many topics and operations are exposed, whether identity integration with your existing provider is new work, whether consumer instance handling must survive restarts, and how much load testing your throughput targets require. Fixed-scope engagements agree timelines in writing before work starts, so the timeline should appear in the statement of work with its assumptions listed. Confirm with the bridge or proxy project which versions and features your target Kafka release supports, because version alignment can change the build plan.

Yes, if your team already operates Kafka and an HTTP gateway and has the security skills to review the authentication design. The work is mostly configuration and integration, but it becomes harder when you need multi-tenant isolation, per-client authorization mapped to Kafka ACLs, or evidence for an assessment. Kafka's authorization model stores ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so an HTTP layer that maps identities incorrectly can expose more than intended. Teams without these skills often bring in a partner for design review and testing, then run the result themselves.

The client team should provide a list of HTTP clients and their use cases, the topics and operations each client needs, the identity provider and token format in use, and the throughput and latency expectations for each client. They should also supply the Kafka cluster details: version, listener configuration, TLS arrangements and current ACL structure. Application owners must decide who approves new consumer groups and who handles client credentials. Without these inputs, the design cannot fix the endpoint count or the auth model, which are the largest cost drivers.

The platform team provides the Kafka cluster or access to it, a place to run the bridge or proxy (containers, virtual machines or Kubernetes), network paths between the HTTP tier and the brokers, certificates and secrets storage, and logging and monitoring pipelines. They also own the upgrade plan, because the HTTP component must stay compatible with the Kafka release in use. Kafka supports rolling upgrades from each release in the past year to the latest version, so the platform team should confirm the bridge's compatibility matrix before scheduling any broker upgrade.

Usually yes. HTTP clients add request and response overhead, serialization costs and connection handling on the bridge, and the bridge becomes a shared bottleneck that native clients bypass. Consumers over HTTP also need instance handling that native clients do not, such as creating, subscribing and deleting consumer instances. Treat HTTP access as suited to low-volume integration, browser or serverless callers and legacy systems, and benchmark your real message sizes and fan-out before committing. Ask the bridge or proxy project for its documented limits and test them in your environment.

Auditors will look for documented authentication and authorization, encryption in transit from client to bridge and from bridge to brokers, access logs tied to identities, change control for ACLs and credentials, and tested recovery. DeployOpen prepares systems, controls and evidence for compliance assessments, but independent auditors and certification bodies issue the reports and attestations. The evidence volume grows with the number of endpoints and tenants, so scope the controls before the build starts.

Choose based on your platform and whether you also need schema management. Karapace implements the Kafka REST API alongside its Schema Registry under Apache 2.0, so it fits teams that want both in one component. The Strimzi HTTP Bridge fits teams already running Kafka on Kubernetes with Strimzi. Confirm the supported Kafka versions, authentication options and license terms for each option against its repository before deciding, as these change between releases.