Indexing Kafka topics into OpenSearch

Indexing Kafka topics into OpenSearch for search and log analytics

A team that already publishes events to Kafka and needs them searchable in OpenSearch can run this integration through Kafka Connect, using an Apache 2.0 OpenSearch sink connector. The work sits in four places: index naming, document IDs, schema handling, and TLS to OpenSearch, with mapping errors handled as a fifth concern that touches all of them. Before you commit to a connector, confirm its configuration keys against that connector's own documentation and against the Kafka and OpenSearch versions you run, because no connector-specific documentation was supplied for this page. Both Apache Kafka and OpenSearch are released under the Apache License 2.0 and have no paid edition from their projects, so the whole pipeline can run in clusters and networks you control.

Index naming is the first decision to settle, because it determines how retention, access and searches work later. A common pattern is one index per topic or per time window, for example one index per day for log-style data, but the right choice depends on query patterns and how old data is deleted. Document IDs matter just as much: if the sink derives an ID from the record key or from topic, partition and offset, reprocessing after a failure can either overwrite the same document or create duplicates. Test both paths with replayed records in a non-production cluster before you go live. Ask the implementation partner, and your own team, to write down the chosen ID strategy and the expected behavior on a retry.

Schema handling and mapping errors are where most production surprises appear. OpenSearch infers field types from the first document it sees in an index unless you define a mapping ahead of time, so a field that arrives as a number in one record and a string in another can reject documents or distort searches. Define index templates and explicit mappings before the first write, decide how unknown fields are treated, and confirm how the connector routes documents that OpenSearch rejects. For TLS, confirm the connector's options for trusting the cluster certificate and for client authentication against the OpenSearch Security plugin, and review the OpenSearch security documentation for the roles the connector user needs.

What the Kafka and OpenSearch sides need before the sink runs

The sink sits between two systems that each have their own version, identity and security model. Most failed deployments trace back to a gap on one side that was not checked before the connector was configured. The cards below cover what to confirm on the Kafka side, the OpenSearch side and the path between them.

Kafka and Connect versions

Kafka 4.x requires Java 17 for brokers, Kafka Connect and tools, while clients and Kafka Streams applications need at least Java 11. Confirm the Connect workers run on the same supported Java level as the rest of the platform, and pick a Kafka release from the supported list rather than an archived one. Check the connector plugin's own compatibility statement against your Connect version before you install it on the workers.

Running Connect cluster with plugin

The sink runs as a connector on a Kafka Connect cluster, so you need distributed workers that are already healthy and reachable from the brokers. Install the OpenSearch sink plugin through your plugin path, not by copying jars into a running worker by hand, and confirm each worker lists the plugin after restart. Record the plugin version so that upgrades of Connect and the plugin are planned as one change.

OpenSearch version and deployment

Choose the OpenSearch line you will run and the install method your team can operate, such as Helm, the OpenSearch Kubernetes Operator, packages or containers. OpenSearch 3.9.0 and 3.8.0 are current releases, and each version stays in maintenance for a defined period before it ends support. Plan the upgrade cadence before go-live so that the sink does not fall behind an OpenSearch version that has left maintenance.

Network paths and certificates

The Connect workers must reach every OpenSearch node or the load balancer in front of them on the HTTP port, and the connection must use TLS. Confirm which certificate authority signs the OpenSearch node and client certificates, and that the workers trust that authority before you set truststore paths. Ask your network team to confirm firewall rules in both directions, including DNS names used in the connector configuration.

Dedicated Kafka service identity

Create a dedicated principal for the connector, separate from applications and human users, and store its credentials in your secrets system. In KRaft clusters, the StandardAuthorizer restricts resources without ACLs to super users unless allow.everyone.if.no.acl.found is set to true. The connector principal therefore needs explicit read ACLs on the source topics and consumer group access, and you should not rely on that setting to make the sink work.

Dedicated OpenSearch service identity

Create a separate OpenSearch user for the sink with a role that allows writes only to the index patterns it owns, and avoid reusing an administrator account. Confirm which authentication backend your cluster uses, such as the Security plugin's internal users or an external directory, and test the role with a write to a non-production index before connecting real topics. Document who owns the role mapping so that access reviews cover this identity.

Configuring the OpenSearch sink in the order that avoids rework

This sequence sets up access, naming and trust before any data moves, so the first documents that reach OpenSearch are indexed where you expect, with the IDs you planned, and can be traced back to a known source topic.

Create the OpenSearch role and service user
An OpenSearch administrator creates a dedicated role that grants write and index-creation rights only on the target index pattern, then maps a service user to that role using the Security plugin. Avoid granting cluster-wide or admin roles to the connector account. Confirm the exact role and mapping syntax in the OpenSearch security documentation for your version. Done means the service user can index a test document into the target pattern and receives a permission error when it writes to any other index.
Grant Kafka ACLs for source and Connect topics
A Kafka administrator creates a principal for the connector and grants READ on the source topics, plus the access Kafka Connect needs for its internal configuration, offset and status topics and for its consumer group. In KRaft clusters, ACLs live in the metadata log, and resources without ACLs are restricted to super users unless the allow.everyone.if.no.acl.found setting is enabled, so check that before deployment. Confirm the exact operations in the Kafka authorization documentation. Done means a test consumer under the connector principal can read the source topic and a principal without the ACL is denied.
Install the connector plugin on Connect workers
A platform engineer places the OpenSearch sink connector plugin, from a source you have reviewed and approved, into the plugin path on every Connect worker and restarts the workers in a controlled order. Pin the plugin version and record its license, since connector licensing differs across vendors. Done means the connector class appears in the worker's list of installed connector plugins on every node, and the same version is reported by each worker.
Set index naming and document ID strategy
Before any records flow, the data owner and the platform team agree on the index naming pattern, any routing or date suffix, and the document ID source, such as a key field, a composite of fields, or the Kafka topic, partition and offset. Record the choice in the connector configuration plan and confirm the exact property names in the connector documentation. Changing the ID strategy later usually means reindexing, so this decision is final for the test. Done means a written naming and ID specification has been approved by the owners of the data and the search cluster.
Configure TLS trust for OpenSearch and Connect
The platform team imports the certificate authority that signed the OpenSearch HTTP certificates into the Connect worker truststore, then sets the connector to use HTTPS with certificate validation enabled rather than disabled. Confirm the truststore and TLS property names in the connector and OpenSearch security documentation. Done means a worker can open an HTTPS connection to the cluster with a valid chain, and a deliberate test with an untrusted certificate fails before any connector is started.
Start the connector with a small test topic
An engineer creates a low-volume test topic with a few representative records, including one with an unexpected field type, and starts the connector against it using the agreed index pattern and ID strategy. Check the connector and task status through the Connect REST interface and watch the worker logs for errors. Done means the test documents appear in the expected index and the _id values match the specification exactly, including on a repeated run of the same records.
Verify document IDs and mapping before scaling
Compare a sample of indexed documents with their source records, checking the _id values, field mappings and the handling of the unexpected field type from the test. If the connector routes failures to a dead letter topic or error log, confirm that the bad record was captured with context. Only then move the same configuration to the production topics. Done means the sample matches the specification, the repeated run produced no duplicates, and the team has signed off the mapping in writing.

How Kafka record fields map to OpenSearch index names, IDs and fields

This table compares each Kafka element with its OpenSearch counterpart, the decision your team has to make for it, and the risk if the connector defaults are kept. Read it before the first connector runs, because index names, document IDs and mappings are hard to change once data is indexed. The main takeaway is that a stable document ID, which depends on how producers set record keys, is what makes redelivered records overwrite instead of duplicating.

Kafka element	Decision for your team	Risk if left to defaults
Topic name to index name	Choose a naming pattern, lowercase names, and whether to add a date suffix for time-based retention	Topic names with characters OpenSearch rejects or mixed case cause failed writes or unpredictable index names
Date suffix on index names	Decide between one index per day or per month, or a single index with lifecycle rules for rollover and deletion	Daily indices with many small shards inflate cluster state and slow searches over long ranges
Record key to document ID	Confirm producers set a key that is stable per business entity, such as an order or device identifier	Null or random keys give each redelivery a new ID, so retries and replays create duplicate documents
Record value fields to document fields	Choose which fields to index, which to store only, and whether nested payloads are flattened	Unneeded fields bloat the index, and sensitive fields become searchable by users with broad read access
Schema-based types to index mappings	Define explicit mappings for timestamps, keywords, numbers and IP addresses before the first write	Dynamic mapping may index a numeric string as text or a timestamp as a plain string, which breaks range queries and aggregations
Partition and offset to traceability fields	Decide whether to keep partition and offset as fields for tracing a document back to its source record	Without them, investigating a bad document means searching the topic by hand, and the source position is lost
Kafka timestamp to event time field	Choose which timestamp drives time-based indices and searches, the producer event time or the broker append time	Using the broker time can place late events in the wrong index and skew dashboards

Securing the connection between Kafka, Connect and OpenSearch

A Kafka to OpenSearch pipeline crosses three trust boundaries: the Kafka cluster, the Connect worker and the OpenSearch cluster. Each boundary needs its own encryption, identity and permission model, and the connector's service account must be scoped to the indices it writes.

TLS between Kafka clients and brokers

Kafka supports SSL/TLS for traffic between clients, brokers and tools, and security is optional, so a cluster can run with unsecured listeners next to secured ones. For this pipeline, confirm that the Connect worker's consumer listener uses TLS and that no plaintext listener is reachable from the worker network. Check the certificate chain and hostname verification in the worker's client configuration, and record which listener the connector is pinned to so a later listener change does not silently downgrade traffic.

TLS from Connect to OpenSearch

Encrypt both node-to-node and client traffic on the OpenSearch side, then configure the sink's connection so the worker validates the cluster certificate against a trusted CA. Verify that the connector does not disable certificate checks to get past a self-signed certificate during testing, because that setting often survives into production. Confirm with the connector documentation which TLS options it exposes, such as truststore format and client certificates, before you choose the trust model.

Least-privilege OpenSearch role for target indices

Create a dedicated OpenSearch role that grants write and index-creation rights only on the index pattern the connector produces, such as a prefix per topic group. Avoid cluster-wide write permissions and admin roles. If the connector creates indices and mappings, decide whether that permission is granted or whether indices and templates are pre-created by the platform team, since the answer changes the privileges the role needs.

Kafka ACLs for the connector principal

Give the Connect worker's Kafka principal read access to the source topics and its consumer group, plus the topics and groups it needs for offsets, configuration and status. In KRaft clusters, ACLs live in the cluster metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is enabled, so a missing rule usually shows as an authorization failure rather than an open topic. Keep the principal separate from producers and other consumers.

Credential storage outside connector config

Keep the OpenSearch password or certificate key, and the Kafka client secret, out of connector JSON and worker properties files that teams commit or copy around. Use a secrets provider or mounted files with restricted permissions, and reference them from configuration. Confirm that the connector version supports externalized configuration values, and plan rotation so that rotating a credential does not require editing every connector definition by hand.

Audit logs and restricted fields in OpenSearch

Enable audit logging in the OpenSearch Security plugin so writes by the connector's account are attributable and reviewable. If indexed events contain personal or regulated fields, consider document-level and field-level security or field masking so that analysts' roles see only what they need. Human users can authenticate through SAML, OpenID Connect, Active Directory and LDAP, Kerberos or JSON Web Token backends, but the connector's service account must use a method the connector actually supports, so confirm that before selecting a backend.

Running the sink on VMs, Kubernetes or a managed Connect service

Connect workers on VMs that your team manages give you direct control over plugin directories, JVM settings and the network path to OpenSearch. The sink is installed as a set of JARs in a directory listed under plugin.path, so every worker in the group must hold the same plugin version before a rollout starts. Your team also owns the operating system patching and the restart sequence.

Plugin installation: copy the sink plugin into the same directory on every worker, then restart workers one at a time so the group keeps running.
Secret handling: keep OpenSearch credentials out of connector JSON stored in source control. Use a config provider or a file readable only by the Connect service account, and confirm which config providers your Connect distribution ships.
Network policy: open the path from each worker to the OpenSearch HTTPS port only, and recheck host firewall rules after every host change.
Upgrade order: upgrade Kafka brokers first, then Connect workers, following the Kafka upgrade notes. Check that the sink plugin version matches the Connect runtime and that Java 17 is installed on each worker.

Confirm how your Connect distribution loads plugins, including whether it scans plugin.path only at startup, before you schedule the change window.

Common OpenSearch sink failures, their causes and fixes

This table lists the failures teams most often hit when running an OpenSearch sink connector, with the usual cause and the fix for each. Read each row as a checklist item to verify in your own environment before the first production load. The main takeaway is that most failures trace to mapping, access rights, trust configuration, topic ACLs, document identity or naming, and each can be checked before data flows.

Failure	Cause	Fix
Mapping conflict after a field changes type	Dynamic mapping fixed a field's type from the first document, so a later document with a different type is rejected	Define explicit index templates for known fields, and route rejected records to a dead letter topic for review before changing the mapping
Documents rejected for lack of write rights	The OpenSearch service user has no index write or create privilege on the target index pattern	Grant the sink role write and create rights scoped to the index pattern, then confirm with a test document before starting the connector
TLS handshake failure to OpenSearch	The OpenSearch certificate authority is missing from the Connect worker truststore, or the hostname does not match the certificate	Import the CA into the worker truststore, confirm the certificate subject matches the host name, and test the handshake from the worker host
Connector stalls with no records consumed	The Connect principal has no read ACL on the source topic or no read access to its consumer group	Add READ on the source topic and the consumer group for the Connect principal, then check the worker log for authorization errors
Duplicate documents after restarts	Document IDs are generated per delivery or are missing, so retried writes create new documents instead of replacing existing ones	Derive document IDs from a stable key such as topic, partition and offset, or from a business key, so retries overwrite the same document
Index names rejected	Topic names contain characters or uppercase letters that OpenSearch index names do not allow, or the name starts with a disallowed character	Use a name transform or topic-to-index mapping that lowercases the name and replaces unsupported characters, and test the mapping with sample topics

Testing the pipeline before search users depend on it

The test plan should prove that the sink writes what you expect before any search user, dashboard or alert depends on the index. Start by replaying a sample of real topic data into a non-production index, using the same connector version, converters and transforms you plan to run in production. Then compare document counts against the source topic for the replayed offsets, and compare a set of sample documents field by field, including timestamps, nested objects and any values that pass through a transform. A count match with wrong field types is still a failed test, so check the mapping that OpenSearch creates against the mapping you designed, not just whether documents arrived.

Next, test the failure paths on purpose. Send one record that conflicts with the mapping, for example a string where the index expects a number, and confirm that the behavior matches your error-handling configuration. Depending on how the connector and the team's errors.tolerance and dead letter queue settings are set, the record should either fail the task, be logged with context, or be routed aside. Restart Connect workers in the middle of a sustained load, then compare document counts and IDs to confirm that deterministic document IDs prevent duplicates after the restart. Finally, rehearse any mapping change on a new index and verify the reindex path before touching production. Record the connector version and error settings with the results, because a different connector version or error configuration can change these outcomes, so repeat the tests whenever either changes.

Questions teams ask before building a Kafka to OpenSearch pipeline

What drives the cost and effort of indexing Kafka topics into OpenSearch?

The main drivers are the number of topics, how varied their schemas are, how complex the index mappings must be, the security requirements, and how many environments the pipeline must run in. Each topic usually needs its own index naming rule, document ID strategy and error handling, so a dozen topics with stable schemas is far less work than a hundred topics with records that change shape over time. Mapping design takes the most judgment, because field types chosen early are hard to change later without reindexing. TLS, authentication and role design add work in proportion to the number of clients and roles. Each additional environment, such as test, staging and production, repeats the configuration and verification. Ask any partner to price the work against a written list of topics, schema samples, target OpenSearch clusters and security requirements.

Can our team build a Kafka to OpenSearch pipeline without outside help?

Yes, if your team already runs Kafka Connect and administers OpenSearch security, and if it can check the sink's configuration against the connector's own documentation before going live. The hard parts are not the initial connector setup. They are the decisions that are costly to reverse: index naming, document IDs, mapping strategy, and how bad records are routed. Teams without someone who owns OpenSearch roles, TLS and index lifecycle often find that gaps show up in production rather than in testing. A useful self-check is whether you can write down, for each topic, the index name, the document ID source, the expected schema, and what happens to a record that OpenSearch rejects. If you cannot, that is the work to plan first, with or without a partner.

What do we need to provide before an implementation starts?

The work needs named owners on your side for Kafka, OpenSearch and data schemas, because most decisions depend on knowing what a topic means and who consumes it. Plan for access to test environments that mirror production topics, OpenSearch clusters and Kafka Connect workers, with credentials that let the team create indices and roles there. Provide the certificate authority details that will sign the TLS certificates for brokers, Connect workers and OpenSearch nodes, including the chain, the expiry policy and who can issue new certificates. Sample messages from each topic, including records that are known to be malformed, make mapping and error handling far more accurate. Agree in writing who approves mapping changes before they reach production.

How long does a Kafka to OpenSearch integration take?

Duration depends on the same drivers as cost: how many topics are involved, how stable their schemas are, how complex the mappings must be, how strict the security requirements are, and how many environments need to be prepared. A pipeline for a few topics with clean, documented schemas and an existing OpenSearch cluster moves much faster than one that combines new roles, new clusters and a long list of undocumented record formats. Schedule time for review cycles with the data owners, for testing malformed records, and for the change control that production search clusters usually require. Ask for a timeline built from a written scope with named milestones, rather than a single estimate for the whole project.

Does OpenSearch licensing add cost to the pipeline?

OpenSearch is released under the Apache License, Version 2.0, and the OpenSearch FAQ does not describe any paid edition of the software. The costs that remain are infrastructure, which includes compute, storage, network capacity and backups in the environment you operate, plus the staff time to run and secure the cluster. The sink connector also has its own license, which you should check separately before you depend on it. Confirm the license of the specific connector package and its dependencies with the publisher, and record the result in your architecture review. Any vendor offering a paid support or hosted option is a separate purchase, and it should be priced against a written scope.

What happens if the sink connector changes or is abandoned?

Before you depend on a connector, confirm its current maintenance status, the release history, the open issues related to your Kafka and OpenSearch versions, and its license. Check whether the project is actively maintained and whether it supports the Kafka Connect version you run. If the connector is abandoned, the pipeline does not stop immediately, but security fixes and compatibility updates stop, and upgrades to Kafka or OpenSearch become riskier. Plan an exit path in advance: pin the connector version, keep the configuration in version control, and document the replacement option, which may be another connector or a small stream processor. Re-check maintenance status on a regular schedule, not only at launch.

How should TLS and access control be set up between Kafka Connect and OpenSearch?

Use TLS for traffic between the Connect workers and OpenSearch nodes, and give the sink a dedicated OpenSearch user with only the index and write permissions it needs. OpenSearch Security provides TLS for node and client traffic and role-based access control, so the sink's role can be limited to specific index patterns. Trust only your own certificate authority in the Connect worker truststore, and store credentials in a secrets mechanism rather than in plain connector configuration. Test the connection with an invalid certificate and a wrong password to confirm that failures are visible in logs. Confirm the exact configuration keys against the sink's documentation, since key names differ between connector versions.

How do index naming, document IDs and mapping errors affect a later migration?

Index naming, document IDs and mappings are the decisions that are hardest to change once data is in production. A stable document ID, taken from a key that the source system guarantees to be unique, prevents duplicates when a record is redelivered. Index names that include a date or a version make retention and reindexing easier, but they also multiply the number of indices and the cluster's shard count. Mapping conflicts, such as a field that arrives as a string in one message and a number in another, cause rejected documents that go to your error route. Test these cases with sample records before go-live, and define what happens to a rejected record, whether it is stored in a dead letter topic, logged, or retried after a mapping change.

How DeployOpen scopes and delivers a Kafka to OpenSearch integration

DeployOpen plans and builds the Kafka to OpenSearch path inside your own infrastructure, starting from the topics, index naming, document IDs and mapping rules your teams need. Each engagement is fixed-scope, with deliverables and timelines agreed in writing before work starts.

How we scope

Inventory the source topics, their producers, message formats and the OpenSearch indices they should feed.
Agree in writing how index names, document IDs and schema handling will work for each topic.
Confirm the OpenSearch version, Kafka Connect deployment, TLS and authentication requirements before any build begins.

How we build

Configure the OpenSearch sink connector in a test environment, with connector settings and error handling reviewed against the scope.
Test mapping error behavior using sample records, including malformed documents and unexpected field types.
Verify TLS to OpenSearch and the Kafka access controls for the connector's service account in your environment.

How we hand over

Document index naming, document ID rules, mapping decisions and connector configuration for your operations team.
Provide runbooks for reprocessing failed records and for adding a new topic to the pipeline.
Walk your engineers through the test evidence so they can repeat the checks after handover.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer, scoped separately from the build.
Plan connector and OpenSearch upgrades against the version support status you confirm with each project.
Review new topics and mapping changes before they reach production indices, under the agreed retainer scope.

Indexing Kafka topics into OpenSearch