How OpenSearch Data Prepper log ingestion works
A team that needs logs searchable in OpenSearch usually needs them parsed, enriched and routed before search or dashboards are useful. Data Prepper is the ingestion component from the OpenSearch project for building pipelines that do this work, moving log records from their origin into OpenSearch indices in a form that queries and dashboards can rely on. Because it is part of the same project, it is the first tool to evaluate when the destination is OpenSearch and the logs need shaping on the way in.
Data Prepper uses a pipeline model with three stages. A source receives or pulls events, such as log records sent by an agent or read from a messaging system. One or more processors then transform those events, for example by parsing unstructured text into fields, adding context, dropping noise or applying conditions that decide where an event goes. A sink writes the processed events to OpenSearch or another destination. Thinking in these three stages helps a team map each log type to a concrete chain of components before any configuration is written.
Exact component names, processor options, configuration syntax and supported OpenSearch versions must be confirmed in the Data Prepper documentation for the OpenSearch version you run. The pipeline stages described here are the general model only. Do not copy example configuration from older blog posts or other versions, because source and processor options change between releases and an unsupported option can stop a pipeline from starting. Record the exact Data Prepper and OpenSearch versions in your design document so that every later decision refers to the same documentation.
The practical decisions come before configuration. List each log family, its format, expected volume, the fields that searches and alerts depend on, and the index naming and retention each one needs. Decide which parsing belongs in Data Prepper and which can stay with the collecting agent, and whether routing rules should send different log types to different indices or tenants. Confirm how the pipelines will be deployed and restarted, how configuration changes are reviewed, and who owns a pipeline when its logs stop arriving.
Plan for failure from the start. Test how a pipeline behaves when OpenSearch rejects bulk writes, when a log format changes without notice, and when a processor receives malformed records. Check whether failed events are retained, redirected or lost under your chosen version and configuration, and confirm that the cluster's ingest capacity and the sink's indexing settings match the expected peak load. Write these results into the runbook before the pipeline carries production logs.
What you need before the first Data Prepper pipeline runs
Each item below should be settled before the first pipeline file is written, because a Data Prepper pipeline depends on a cluster it can authenticate to, reach over the network and write into under narrow permissions. Working through them in this order means version, identity and network decisions are made before any configuration is copied into a pipeline.
A supported OpenSearch cluster
Start with a running OpenSearch cluster on a version that is still in maintenance or general availability, with the Security plugin configured and enabled. OpenSearch versions receive bug fixes and security patches during maintenance, and version 1 reached end of life on May 6, 2025. Confirm the support status of your chosen minor version on the releases page before you build anything on top of it.
A planned node layout
Decide which nodes run the cluster manager, data, ingest and coordinating roles before the cluster is sized. The OpenSearch documentation recommends dedicated cluster manager nodes, replicas placed on separate nodes and shard allocation awareness for availability. Write the layout down with the failure domains you intend to use, because changing node roles later on a busy log cluster is harder than deciding them now.
A matched Data Prepper version
Pick the Data Prepper release that matches your OpenSearch version, and do not assume that the newest release of either product works with the other. Read the compatibility statements in the Data Prepper release notes for the exact OpenSearch version you run, and record the pairing in your change plan. Repeat this check whenever either side is upgraded.
A least-privilege pipeline role
Create a dedicated OpenSearch user for the pipeline and map it to a role that grants only the index patterns, actions and cluster permissions the pipeline needs. Pipelines that write daily log indices usually need index creation and bulk write rights on a named pattern, and seldom need administrative access. Review the role with the security team, and confirm which actions your Data Prepper version requires before granting them.
Network paths and certificates
Map every connection between log senders, Data Prepper and OpenSearch, including ports, firewall rules, DNS names and the direction of each flow. Issue certificates that match those names and confirm that TLS is enabled for both node and client traffic. Test the paths with the same hostnames and trust chain the pipeline will use, because a certificate that works from an administrator workstation may still fail inside the pipeline's network segment.
A test index and sample logs
Prepare a non-production index name, index template and a representative sample of log lines from each source that will feed the pipeline. The sample should include malformed lines, multi-line events and fields that carry personal or sensitive data, so that parsing and masking rules can be tested before production traffic arrives. Keep the sample and expected output together so that the first pipeline run can be checked against them.
When Data Prepper fits better than Logstash or Fluent Bit
Choose Data Prepper when the pipeline should stay inside OpenSearch project tooling and your team is ready to own its configuration, testing and upgrades. The first question is which collector your team already runs. If Logstash or Fluent Bit is established across your hosts and has working runbooks, a switch affects every log source that feeds the pipeline, not only the server that receives the events. Changing the collector on each host can be the largest part of the effort, so size it separately from the pipeline design. Ask the operators who would respond to an alert at night which tool they can already diagnose.
Next, locate the parsing logic. Map where each rule lives today: grok patterns, multiline joins, field renaming, enrichment from lookup tables, drops and routing. If most of that logic sits in Logstash filters, moving it to Data Prepper means rewriting each rule as a pipeline processor and proving that the output matches. If parsing already happens at the edge in Fluent Bit parsers, keeping it there may be simpler, with Data Prepper handling the routing and enrichment that follow. Count the rules, note which ones depend on plugins you use today, and check each one against the current Data Prepper processor documentation before deciding.
Then decide how much of the stack must stay within OpenSearch project tooling. Confirm in the project documentation how Data Prepper versions align with OpenSearch releases, and check that the version you plan to run supports the OpenSearch version in your cluster. OpenSearch versions in maintenance receive bug fixes and security patches but no new features, and new minor versions are targeted about every eight weeks, so set a fixed schedule for pipeline upgrades. Your team will own the pipeline definitions, the secrets for output connections, dead-letter handling and the tests that prove each change. If that maintenance load is not staffed, the collection layer may be better served by the tool your team already maintains.
Do not settle this on general reputation or feature lists alone. Build a short evaluation for each candidate using your own log sources, a sample of your parsing rules, your average and peak throughput, and the skills your operators have today. Record parse failures, dropped or duplicated events, behavior when OpenSearch is unavailable, the steps to restart or reload configuration, and the effort needed to change one rule. Confirm current capabilities for each candidate in that project's documentation at the version you plan to deploy. Write the decision with the test evidence beside it so that the next review can repeat the same tests.
How to build a Data Prepper log pipeline in sequence
This sequence moves a Data Prepper log pipeline from design to production traffic in controlled stages. Each step ends with a check the team records, so a failed check stops the rollout before unvalidated logs reach production indices.
Confirm cluster health and pipeline role
The platform owner confirms the OpenSearch cluster reports a healthy status and that the node roles the design depends on are in place, including dedicated cluster manager nodes where the topology calls for them. The team then names the component that runs Data Prepper, the nodes that receive its writes and the network path between them. Record the cluster health output and the role assignments in the change ticket. The step is closed when the health check passes and every team member can point to the host that runs the pipeline and the nodes it writes to.
Define the source and its authentication
The pipeline owner selects the source plugin that matches how the logs arrive and confirms it is supported by the Data Prepper version in use. Configure TLS for the source endpoint and issue a dedicated credential for each sending system. Keep the credentials in your secrets store and reference them from the pipeline configuration instead of writing them into the YAML file. The step is closed when a sender with a valid credential is accepted, a sender with an invalid credential is refused, and both results are logged.
Write parsing and enrichment processors on a sample file
The engineer takes a representative sample of production-style log lines with sensitive values removed and writes the processors that parse fields, normalize timestamps, rename fields to the agreed schema and add enrichment such as environment or service labels. Run the pipeline against the sample file in a local or test setup and inspect the output documents. The step is closed when every sample line produces the expected fields, and any line that fails to parse is tagged so it can be counted and reviewed rather than silently dropped.
Define the OpenSearch sink, index naming and permissions
The engineer configures the OpenSearch sink with the target endpoint, the index naming pattern and any index template or mapping the team has agreed on, including whether a date-based index or a data stream fits the retention plan. A dedicated role is created for the pipeline user with write access only to the pipeline's indices and no cluster administration rights. The step is closed when the pipeline user can write to the target index, and a test request confirms it cannot read unrelated indices or delete documents.
Deploy to a non-production cluster
The platform team deploys the pipeline and its supporting components to a non-production OpenSearch cluster. The install method should follow the install options documented for the OpenSearch version you run, which include Docker, Helm, the OpenSearch Kubernetes Operator, tarball, RPM, Debian packages, an Ansible playbook and Windows. Choosing the method is a planning decision that should be made with your platform standards in mind before this step starts. The step is closed when the pipeline reports a running state and its logs show no startup or authentication errors.
Send a controlled test set and verify results
The team sends a known test file with a line count recorded in advance, including a few deliberately malformed lines to exercise the failure path. The engineer then compares the document count in the target index with the expected count, accounting for the lines the parser is designed to reject. Spot-check field values, timestamps and data types against the source lines. The step is closed when the counts match with every difference explained, and the field values in the spot checks equal the source values.
Enable production senders and set monitoring
The operations team enables production senders in stages, starting with a small group of sources and widening the set as the pipeline holds steady. Monitoring covers the pipeline's throughput and error counts, the cluster's indexing rejections, heap, disk and node availability, and the age of the newest indexed document. Alert thresholds are agreed with the service owners before the rollout starts. The step is closed when a deliberately triggered test alert reaches the on-call channel and the dashboard shows live ingestion from each enabled sender.
How log fields, indices and roles map into OpenSearch
This table lists the decisions that connect a log record to its stored form in OpenSearch, from the source field through index type, index pattern, pipeline identity and role. Read each row as a decision to make and record before the first pipeline runs, because changing one later usually means reindexing or rewriting access rules. The main takeaway is that field types, index layout and roles should be designed together, not patched after logs arrive.
| Mapping step | What to decide | Confirm before go-live |
|---|---|---|
| Source log field to parsed field | Which raw keys become named fields after Data Prepper parsing, and which stay in the raw message only. | Sample real records from each source and confirm that every field you plan to query or filter on is extracted. |
| Parsed field to index field type | Whether each parsed field is keyword, text, date, numeric or object, based on how analysts will search and aggregate it. | Check field types against your chosen OpenSearch version and test a mapping on a non-production index first. |
| Log stream to index pattern or alias | Which sources share an index or pattern, and whether writes go through a write alias that rolls over. | Confirm the index naming, rollover and alias behavior in the OpenSearch version you deploy. |
| Pipeline identity to OpenSearch role | Which service account or identity each Data Prepper pipeline uses to write, and whether pipelines share credentials. | Give each pipeline its own identity where your audit model requires per-source attribution, and confirm authentication settings with the Security plugin docs. |
| Role to index and document-level permissions | Which roles can read or write which indices, and which documents a role may see based on a query filter. | Test each role with a sample user and confirm that the document-level filter returns only the intended records. |
| Role to field-level permissions | Which fields are hidden or masked for each role, such as personal data or credentials in log messages. | Verify masked fields in Dashboards and in direct API queries, since masking applies per role. |
| Retention plan to index lifecycle | How long each index group is kept and when it moves or is deleted, which must match the retention rules you have agreed internally. | Confirm the retention period and lifecycle settings with the team that owns the policy, then test deletion on a copy. |
Securing the pipeline's identity, transport and log data
A Data Prepper pipeline holds credentials, reads logs from many senders and writes into an index the rest of the organization may search. The controls below cover who the pipeline is, how its traffic is protected, what data it can see, and what gets recorded, and each one needs to be checked against your own cluster configuration.
Dedicated pipeline user and role
Create a separate OpenSearch user for Data Prepper instead of reusing an administrator or a person's account. Bind it to a role that grants write access only to the target index pattern, plus the minimum cluster permissions the sink needs. Review the role after the first test run, because a pipeline that only writes rarely needs search or index management rights. Confirm the exact permission names against the Security plugin documentation for your version.
TLS for node and client traffic
Enable TLS for transport between cluster nodes and for HTTP connections from Data Prepper and any client that queries the cluster. Confirm that the pipeline's sink settings verify the server certificate rather than disabling verification. Check that plaintext ports are closed to the network segments where senders and pipelines run.
Certificate issuance and rotation
Decide which certificate authority signs the node certificates, the Data Prepper client certificate and the certificates on log senders, and record who renews each one. Set expiry dates in your monitoring so a lapsed certificate does not stop ingestion silently. Test a rotation in a non-production cluster before the first production expiry, and confirm that Data Prepper reloads or restarts cleanly with the new files.
Pipeline credential storage
Keep the pipeline password, client key and any token out of the pipeline YAML file in version control. Load them from a secrets store or from files with restricted ownership on the host or container, and limit which service accounts can read them. Confirm how your deployment method passes secrets into the container, because environment variables and command-line arguments can appear in process listings and logs.
Authentication backend selection
Choose the backend the cluster uses for human and service logins from the options the Security plugin supports: SAML, OpenID Connect, Active Directory, LDAP, Kerberos or JSON Web Token. Decide whether the pipeline user authenticates through the same backend or through the internal user database, and document the reason. Confirm the chosen backend's settings, group mapping and failover behavior in your own environment before relying on them.
Field masking for sensitive log content
Identify fields in your logs that contain personal data or secrets, such as account numbers, email addresses, session tokens or authorization headers. Use field-level security or field masking on the roles that query those indices so that only approved groups see the raw values. Apply the processor-side redaction in the pipeline where possible, because masking at query time does not remove the data from storage. Test the masking with real sample events from each sender.
Audit logging of pipeline access
Turn on the Security plugin audit log and decide which categories to record, such as failed logins, index writes and changes to roles or users. Send the audit output to a location that the cluster administrators cannot quietly alter, and set a retention period that matches your policy. Verify that a write by the pipeline user appears in the audit trail and that a rejected write is also recorded.
Testing Data Prepper pipelines before production traffic
A Data Prepper pipeline should not receive production logs until a written test plan has shown that it parses, enriches, routes and recovers the way the operations and security teams expect. Start by agreeing on acceptance criteria before any test runs. Write them down with the security owner and the operations owner, covering which log classes must be parsed, which fields must be present after enrichment, which indices each class must land in, how long recovery may take after a failure, and who signs off on each result. Criteria written after the results are in tend to be fitted to what the pipeline already does.
Parse testing comes first. Build a set of representative sample files from each source you plan to onboard, taken from a non-production environment or from sanitized exports. Include deliberately malformed lines: truncated records, mixed encodings, unexpected delimiters, multi-line stack traces, and events with missing timestamps. For each sample, check the documented output of the grok, date or key-value processors you configured, and confirm that malformed lines are handled as you intended. Either they are tagged and sent to a dead-letter or quarantine index, or they are dropped and counted. Silent loss of malformed lines is the failure to rule out first.
Enrichment and routing tests come next. Check every lookup or enrichment step against the reference data it uses, including what happens when a key is absent or the lookup source is stale. Then run routing tests that send each log class through the pipeline and confirm that it reaches the index you specified, and only that index. Include edge cases such as a record that matches two routes. Load tests should run at the peak event rate and burst pattern your sources are expected to produce, with headroom agreed in advance. Record the measured rates against that plan rather than against a number chosen after the fact.
Failure tests are where many pipelines are first proven. Stop and restart individual OpenSearch nodes, including a cluster manager node if your topology has dedicated ones, and observe whether Data Prepper buffers, retries and resumes without duplicating or dropping events beyond the criteria you agreed. Restart the Data Prepper process itself and confirm that the source acknowledgments and buffer behavior match your configuration. Confirm that shard allocation and replica settings keep the indices writable during the test. Run these tests on the same version and deployment method you will use in production, and confirm the exact behavior with the Data Prepper and OpenSearch documentation for those versions.
Finish with a rollback test. Practice reverting the pipeline configuration to the previous version, redirecting sources back to the former path such as Logstash or Fluent Bit if you run both during cutover, and removing any index templates or routing changes you introduced. Time the steps with the operations team and record who must approve each one. A rollback that has never been rehearsed is an untested change. Store the test results, sample files and acceptance sign-offs with the change record so that auditors and the next engineer can see what was verified and when.
Why logs stop arriving and how to fix them
This table lists the failures most often seen when Data Prepper log pipelines stop delivering events to OpenSearch, with the likely cause and first fix for each. Use it as a starting point, then find the exact error text in the Data Prepper and OpenSearch logs before changing settings. The main takeaway is that many stoppages come from authentication, permissions, parsing or retention settings that can be checked before go-live.
| Symptom | Likely cause | First fix |
|---|---|---|
| Pipeline fails to start after a version change | Pipeline configuration or plugin options no longer match the new Data Prepper release. | Check the startup log for the exact error text, compare your settings with the release notes, and correct the pipeline file before restarting. |
| Authentication or TLS errors between Data Prepper and OpenSearch | Wrong credentials, a certificate Data Prepper does not trust, or a host name that does not match the certificate. | Confirm the user, password and certificate paths, and check that the CA chain and host names match the cluster's TLS setup. |
| Role lacks index write or create permission | The user mapped to the pipeline cannot create indices or write documents to the target index pattern. | Grant the minimum index-level write and create permissions on the target pattern, then retest with a single sample event. |
| Parsing rules drop or mangle fields | Grok or other processor patterns do not match the real log format, or a later step overwrites a field. | Test patterns against sample lines from each source and inspect each processor's output before enabling the sink. |
| Index mapping conflicts from changed field types | A field arrives as a string in one event and as a number or object in another, so OpenSearch rejects documents. | Fix the field type in the parser, and create a new index with an explicit mapping if the existing mapping is already wrong. |
| Unbounded index growth from missing retention | No lifecycle policy deletes or rolls over old indices, so storage fills gradually. | Define index lifecycle and retention rules before go-live, and monitor index size and count for each data stream. |
| Back pressure under cluster load or node capacity limits | Bulk requests slow or fail when the cluster is saturated, and the pipeline buffers until it stalls. | Check cluster health, heap and disk pressure, then tune bulk size and buffer settings or add data node capacity. |
What do buyers ask before hiring help for Data Prepper ingestion?
What drives the cost of a Data Prepper ingestion project?
The main drivers are the number of log sources, parsing complexity, enrichment sources, security review, testing scope and deployment method. Each log source adds sample collection, a parser to write and test, and routing rules to verify. Parsing gets harder when formats are multi-line, deeply nested or inconsistent across teams. Each enrichment source, such as an asset inventory or identity directory, adds a connection, credentials and failure handling. Security review grows with the number of network paths and the sensitivity of the data. Deployment method matters because pipelines on containers, virtual machines or bare hosts need different packaging and operations. When you request a quote, ask for one-time build work to be separated from recurring operations, and ask which testing is included.
Can our own team build Data Prepper pipelines without a partner?
Yes, if your team already has OpenSearch administration skills and experience running log pipelines. The decision comes down to skills and available time. Engineers need to understand index design, cluster roles and the pipeline configuration model, and they need to test parsers against real log samples. A partner is most useful when the team lacks that experience, when the schedule conflicts with production support, or when an assessment needs documented controls and evidence on a fixed date. To estimate internal effort, list the sources, parsers, enrichment steps and test cases, then compare that list with the hours your staff can commit without pulling them from operations. Include the ongoing work of maintaining pipelines after go-live.
What inputs do we have to provide before a Data Prepper project starts?
You need to provide representative log samples, cluster access, network approvals and named owners. Samples should include normal events, error cases and every format variant, because parsers are tested against them. Cluster access means accounts or a change-controlled path into the OpenSearch environment and the systems that emit logs. Network approvals cover flows from log sources to the pipeline host and from the pipeline to the cluster, and your network and security teams often need to review them. You also need an owner for index retention, who decides how long each log type is kept and where it is stored. A security sign-off owner should accept the access model, TLS settings and audit logging. Confirm approval lead times with your change board early, because they often set the schedule more than the build does.
How long does a Data Prepper log ingestion project take?
Duration depends on scope. The main factors are the number of log sources, parsing and enrichment complexity, security review, testing depth and deployment method. A project with few sources and clean formats moves faster than one with several enrichment lookups and formal test evidence. Network and security approvals can run alongside build work, but they often become the critical path. Retention decisions and sign-off also affect timing, because changes made after testing are costly to repeat. Ask for a timeline in the written scope that lists milestones, the client inputs each milestone depends on and the assumptions about sample availability. Treat any date set before samples and approvals are in hand as provisional.
Should we move from Logstash or Fluent Bit to Data Prepper?
Move only when you have a specific operational reason, such as consolidating ingestion on OpenSearch-native tooling, and only after you inventory what your current pipelines do. Logstash and Fluent Bit already run many production pipelines, so a move should be justified by operational needs alone. Start by listing every input, filter, output and enrichment step in the current configuration. Map each one to a Data Prepper equivalent, and confirm those equivalents against the current Data Prepper documentation for your version. Mark gaps that need custom work or a different design. Run old and new pipelines in parallel on the same samples and compare outputs before cutover. Validation is where migration effort tends to concentrate, so plan test time explicitly and keep a rollback path for each source.
How should Data Prepper pipelines authenticate and handle secrets?
Give each pipeline its own OpenSearch identity with permissions limited to the indices it writes, and keep credentials in a managed secret store instead of plain configuration files. Use TLS between components and verify certificates. Apply role-based access so that only the pipeline role can write to ingest indices, and restrict who can change pipeline configuration, because a parsing change can drop or expose fields. The OpenSearch Security plugin provides role-based access control, field-level security, TLS and audit logs, but confirm the exact role and audit settings for your version in its documentation. Enable audit logging for write activity where your security team requires it. Security sign-off should cover these items before production traffic flows.
How DeployOpen scopes and delivers Data Prepper ingestion
DeployOpen plans OpenSearch Data Prepper log ingestion as part of the cluster it feeds, in infrastructure you control. Pipeline design, parsing and enrichment rules, and the OpenSearch access and retention settings are agreed before any build starts. Engagements are fixed-scope, with scope, deliverables and timelines confirmed in writing first.
How we scope
- Written scope lists each log source, its format, and the parsing and enrichment each pipeline must perform.
- Routing decisions set which indices receive which logs, and how index naming and retention are agreed before build.
- Engagements are fixed-scope, and ongoing support for the pipelines is available as a retainer.
How we build
- Pipelines are built in a non-production environment first, with sample logs from each source you provide.
- Each pipeline is tested against malformed, oversized and unexpected events to confirm how failures are handled and recorded.
- Access roles, TLS for client traffic and each pipeline's writing identity are set through the OpenSearch Security plugin.
How we hand over
- Handover includes pipeline configuration files, a source-to-index map, and the access and retention settings in use.
- Runbooks cover adding a log source, changing a parser, rotating credentials and responding to a stalled pipeline.
- Documentation is written so your staff can run the pipelines after the engagement ends.
How we support
- Retainers cover pipeline changes, OpenSearch upgrades that affect ingestion, and reviews of new log sources.
- Upgrade planning checks Data Prepper changes against your OpenSearch version's support status, confirmed on the releases page.
- Index lifecycle, shard sizing and ingestion monitoring are reviewed with your team on the cadence agreed in the retainer.
