How to migrate Datadog logs to OpenSearch without losing the trail
Moving logs from Datadog to OpenSearch is a pipeline and rebuild project that runs in parallel with Datadog, not a switch you flip on a renewal date. Keep Datadog ingesting while OpenSearch is built out, so you have a working reference for every log source until the new system matches it. OpenSearch is released under Apache-2.0, is a fork of open source Elasticsearch 7.10, and ships OpenSearch Dashboards derived from Kibana 7.10.2. Because OpenTelemetry Collectors can export the same log stream over OTLP to more than one destination, a single pipeline can feed both systems during cutover, which lets you compare event counts and field values before you stop anything. The work is usually done by a platform or SRE team, with a security engineer owning access control, audit logging and retention decisions. Start by inventorying the Datadog side: which log sources are shipped by the Datadog Agent and which by other shippers, which processing pipelines parse and enrich them, and which saved views, log-based monitors and dashboards depend on them. Effort scales mainly with the number of pipelines and the number of dashboards and monitors that query them, so ask for that inventory in any quote. Confirm with your team whether your Datadog Log Archives will be exported to your own object storage, since archived JSON is the history you will load into OpenSearch indices.
Build the OpenSearch side before cutover. Stand up the cluster with one of the documented install methods, then configure the Security plugin for SAML or OpenID Connect sign-in, role-based access, document-level and field-level security for sensitive fields, and audit logs. Define index templates and index lifecycle policies that match the retention you actually need, because storage is the cost that recurs. Then rebuild each Datadog parsing rule as an ingest or Collector processor, and check field names carefully, since a renamed field silently breaks saved views and alerts. Load archived history into separate indices so it does not mix with live data, and validate counts per source per day against the Datadog archive before you retire a pipeline. Keep Datadog running until every alert and dashboard you depend on has been recreated and tested against OpenSearch data.
Which Datadog log capabilities move cleanly to OpenSearch and which need rework
This table maps the main Datadog log management capabilities to their OpenSearch equivalents and marks each as clean, needs rework or gap. Read it as a planning checklist: clean rows still need validation with your own log volumes, while rework and gap rows set the scope of your migration work. Features such as Live Tail and Watchdog-style analytics need confirmation in your target environment before you commit.
| Capability | OpenSearch equivalent | Status and what to confirm |
|---|---|---|
| Log ingestion | Data Prepper, Fluent Bit, Logstash or OpenTelemetry Collector exporters into OpenSearch indices | Needs rework: re-point Datadog Agent log shipping to a new shipper and confirm field names and tags survive the change |
| Parsing pipelines | OpenSearch ingest pipelines with processors, plus Collector processors upstream | Needs rework: Datadog Grok and pipeline processors must be rewritten and tested against sample events |
| Full-text search and query | OpenSearch Dashboards Discover and the OpenSearch query DSL | Clean for basic search; confirm query syntax differences for facets and wildcard patterns |
| Saved views and dashboards | OpenSearch Dashboards saved searches and visualizations | Needs rework: recreate views manually or from exported definitions, since no direct import from Datadog is assumed |
| Log-based monitors and alerts | OpenSearch Alerting plugin monitors | Needs rework: translate each monitor's query, threshold and notification route one by one |
| Log archives and history | Datadog Log Archives as compressed JSON in your object storage, loaded with bulk indexing | Needs rework: confirm the archive format and plan index templates before loading history |
| Role-based access control | OpenSearch Security plugin roles and tenants | Clean for index and cluster roles; confirm mapping of Datadog team and org permissions |
| Document and field-level security | OpenSearch Security plugin document-level and field-level security, plus field masking | Clean, if your Datadog restrictions can be expressed as index, document or field rules |
| Audit logs | OpenSearch Security plugin audit logging | Clean for cluster activity; confirm which events your assessors need and where they are stored |
| Live Tail and Watchdog-style analytics | No direct equivalent assumed | Gap: confirm in the target environment with a test cluster before planning around these features |
What data and integrations can leave Datadog and what must be rebuilt
Log history is the part of a Datadog move that has a documented export route, so it is the natural starting point. Everything that defines how logs are collected, enriched, searched and acted on in Datadog has to be recreated in OpenSearch and its surrounding tooling. Plan the two tracks separately so that history loading does not block the rebuild of live ingestion.
Log history through Log Archives
Datadog Log Archives can write logs to your own object storage bucket as compressed JSON. That gives you a copy of past logs that does not depend on Datadog staying under contract. Confirm which cloud bucket type your archive configuration supports, how the files are partitioned by date, and whether the archive covers only the indexes you need. Plan a bulk loader that reads these JSON files and writes them into OpenSearch indices with the field mapping you chose.
Agents, shippers and collectors
Datadog Agent log collection, and any Datadog-specific log shipping config, must be replaced with a self-hosted shipper or the OpenTelemetry Collector that sends to OpenSearch. Decide per host whether a Collector runs as an agent beside services or as a gateway that receives from many sources. Check that each source type still emits the fields your parsing and dashboards expect, because tags and attributes added by the Datadog Agent will not appear automatically.
Pipelines, parsing and enrichment
Datadog log pipelines handle grok-style parsing, attribute remapping, status mapping and enrichment with service and host tags. These rules do not transfer as files. Inventory each pipeline, record which processors it uses, and rebuild the same transformations in the Collector or in your ingest layer. Test with sample records from the archive so you can compare field names before and after cutover.
Saved views, dashboards and monitors
Saved views, log-based dashboards and log monitors are Datadog objects that describe queries, facets and alert thresholds. You can export dashboard definitions as JSON through the API, which serves as a reference, but each query must be rewritten in OpenSearch Dashboards query syntax. Recreate each monitor as an alerting rule and confirm that notification channels, grouping and recovery behavior match what your on-call team expects.
Integrations and user roles
Datadog integrations for cloud services and frameworks ship their own log parsing and dashboards, which you must rebuild or replace one by one. User accounts, Datadog roles and team-based access to log indexes need a mapping to OpenSearch Security roles, and identity federation should reuse your existing provider through SAML, OpenID Connect or LDAP. Confirm early which users need access to which indexes, since document-level and field-level security rules must be written explicitly.
Other telemetry may lack bulk export
Log history has a documented export path, but the same is not established for every telemetry type. Confirm with Datadog whether metrics and traces can be retrieved in bulk for your account, and plan a dual-shipping period for them instead of assuming a copy. Keep the decision about retained metrics and traces separate from the log migration so that a gap in one track does not hold up the other.
The migration sequence from inventory to Datadog decommission
This sequence moves Datadog log management to self-hosted OpenSearch without a single cutover day that risks lost logs or broken alerts. Each step ends with a written output that the next step depends on, so the team can prove parity before Datadog is switched off.
Inventory log sources and pipelines
List every host, container, serverless function, and application that sends logs to Datadog, along with the Datadog Agent configuration, log pipelines, processing rules, exclusion filters, and index settings that shape each stream. Record who owns each source and which teams query it. The output is a source register that names each source, its owner, its current pipeline, and its business criticality. Ask the Datadog administrator to export pipeline and index configuration so nothing is recreated from memory.
Build the OpenSearch pilot index
Stand up a self-hosted OpenSearch cluster in your own infrastructure, using one of the documented install methods such as Docker, Helm, the OpenSearch Kubernetes Operator, or packages, and enable the Security plugin for TLS, role-based access control, and audit logging before any data arrives. Pick one low-risk service and create an index template for its fields. The output is a validated field mapping that shows how Datadog attributes, tags, and timestamps map to OpenSearch fields, reviewed by the engineers who own that service.
Run a parallel pipeline through a Collector fan-out
Deploy an OpenTelemetry Collector between the log sources and the backends. Configure a pipeline that receives the logs and exports them to both Datadog and OpenSearch at the same time, with the pilot service first. Use processors to normalize fields so both sides receive comparable records. Authenticator extensions can secure the receiving and exporting endpoints. The output is a running dual-write path and a comparison report of record counts, timestamps, and sample field values between the two systems over an agreed observation window.
Rebuild dashboards, saved views, and monitors
Recreate each Datadog saved view and log-based monitor as an OpenSearch Dashboards visualization, saved search, or alerting rule for the pilot service, then expand to the other sources. Compare each rebuilt item with its Datadog original using the same time range and the same sample data. The output is a dashboard parity sign-off recorded by the owning team, listing each Datadog item, its OpenSearch equivalent, any intentional differences, and the approver.
Cut over source groups in stages
Move sources to the OpenSearch path in groups, starting with the least critical and ending with the services that produce audit or security-relevant logs. For each group, remove its Datadog export only after its parity sign-off is complete, and keep the dual-write path available for a rollback window agreed in advance. The output is a cutover log that records the date, the source group, the rollback decision, and the owner who accepted the change.
Load archived history into OpenSearch
If Datadog Log Archives were configured to write compressed JSON to your own object storage, load that history into dated OpenSearch indices using an ingestion job you test on a sample first. Map archived fields with the same template used in production so historical and live queries match. Confirm the index lifecycle policy before loading large volumes, since sizing drives storage cost. The output is a historical load report with record counts per time range, checked against the archive manifests, and a retention record stating which periods are searchable in OpenSearch and which remain only in the archive.
Decommission Datadog log ingestion
Once every source group is cut over and history is confirmed, disable the remaining Datadog log exports, revoke Datadog API keys used for log ingestion, and retain or delete archive buckets according to your records policy and any legal hold. Remove the dual-write exporter from the Collector configuration. Confirm with the vendor how the account is closed and what happens to stored data at that point. The output is a decommission record that lists the revoked credentials, the final export date, the archive location, and the owner who signed off.
How OpenSearch security and OpenTelemetry fit the target design
The target flow places an OpenTelemetry Collector gateway between log sources and OpenSearch. Application hosts, containers and the Datadog Agent or OpenTelemetry agents send logs to the gateway over OTLP, and the gateway exports them to the OpenSearch cluster. This keeps sources independent of the storage backend, so the cluster can be changed later without re-instrumenting every host. On the OpenSearch side, the Security plugin is not optional for a regulated deployment: enable TLS for both node-to-node and client traffic, set role-based access control so ingest, search and administration are separate roles, and apply document-level and field-level security where some teams should see only part of a log stream. Enable audit logs so administrative and access events are recorded outside the application. Confirm the exact configuration keys and plugin behavior for your version in the OpenSearch security documentation before you write role mappings.
Authentication has two directions, and both need a decision. For incoming connections, Collector authenticator extensions can require credentials on the OTLP receiver, for example with OIDC, so unknown agents cannot push logs into the gateway. For outgoing connections, the oauth2client extension or a comparable mechanism can attach credentials to the exporter requests that reach OpenSearch. Store those secrets in your secrets manager rather than in plain configuration files, and restrict who can change the gateway configuration. The OpenTelemetry Collector configuration best practices cover these controls and should be reviewed against your gateway settings before go-live. Check the extension names and options against the release you deploy, because the Collector ships frequently and component stability varies.
Before you choose an install method, confirm the current OpenSearch version and the way you will deploy it. The OpenSearch install documentation lists Docker, Helm, the OpenSearch Kubernetes Operator, tarball, RPM and Debian packages, an Ansible playbook and Windows. Your choice should follow your platform standards and patching process, since it determines how security settings, certificates and upgrades are managed. Check the release support window on the OpenSearch releases page so you pick a version that will still receive security patches during the planned operating period, and confirm the Collector release you pin, because its contrib components and their stability differ by release.
How the migration changes by starting point and team size
Teams that run only the Datadog Agent with its log collection enabled usually have the most rework ahead of them. Every log source is configured in Agent integration files, so each source must be inventoried, mapped to a log shipper or OpenTelemetry Collector pipeline, and sent to OpenSearch before the Agent configuration can be switched off. The work is mostly configuration and parsing, not application change.
- Inventory first: List every log source enabled in the Agent, including integration defaults that were never reviewed.
- Replace the shipper: Choose between the OpenTelemetry Collector and another shipper, then rebuild parsing rules that Datadog's pipelines previously applied.
- Dual-ship during cutover: Keep the Agent sending to Datadog while the new path writes to a test OpenSearch index, and compare counts per source.
- Rebuild saved views and monitors last: These depend on fields that must already exist in OpenSearch with the same names.
- Plan the Agent removal as its own step: Turn off log collection source by source rather than in one change.
Before starting, confirm with your team which Datadog log sources carry the most volume and which pipelines alter fields, because those drive the rebuild effort.
Teams that already send traces, metrics or logs through OpenTelemetry have a shorter path, because the collection layer exists and only the destination changes. Datadog documents OTLP ingestion alongside its Agent, so logs can be exported through a second exporter to OpenSearch while the existing pipeline stays in place. Effort concentrates on the exporter, field naming and index design rather than on instrumentation.
- Add an OpenSearch exporter alongside the existing one: Use the Collector's processors to keep the same resource attributes in both destinations.
- Align field names early: Agree on attribute names for service, environment and trace identifiers so that correlation works in OpenSearch Dashboards.
- Confirm exporter availability in your build: Check the Collector distribution in use, since the contrib distribution carries more components than a minimal build.
- Keep trace-log correlation in view: Verify that trace identifiers are present on each log record before switching off the Datadog path.
- Document the stability level of each component: Core Collector components have mixed stability, so record which versions you tested.
Confirm your Collector version and its exporter components against the OpenTelemetry Collector release notes before committing to a design.
Organizations with many teams, many pipelines and several Datadog log indexes or sites need a migration that is sequenced by ownership. The main effort is coordination: each team owns sources, parsing, dashboards and alerts, and OpenSearch index templates and lifecycle rules must serve all of them without conflict. Datadog sites are independent, so any cross-site log view must be rebuilt inside OpenSearch on purpose.
- Assign owners per index pattern: Map each team to its indexes, saved views and monitors before cutover.
- Standardize index templates: Use shared mappings for common fields, and allow team-specific fields in separate patterns.
- Move one team as a pilot: Choose a team with clear sources and a sponsor, then reuse its runbook for the next team.
- Size index lifecycle policies per tier: Hot, warm and delete timings usually differ by data class, so set them with each owner.
- Track alert parity per team: Record which monitors have been recreated and signed off by their owners.
Ask the owners of each index to confirm their retention and search needs in writing before any lifecycle policy is applied.
Regulated organizations need field masking, access control and audit evidence before logs move, not after. OpenSearch Security provides role-based access control, document-level and field-level security, field masking, user impersonation, audit logs and TLS for node and client traffic, and the Security plugin supports SAML, OpenID Connect, LDAP and other authentication backends. Those controls must be configured and tested, because the migration itself moves personal or security-relevant data into a new system.
- Classify fields before ingestion: Identify personal data, credentials and payment fields, then decide which are dropped, hashed or masked at the Collector.
- Configure field masking and field-level security: Test that restricted roles cannot see masked values in Discover, dashboards or exports.
- Enable audit logs and TLS from the start: Confirm that audit configuration captures reads of sensitive indexes, not only writes.
- Connect identity early: Integrate SAML, OpenID Connect or LDAP so that access follows your existing joiner and leaver process.
- Keep evidence: Retain configuration, test results and access reviews as records for your assessors, since DeployOpen prepares evidence but auditors issue the opinions.
Ask your compliance lead which log categories must stay in a specific region or retention period before choosing the index lifecycle design.
What drives the cost and duration of the move
The cost and duration of moving Datadog logs to OpenSearch depend on a small set of measurable inputs: how much data you ingest, how many pipelines and dashboards you rebuild, how much archived history you load, and how long you pay for Datadog and OpenSearch at the same time. Each card below names one driver, whether it is paid once or recurs, and what to put in a written scope.
Daily log volume and index sizing
Your ingest volume per day, after exclusion filters, sets the size of OpenSearch indices, shard counts and the number of data nodes. Teams often size from the Datadog billing figure, which reflects ingested bytes and may not match indexed size once fields are mapped. Ask for a sample of indexed size per source over at least one representative week, and for the mapping strategy for high-cardinality fields. This is a recurring driver because storage and compute grow with every day of retained logs.
Parsing pipelines and log sources
Each Datadog Pipeline, grok parser, remapper and processor must be rebuilt as an OpenTelemetry Collector processor or as an OpenSearch ingest pipeline. Count the distinct sources, their parsers and any custom enrichment, because a single shared parser usually costs less than many per-service variants. This work is one-time, but it grows with source diversity and with the number of teams that own a service. Confirm which team signs off on parsed field names before cutover.
Dashboards, saved views and monitors
Saved views, dashboards and log-based monitors must be recreated in OpenSearch Dashboards, and each query language change adds translation time. Inventory every monitor with its owner, threshold and notification route, then mark which ones are still used. Retiring unused monitors is usually cheaper than translating them. Ask the implementer to name which items are rebuilt, which are retired and which need a business owner decision.
Archived history volume
Datadog Log Archives in your own object storage hold compressed JSON, and the total archived size determines how much history you can load into OpenSearch and how long the import runs. Loading history is a one-time cost, but the indices it creates recur as storage under your retention policy. Decide whether history loads into the same cluster or a separate read-only index set, and confirm that the archive format and bucket access are documented before the scope is signed.
OpenSearch hosting and security controls
OpenSearch is licensed under Apache-2.0, so the software carries no license fee, but you pay for the infrastructure, operations time and the Security plugin configuration that enables SAML or OpenID Connect sign-in, role-based access and audit logs. Hosting choice (Kubernetes Operator, Helm, tarball or managed cloud) changes the recurring operations effort. Ask for the topology, node count, backup target, TLS design and who patches the cluster in each maintenance window.
Retention policy and lifecycle rules
Retention decides how much storage recurs every month. Index lifecycle rules that move older indices to cheaper storage or delete them cut costs, but they must match the retention periods your compliance and incident response teams require. Write retention per log class, not one value for all logs, and ask the implementer to show the index lifecycle policy in the scope. Changing retention later means reindexing or accepting gaps in history.
Dual-running Datadog during overlap
Until OpenSearch is verified, logs are usually shipped to both Datadog and OpenSearch, so Datadog ingest and retention charges continue during the overlap. The length of that overlap is the largest controllable cost in the move. Set explicit acceptance criteria for parity, such as matching counts per source and alert parity, and agree a shutdown date for Datadog log ingest in writing. Ask for the dual-run period as a separate line item in the quote.
Questions teams ask before moving Datadog logs to OpenSearch
What drives the cost of moving Datadog logs to OpenSearch?
The cost is mostly infrastructure and people, not licensing, because OpenSearch is released under the Apache License, Version 2.0, and the OpenSearch FAQ does not describe a paid edition. The recurring costs are compute and storage for the cluster, backups, and staff time to operate it. The main one-time costs are pipeline rebuilding, dashboard and monitor translation, and historical data loading. Ask any quote to split these components, state the node sizing and retention assumptions, and show which items are fixed scope and which depend on log volume.
How long does a Datadog to OpenSearch log migration take?
No duration is fixed by the platform itself, so the timeline depends on drivers you can measure. The main ones are how many log sources and agents must be re-pointed, how many pipelines, saved views and monitors must be rebuilt, how much archived JSON history must be loaded, and how long parallel shipping runs before cutover. Teams with a small number of sources and a clear inventory move faster than those with many undocumented pipelines. Ask for a written scope with a timeline tied to these counts before work starts.
Can our internal team migrate Datadog logs to OpenSearch alone?
Many teams can, provided they have engineers who understand their log sources, index design and the OpenSearch security model. The work is largely configuration and testing, but it touches shippers, pipelines, access control and retention at the same time. Check whether you have someone who can operate a cluster after go-live, since OpenSearch versions move through maintenance on a fixed schedule and need regular upgrades. If capacity or experience is short, a fixed-scope engagement from DeployOpen can cover planning, implementation, hardening, documentation and handover.
What does the work need from our team during the migration?
The migration needs named owners and access to several areas:
- An inventory of log sources, agents, Log Archives, pipelines, saved views and monitors that Datadog currently uses.
- Platform access to build the OpenSearch cluster, including the chosen deployment method such as Docker, Helm, the OpenSearch Kubernetes Operator or packages.
- Identity decisions, because the OpenSearch Security plugin supports SAML, OpenID Connect, LDAP and other backends, and roles must map to your existing groups.
- Business owners who confirm which alerts and dashboards matter, so that rebuilt monitors are not copied blindly.
- Time from the SOC or on-call team to test alerts during parallel running.
Do we need Datadog Log Archives to move our logs?
No, Log Archives are not required for new logs, because you can point shippers or OpenTelemetry Collectors at OpenSearch from the cutover date. Archives matter only if you need historical data in OpenSearch, since the archived compressed JSON in your own object storage is the practical source for loading past logs. Confirm with Datadog which archive destinations you have configured and whether retention in those buckets meets your audit needs. If you have no archives enabled, history stays in Datadog for its retention period, and you should decide whether that is acceptable.
How do we validate log parity before cutting over from Datadog?
Run both systems in parallel and compare them against agreed checks before switching any source. Useful checks include matching event counts per source and service over the same time window, confirming that key fields such as timestamps, severity, service and trace identifiers are parsed the same way, and checking that each rebuilt monitor fires on a known test event. Include the security and compliance views, since access rules and field masking must hold in OpenSearch before real data moves. Record the results in a sign-off document so auditors and operators can see what was tested.
How should we size OpenSearch indices and retention for logs?
Size from your measured daily log volume, retention requirements and the index lifecycle policies you will apply, then test the result with real data. Hot, warm and delete stages control how long data stays on fast storage and when it is removed, and each stage has a cost in storage and nodes. Check how many shards each index creates, because oversized shard counts are a common cause of poor performance. Ask for the sizing assumptions in any quote, and confirm the retention periods with your compliance owner, since retention drives storage cost more than almost any other setting.
What the migration needs from your team
The migration starts with a log source inventory, not with OpenSearch. Your team needs a list of every service, host, cloud account and forwarder that sends logs to Datadog today, with the log volume per source where you can get it, the index or pipeline each one uses, and whether it ships through the Datadog Agent, an OpenTelemetry Collector or a direct API call. Ask the platform owner to confirm which sources are actually in use, because Datadog often holds forwarders that nobody has reviewed in years. Each source needs a named owner who signs off that its logs can move and that its fields still mean the same thing in OpenSearch.
Several inputs come from people outside the platform team. Pipeline and dashboard owners must export or describe the Datadog log pipelines, saved views and log-based monitors they depend on, and an administrator with Datadog access must produce the Log Archives configuration and the JSON exports. Your cloud or storage team must provide the object storage bucket that will hold archived history, with confirmed access rules and a retention policy. The security or compliance lead owns the decisions on who may read which indices, how long each log class is kept and whether any log class must stay in a specific region. DeployOpen can prepare the technical options for these decisions, but the choices belong to your organization.
Sign-off should be explicit at each phase: the inventory and retention decisions before any pipeline is changed, the dual-shipping and parity check before Datadog forwarding is turned off, and the final cutover and archive handover before Datadog log indexing is retired. Name one accountable approver per phase in the statement of work, and agree in writing which Datadog access the engagement needs and when it ends. Confirm the exact Datadog export capabilities for your contract tier and the current OpenSearch release and support window with the vendors before scheduling, since both change over time.
How DeployOpen scopes, runs and supports a Datadog to OpenSearch migration
DeployOpen plans and builds the OpenSearch log platform inside your own infrastructure, then hands over pipelines, access controls and runbooks your team can operate. We start from a written scope that names the log sources, Datadog exports and retention rules involved before any change reaches production.
How we scope
- Inventory every log source, forwarder and Datadog pipeline that currently ships logs off your hosts
- Confirm which Log Archive buckets, saved views and monitors must be rebuilt or retired
- Agree in writing the index lifecycle, retention periods and cluster sizing assumptions for OpenSearch
- Engagements are fixed-scope, with deliverables and timelines agreed before work starts
How we build
- Re-point Datadog Agents or OpenTelemetry Collectors to OpenSearch while dual-shipping during cutover
- Rebuild ingest pipelines, field mappings and OpenSearch Dashboards views against your agreed log schema
- Load archived JSON history from your own object storage and test search results against samples
- Configure Security plugin roles, TLS and audit logging to match your access and data handling rules
How we hand over
- Deliver pipeline configuration, index templates and lifecycle policies as version-controlled files
- Document each rebuilt monitor and saved view with its original Datadog definition for traceability
- Run a handover session covering daily operation, upgrades, backup checks and access requests
- Provide test evidence and control mappings your auditors can review, without issuing any attestation
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after cutover
- Track OpenSearch maintenance releases and plan upgrades within the versions your team approves
- Review storage growth and retention against the sizing assumptions recorded at handover
- Ask your procurement team to confirm retainer terms and quote scope in writing before engagement
