Migrating from Elasticsearch to ClickStack for logs and traces

Migrating from Elasticsearch to ClickStack: what the move involves and how long it takes

The short answer is that the effort depends on what you ingest, how many collectors you run, and how many dashboards and alerts your teams rely on, not on how much data sits in Elasticsearch today. Moving an Elastic Stack used for logs and traces to self-hosted ClickStack on ClickHouse is mostly data modeling, collector change and parallel validation. It is rarely a single bulk copy of indexes. Expect the scope to be set by four inputs: daily ingest volume, the number and type of collectors (Beats, Elastic Agent, Logstash pipelines, or application SDKs), the count of Kibana dashboards, saved searches and alert rules, and the retention periods your teams and auditors expect. Retention matters in both directions, because keeping years of history means deciding whether to migrate it or let it age out in Elasticsearch.

Ask the partner or your own team to inventory each of those four inputs before any quote is written. For the data model, the ClickHouse documentation includes an Elastic to ClickStack type mapping that you should check field by field against your real index templates, since dynamic mappings and nested objects are where surprises appear. For collectors, each Beat or Logstash pipeline has to be replaced or re-pointed at OpenTelemetry Collector pipelines, and that work recurs whenever a new service is added. Confirm with the vendor and in your own environment which ClickStack release and deployment model you will run, and which ingestion path each collector will use. Scoped quotes should state whether parallel running, dashboard rebuilding and alert rule translation are included, and who owns each of them.

Plan for a parallel period in which both stacks receive the same telemetry, so that query results, dashboards and alerts can be compared before anyone relies on ClickStack. Parallel running is a recurring cost in infrastructure and licensing time for as long as it lasts, so agree its exit criteria in writing. Elasticsearch retirement is a separate decision: confirm your Elastic subscription terms and the end-of-maintenance dates for your current release line on Elastic's support lifecycle page before setting a date, because unsupported clusters may stop receiving security fixes while data is still being moved.

Which Elasticsearch capabilities map cleanly to ClickStack and which do not

This table compares the Elasticsearch capabilities that observability estates usually depend on with their ClickStack or ClickHouse equivalents. Each row is marked clean, partial or needs redesign, so you can see where migration effort will concentrate. The main takeaway is that schemas, ingest pipelines and alerting must be rebuilt, so inventory what your team actually uses before planning.

Elasticsearch capability	ClickStack or ClickHouse equivalent	Fit
Field types (keyword, date, long, text, object)	Documented type mapping to ClickHouse types such as String and DateTime64	Partial
Index templates and mappings	Table definitions with column types and an ordering key, kept in version control	Needs redesign
Data streams and ILM hot, warm and cold tiers	Partitions with TTL rules and storage policies across disk or object storage tiers	Partial
Ingest pipelines	OpenTelemetry Collector processors or ClickHouse materialized views	Needs redesign
Kibana dashboards	Rebuilt in ClickStack or Grafana; confirm any export path with the vendor	Needs redesign
Kibana alerting rules	Rebuilt as queries in the alerting tool you choose	Needs redesign
Anomaly detection alerts	No direct equivalent; separate detection logic must be designed and tested. Paid only on Elastic Platinum and Enterprise tiers, so confirm use	Needs redesign
Field-level and document-level security	Row policies and column grants per user or role; confirm coverage against your access requirements	Partial
ECK-managed clusters	Self-run ClickHouse with your own Kubernetes operator and ClickHouse Keeper	Needs redesign

Documented migration paths and the tools each one needs

ClickHouse documents an Elastic to ClickStack migration guide, and its type mapping page explains how Elasticsearch field types translate into ClickHouse column types. Start there rather than designing a mapping from scratch, because the documented mapping is the reference your data-model review should check against. Plan for the source side too: Elasticsearch node roles determine which nodes hold the data you need to read, and an Elastic Cloud on Kubernetes deployment changes where configuration and credentials live.

Type mapping review: Compare every index template field against the documented Elastic to ClickStack type mapping and record each exception.
Source topology: Identify data, hot, warm, cold and frozen tier nodes, since reads on frozen or searchable snapshot data may behave differently from hot indexes.
Deployment model: Note whether Elasticsearch runs on virtual machines or through ECK, because the operator owns some configuration and secret paths.
Version check: Confirm the ClickStack and ClickHouse versions in your environment against the current documentation before you copy any command.
Scope: Decide which indexes move, which are rebuilt from source, and which are retired.

Before you start, confirm the exact versions of ClickStack, ClickHouse and Elasticsearch you run, and check the migration guide for the version it covers.

Importers, APIs and collector paths available for this move

Five documented pieces cover the move from Elasticsearch to ClickStack: a type mapping guide, the Collector path for new telemetry, the gateway pattern for many sources, authentication on the wire, and the Elasticsearch APIs for reading history. Each one needs a check against the current docs and your own environment before a cutover plan depends on it.

Elastic to ClickStack type mapping

ClickHouse publishes a guide that maps Elasticsearch field types to ClickHouse column types for this migration. Use it as the starting point for your index templates, not as a finished schema. Check how your dynamic mappings, nested objects, keyword fields used for aggregations and timestamp fields translate, then test each against a sample of real log and trace documents. Confirm every mapping on the page against your own field list before creating tables.

Collector receivers and exporters

OpenTelemetry Collector receivers accept telemetry from applications and agents, and exporters send it on to a backend. Map each Beats, Elastic Agent or Logstash output you run today to an equivalent receiver and pipeline. Confirm that the ClickHouse exporter is present in the Collector distribution you plan to deploy and check its current configuration keys and stability level in the component README, since components vary in maturity.

Gateway pattern for many sources

A gateway Collector receives data from many agents and services, then applies processing and exports to ClickHouse. This centralizes batching, retries and credentials, but it also becomes a single point of loss if sized poorly. Decide early whether agents run next to workloads, whether a gateway tier is needed, and how many gateway instances you need for peak ingest. Load test the gateway with production-shaped log volume before moving any collector.

OIDC and oauth2client authenticators

Authenticator extensions can check identity on incoming Collector connections, for example with OIDC, and attach credentials to outgoing exporter requests, for example through the oauth2client extension. Use these to replace shared Elasticsearch API keys and basic credentials in your collector configs. Verify which receivers and exporters support each authenticator in your Collector version, and test token expiry and renewal, because a silent auth failure will drop telemetry.

Elasticsearch APIs for historical reads

Historical logs and traces often need to be read back during the parallel run and after retirement. Elasticsearch search and scroll-style APIs, along with snapshot and reindex tooling, are the routes for exporting that data. Inventory which dashboards, alerts and investigations depend on old indexes, and decide which history is copied into ClickHouse and which stays in a read-only Elasticsearch cluster until its retention ends. Confirm the export method and throughput with a sample index before committing to a copy window.

The sequence: inventory, pilot, parallel run, cutover and index retirement

The sequence moves observability data from Elastic to ClickStack in stages, so each source group is proven on ClickHouse before anything on the Elasticsearch side is switched off. It produces a tested, retained copy of the data and a signed-off decommission record.

Inventory indexes, pipelines, dashboards and alerts
The platform team, with the owners of each dashboard and alert, lists every Elasticsearch index pattern, ingest pipeline, Beats or Elastic Agent policy, Logstash pipeline, Kibana dashboard and alerting rule. Each item is tagged with its source group, owner, retention requirement and whether it feeds a compliance or on-call workflow. The inventory is done when every active index has a named owner and a retention period confirmed by the business, and anything unowned is flagged for a decision before work continues.
Map fields and types to ClickHouse
Engineers use ClickHouse's Elastic to ClickStack type mapping guide to translate each field in the inventory into a ClickHouse column type, and decide where dynamic or nested Elasticsearch fields need a fixed schema. Ingest pipeline processors that enrich or rename fields are rewritten as OpenTelemetry Collector processors. The step closes when a sample of real events from each source group has been loaded into a scratch ClickHouse table and every queried field returns the expected values.
Pilot one service on OpenTelemetry collectors
Choose one low-risk service and deploy OpenTelemetry Collector agents beside it, with a gateway that exports logs and traces to ClickStack. Collector configuration should follow the vendor-neutral security practices for receivers, exporters and authentication, and the collector build should come from a release you have verified against your own checksums and SBOM. The pilot is complete when the service's logs and traces are queryable in ClickStack, dashboards built for that service show the same values as Kibana, and the team has tested a rollback to the existing agent.
Run both stacks in parallel and compare results
Fan out the same telemetry to Elasticsearch and ClickStack for each source group, keeping the existing Elastic agents running until the comparison is accepted. Compare event counts, error rates, latency percentiles, alert firings and sampled query results over a window that covers normal and peak load, and record every difference with its cause. Ingestion sizing and query performance are checked here, because ClickHouse storage and retention settings will be tuned from these results. The step closes with a written comparison report that the service owners have accepted.
Cut over each source group in turn
Move one source group at a time from Elastic agents to OpenTelemetry collectors writing only to ClickStack, starting with the groups that carry the lowest risk. Dashboards and alerts are repointed to ClickStack after their owners confirm the comparison, and the Elasticsearch ingest path for that group is stopped while the old indexes stay readable. The cutover for a group is done when its alerts fire from ClickStack, nothing new is written to its Elasticsearch indexes, and the on-call team has used the new dashboards during a real incident or drill.
Set up ClickHouse replication and backups before retirement
Before any Elasticsearch index is deleted, the ClickHouse side must have replication and backup in place. Confirm that ReplicatedMergeTree tables are coordinated by a ClickHouse Keeper ensemble on hosts sized for production, and that a recovered replica resyncs from a healthy one. Configure BACKUP to a separate storage target on a schedule that matches your retention policy, and record the backup location and encryption settings. This step is complete when a replica has been failed and recovered in a test environment and the backup chain is documented.
Retire Elasticsearch indexes after restore and retention checks
Delete Elasticsearch indexes only after three checks pass for each source group: a full restore from ClickHouse BACKUP onto a spare cluster returns the expected row counts and sampled queries, retention rules in ClickHouse match the period the business approved, and any data that must be kept for legal or audit reasons has been exported and verified. Snapshots kept in Elasticsearch are reviewed before deletion, and Kibana, Elastic Agent and Logstash are switched off once no source group depends on them. The sequence ends with a signed decommission record for each index set.

Validating ClickHouse results against Elasticsearch during the parallel run

Validation during the parallel run answers one question: would the team make the same decisions from ClickHouse that it makes from Elasticsearch? Start by comparing event counts per source, per index pattern and per hour over the same window in both systems, then move to field-level checks on service.name, level, trace.id and the status fields your alerts depend on. Compare timestamp handling explicitly. Confirm which field each system uses as the event time, how the collector sets it when the source omits it, and whether timezone conversion happens at ingest or at query time. A one-hour offset in a single collector will look like missing data in a dashboard and will never show up in a raw count. Rebuild the top twenty Kibana dashboard panels and alert queries in ClickStack, then record the result of each pair side by side, including the rows that differ and the reason.

Late-arriving and duplicate events need their own rules. Elasticsearch accepts an event into the index it names, so a late record usually lands in the correct time bucket, while a retried OpenTelemetry export can create a second copy. Decide how each system counts both cases before you read any mismatch as a defect. Where your table design relies on deduplication with ReplacingMergeTree, confirm that duplicates merge only after background merges run, so validation queries should use FINAL or deduplicate explicitly in the comparison. Record the lateness window you accept for the alerting path, and state it in the cutover criteria.

Validation is not complete until a restore has been tested. The ClickHouse replication guide says a recovered replica automatically syncs missing data from a healthy replica, so you can run a node-failure drill in the parallel environment and confirm that counts return to the baseline without manual steps. Replication does not protect against accidental deletes or corruption, though. The backup documentation warns that a backup never test-restored will probably not restore properly when needed, so run a RESTORE of the log and trace tables into a spare cluster, then repeat the count and dashboard comparison against it. Confirm the restore time you actually achieved with your own data volume, because no figure for it has been established for your environment.

What drives effort and cost in an Elasticsearch to ClickStack move

This table lists the factors that change the size of an Elasticsearch to ClickStack move, the direction each one pushes effort or cost, and how to reduce it. Read it with your own inventory open: the counts you collect are what a written scope and quote depend on.

Driver	Why it changes effort or cost	How to keep it down
Ingest volume and retention	Higher daily volume and longer retention increase storage, Keeper and replica sizing, backup size and the time needed for parallel running and validation.	Confirm which indexes need the full retention window and set TTL and tiering rules before the build.
Index templates and ILM policies	Each template, component template and lifecycle policy must be mapped to a table design, retention rule or tier, so more templates mean more design decisions and tests.	Consolidate templates that share mappings and retire unused policies before mapping starts.
Kibana dashboards	Every dashboard that queries Elasticsearch must be rebuilt against ClickHouse data and checked for matching results, so the count drives rebuild and review time.	Inventory dashboards by owner and usage, and rebuild only those still in active use.
Alert rules	Each rule needs its query translated, its thresholds re-tested and its notification path confirmed, and alerts that run in parallel need agreed ownership during cutover.	Remove stale rules and group related alerts so each one has a named owner and test case.
Agents and Beats to convert	Every Beat, Elastic Agent policy or Logstash pipeline that ships data must be replaced with an OpenTelemetry Collector configuration and redeployed, and each host type is a separate test.	Group hosts by collector configuration and convert one host class at a time.
Custom ingest pipelines	Ingest processors with grok, enrichment or scripts must be rewritten as Collector processors or ClickHouse-side transforms, which is usually the least predictable part of the work.	Document each pipeline's input, output and business purpose before estimating, and drop fields no one queries.
Security model	SSO, role mappings, field-level security and index-level access must be re-expressed as ClickHouse users, roles and row policies, and the mapping of who sees which data must be re-approved.	Start from the access matrix your security team already holds and confirm which controls are required in the target.
Teams and environments	Each team with its own indexes, spaces or alert ownership adds coordination, and each environment (development, staging, production) needs its own parallel run and sign-off.	Agree a single migration order and owner per team, and limit the number of environments that run in parallel.
Keeper and replica layout	A replicated layout needs at least three ClickHouse Keeper nodes in production, and each added shard or replica adds provisioning, monitoring and recovery testing that recurs after go-live.	Confirm the replica count required for your availability target and size Keeper on dedicated hosts from the start.
One-time versus recurring cost	One-time work includes type mapping, pipeline and dashboard rebuilds, collector conversion and validation. Recurring cost includes running ClickHouse and Keeper, backups, restore tests, upgrades and Elasticsearch licensing until it is retired.	Ask each quote to split one-time and recurring items, and to state which Elasticsearch subscription costs end at cutover.

What usually goes wrong and how to prevent it

Most failed Elastic-to-ClickStack migrations break in predictable places: data types, clocks, duplicate ingestion, quiet dashboard failures, premature deletion and untested recovery. Each card below names one risk, the step that prevents it, and what to check before cutover.

Type mismatches in dynamic fields

Elasticsearch maps dynamic and nested fields automatically, so the same field name can arrive with different types across indexes. A ClickHouse column defined too narrowly will reject or coerce values during the parallel run. Compare the field types in your live indexes against the documented Elastic to ClickStack type mapping before you create any table, and sample nested objects from each log source, not just the template.

Timestamp and timezone drift

Log and trace timestamps that are parsed in a local timezone on one side and in UTC on the other shift every event by the offset, which breaks time-range searches and correlation between traces and logs. Decide one storage timezone up front, then run the same sample events through both pipelines and compare the stored values field by field. Check that OpenTelemetry timestamp fields carry their timezone information from each source.

Duplicate data during parallel runs

Running both stacks means a collector that fans out to Elasticsearch and ClickStack can double-count events if a second exporter is added without removing the first, or if a retry resends a batch. Document one owner for each pipeline, tag each exporter route in the collector configuration, and compare event counts per source over the same window before trusting either side for alerting.

Dashboards and alerts that stop silently

Dashboards, saved searches and alert rules written against Elasticsearch field names often return empty results rather than errors once the field is renamed or moved to a different column. Inventory every dashboard and alert, map each query to its ClickHouse equivalent, and assign a test event that must appear in each panel. Treat a panel that shows no data as a failed check, not an empty day.

Retention gaps from early index deletion

Deleting Elasticsearch indexes once the new stack starts receiving data removes the only copy of history before the new retention rules are proven. Keep Elasticsearch indexes readable until ClickStack has been queried successfully for the full period your investigations and audits need. Write down the retention window per log source, confirm the ClickHouse TTL settings enforce it, and retire indexes in stages only after that check passes.

Untested backups of the new cluster

Replication keeps a second copy of data but does not protect against deletion, bad writes or corruption, so a ClickHouse backup is still required for the observability store. The ClickHouse backup documentation warns that a backup never test-restored will probably not restore properly when needed. Schedule restore tests to a spare cluster, record how long each restore took, and confirm the restored tables answer the same queries as production.

Questions teams ask before moving from Elastic Stack to ClickStack

What does an Elasticsearch to ClickStack migration cost, and what drives it?

The cost is mostly the labor to plan and build the new pipeline, move collectors, rebuild dashboards and alerts, and validate results, plus the infrastructure to run ClickStack and ClickHouse alongside Elastic during the parallel run. Infrastructure is recurring, while design, build and cutover work is one-time. The main drivers are the number of log and trace sources, how many Kibana dashboards and alert rules must be rebuilt, custom ingest pipelines, retention requirements, and the replication and backup design you choose. Ask any partner for a written scope that lists each of these items so the quote can be compared line by line. DeployOpen quotes follow from that written scope.

How long does the migration take, and why does it vary so much?

Duration depends on scope rather than on a fixed rule. A small set of services with a handful of dashboards is far simpler than a fleet of hundreds of hosts, multiple Elasticsearch clusters, and many ingest pipelines feeding compliance-relevant indexes. The steps that most often extend the schedule are inventorying what is actually indexed, agreeing on how Elasticsearch field types map to ClickHouse columns, moving collectors to OpenTelemetry, and waiting for a window in which both pipelines can be compared. Ask for a timeline that is tied to named deliverables and to the dates on which your own teams must review results. DeployOpen agrees timelines in writing before work starts.

What must our team provide before the migration starts?

Expect to provide four things. First, an inventory of indexes, data streams, ingest pipelines, Beats and Elastic Agent policies, and alert rules. Second, access to the Elastic cluster, the collector hosts, and the target infrastructure, under accounts your security team approves. Third, named owners for each dashboard and alert, because someone must confirm that a rebuilt view answers the same questions. Fourth, a test window in which logs and traces flow to both systems so results can be compared. Without these, the team cannot judge whether the new system is complete, and cutover decisions get delayed.

Can our own engineers do this, and what needs outside help?

Experienced platform or SRE teams can run much of the work themselves, especially the collector changes and the ClickHouse deployment, if they have time and clear ownership. Outside help is most useful where the work is unfamiliar or high risk: sizing and deploying a ClickHouse cluster with ClickHouse Keeper, designing the replication and backup plan, mapping field types, and validating that alerts and retention still meet internal policy. A common split is to keep dashboard rebuilding and daily operations in-house while a partner handles architecture and the cutover checks. Decide this split before the scope is written so the quote reflects who does what.

Is OpenSearch an alternative to ClickStack, and how does it differ?

OpenSearch is a real option to evaluate, and it is a different path. OpenSearch is a fork of open source Elasticsearch 7.10, with OpenSearch Dashboards derived from Kibana 7.10.2, and it is wire-compatible with Elasticsearch 7.10. That makes it a closer fit if you want to keep the Elasticsearch query model and most existing clients. ClickStack moves storage to ClickHouse and uses OpenTelemetry for collection, which means a change in query language, data model and collectors. Choose based on your query needs, your team's skills, and the license terms you need. Confirm the current OpenSearch release and feature details directly with the project before deciding.

How do we keep Elastic running during the parallel run?

Keep the existing pipeline in place and fan out the same log and trace data to ClickStack, so both systems receive identical input. The OpenTelemetry Collector can run as an agent next to services or as a gateway that accepts data from many sources, and it can export to more than one destination, which is useful during this period. Compare volumes, field values, counts and alert behavior on the same time ranges, and document every difference. Confirm that Elastic remains supported for the duration of your run, since the Elastic end-of-life page lists dates for the 8.x and 9.x lines. Plan credential, network and cost impacts of double ingestion before the test window opens.

What happens to Elasticsearch data after cutover?

Retire Elasticsearch indexes in stages, not all at once. Once ClickStack is trusted for a given source, stop writing to its Elasticsearch index, keep the old data readable for the period your retention and compliance rules require, and then delete it under a documented procedure. If you need historical logs in the new system, decide whether to re-ingest them, which depends on what the source still holds and what your policies allow. Before deleting anything, confirm that backups meet your retention obligations and that any legal hold is respected. Record the retirement decision for each index so auditors can trace what was removed and when.

Primary sources for the migration

Before you commit a migration plan, read the ClickHouse Elastic to ClickStack type mapping first, because every field decision in your index templates depends on it. The mapping guide is the reference for how Elasticsearch field types such as keyword, text, date and nested correspond to ClickHouse column types. Use it as the checklist for your mapping review: export your current index templates, list every field type in use, and mark each one with its target type and a note on where the match is imperfect. Fields that rely on Elasticsearch dynamic mapping are the ones most likely to need a deliberate decision, so review those first.

The Elastic licensing FAQ explains which Elasticsearch and Kibana code you are running and under what terms. Elastic states that its default distribution has been under the Elastic License 2.0 since 2021 and that AGPLv3 was added as an OSI-approved option, and it notes that ELv2 and SSPL are not OSI-approved. Read it with your legal and procurement teams before you finalize the business case, because the license of the platform you are leaving shapes the reasons you give for leaving it. Confirm the current terms directly with Elastic for your deployment, since licensing text changes.

Check the Elastic end-of-life dates page against the version you run today and the date you expect cutover to finish. Elastic lists end of maintenance for the 8.x line on January 15, 2027, with 8.x support ending July 15, 2027, and for the 9.x core stack on October 15, 2027. If your cluster is on 8.x, the parallel-run period must end before the dates that apply to your version, or you will be running an unsupported Elasticsearch while you migrate. Ask your team to confirm which dates apply to every component, including Kibana and any Elastic Agent or ECK deployment. Finally, read the OpenTelemetry Collector configuration docs before rewriting collector pipelines, since receivers, processors and exporters are defined there and each component's stability is documented in its own README.

How DeployOpen works on this

DeployOpen plans, implements, integrates, documents and supports self-hosted open-source platforms inside your own infrastructure. For this move, the work starts with a written scope that maps your Elastic Stack to self-hosted ClickStack on ClickHouse, and it ends with a handover your team can run without us.

How we scope

The written scope lists the log and trace sources, collectors and Elasticsearch indexes that move in the engagement.
Type mapping from Elasticsearch fields to ClickHouse types is agreed up front, with the fields you rely on named explicitly.
Engagements are fixed-scope, so deliverables, timelines and the parallel-run period are agreed in writing before work starts.

How we build

We deploy ClickStack and ClickHouse in your cloud accounts, data centers or restricted networks, with replication designed in from the start.
We move collectors to OpenTelemetry Collector pipelines and run them beside existing Elastic agents during the parallel period.
We test ingestion, dashboards, alert logic and query results against the Elastic baseline before any source is switched over.

How we hand over

You receive the deployed configuration, collector pipeline definitions, type mapping decisions and the index retirement plan.
Runbooks cover restore tests, retention settings and the steps for retiring each Elasticsearch index after its agreed hold period.
Your team is walked through operation so the handover is complete, and the written scope records what was delivered.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after handover.
A retainer can cover ClickStack and collector version upgrades, which you should check against the changelog before each one.
Ask us to confirm which support terms, response expectations and included services apply before you sign the scope.