Using Kafka replay as a ClickHouse recovery source: when it fits
Suppose your self-managed ClickHouse cluster is fed from Kafka, and a bad deletion or a corrupted part has damaged a table that analysts depend on. Replaying the stream can rebuild that table, but only for the period the stream still holds. Replay can only reach back as far as Kafka retention, so the recovery window is set by the topic's retention settings, not by how much history ClickHouse stores. Before you rely on this path, confirm the retention configured on each source topic, check whether it is time-based or size-based, and confirm that the consumer offsets and the data you need still fall inside it.
Replication does not solve this problem. ClickHouse replicas copy each other faithfully, so a bad DELETE, a faulty migration or a corrupted merge is copied to every replica as well. The ClickHouse backup documentation makes the same point: replication alone does not protect against human error or corruption. Replay is therefore one recovery source, to be used alongside BACKUP and RESTORE, which keeps point-in-time copies independent of the live tables and of the stream. It does not replace them. A stream replay also cannot restore data that was never published to Kafka, or data that was transformed in ways that the original messages do not capture.
Decide early which tables can be rebuilt from replay and which must rely on BACKUP and RESTORE. Replay fits tables whose contents are a deterministic function of retained messages and whose deduplication behavior you have tested. It fits poorly for tables fed by sources outside Kafka or for tables that have been enriched with reference data that has since changed. Ask your team to document, per table, the retention period, the replay target and the verification method, and to run a replay into a spare cluster at least once before you need it in an incident.
Archive destinations and replay options compared
This table compares an object storage archive, an HDFS archive and replay from the live topic across retention length, query and reload effort, access controls and operational load. Read each row as a question to answer with your platform team before choosing, since most of the answers depend on your existing storage and the retention you set on the stream.
| Criteria | Object storage archive | HDFS archive | Replay from live topic |
|---|---|---|---|
| Retention length | Set by the bucket lifecycle and the archive budget, so it can exceed the topic's retention | Set by cluster capacity and replication settings, so it can exceed the topic's retention | Bounded by the topic retention period configured on Kafka, so the recovery window is fixed by that setting |
| Query and reload effort | Raw files must be read and loaded back into ClickHouse, usually through a staged pipeline | Raw files must be read through Hadoop-compatible tooling and loaded into ClickHouse | Consumers re-read messages from a chosen offset, which is the simplest path if the window still covers the damage |
| Access controls | Bucket policies, per-prefix credentials and object-level encryption depend on the provider, so confirm them | Kerberos and HDFS permissions are possible, but integration with your identity provider needs confirming | Kafka ACLs govern who can read the topic, and resources without ACLs are restricted to super users by default |
| Operational load | Low day to day, with lifecycle rules and monitoring of the ingest job as the main tasks | Higher, since the cluster needs its own patching, capacity planning and disk monitoring | Low storage overhead, but a long retention on the topic adds broker disk and capacity planning |
| Failure exposure | Archive job failures can silently create gaps, so track offsets and file completeness | Same gap risk as object storage, plus exposure to HDFS cluster outages | Damage older than the retention window cannot be replayed, so the window must match your recovery needs |
| Confirm with your platform team | Storage class, lifecycle policy, encryption keys and where credentials are held | Whether a supported HDFS cluster already exists and who runs it | Required topic retention, broker disk headroom and whether consumer offsets are preserved |
How Kafka retention sets the recovery window
Replaying Kafka can only rebuild a ClickHouse table as far back as the source data still exists. Five settings decide that depth, and they combine so the shortest one sets the real window. Decide the window from your recovery point objective first, then check each setting against it.
Topic retention period
Retention on the Kafka topic is the most common limit on replay depth. Once the broker deletes a segment, that data cannot be read back from the topic, so a rebuild cannot reach it. Confirm the retention time and size limits for each topic that feeds ClickHouse, and check them against the oldest corruption you need to recover from. Replay from the topic alone only covers that span.
Archive copy retention
The second copy in object storage or HDFS is what extends the window beyond topic retention. Its effective depth is set by the lifecycle rules on the archive, not by the stream. Confirm how long archived records are kept, whether older objects are expired automatically, and who can change those rules. A short archive lifecycle silently shortens the window even when the topic looks generous.
Compaction settings
A compacted topic keeps only the latest record for each key, so it cannot replay the full history of changes. If a ClickHouse table needs every event, a compacted topic breaks the rebuild even with long retention. Check the cleanup policy on each source topic and confirm whether the archive captures every record before compaction runs.
Partition count and ordering
Replay speed and correctness depend on partitions. Rebuilt rows arrive per partition, so ordering is only guaranteed within a partition, and the ClickHouse sorting key must not rely on a cross-partition sequence. Record the partition count when the topic is created, since it is harder to change later without disturbing key-based ordering. Confirm the archive keeps partition and offset metadata.
Offset and checkpoint tracking
A replay needs to know where each ClickHouse table stopped ingesting. Without recorded offsets per partition, operators must guess a start point and risk duplicates or gaps. Store the committed offsets or a loader checkpoint alongside the rebuild job, and verify them against row counts after the replay. Offsets from before a topic was recreated cannot be reused.
Recovery point objective
The window is the shortest of topic retention and archive retention, and it must cover your recovery point objective. Set the objective first, based on how much recent data the business can afford to re-ingest or lose, then compare it with each limit above. If the shortest setting falls short, extend the archive or adjust retention before go-live, not during an incident.
Setting up the archive and a tested replay path
This sequence turns a Kafka topic into a verified, replayable archive and proves that a spare ClickHouse cluster can be rebuilt from it within a measured time. Each step closes with a check, so the recovery path is known to work before an incident depends on it.
Agree the recovery window with data owners
Before any archive is written, the platform team and the data owners agree how far back a rebuild must reach and which tables and topics it covers. The recovery window cannot exceed the retention configured on the Kafka topic unless the archive itself holds older data, so confirm the topic retention setting with the Kafka operators first. Record the agreed window, the affected ClickHouse tables and the topic names in the runbook. The step is done when the data owners have signed off on that written window and someone has confirmed the current topic retention value in the cluster configuration.
Create the archive consumer with its own credentials
Create a dedicated consumer group and a separate Kafka principal for the archive job, so its access can be granted, audited and revoked without touching ClickHouse or application consumers. Grant that principal read access only to the topics in scope, using the authorization model your Kafka version documents, and confirm that resources without ACLs are not left open to unintended principals. Use a consumer group name that no other system uses. The step is done when the principal can read the topic from its committed offsets and is denied access to a topic outside scope.
Write raw messages with offset metadata
Have the archive job write each message exactly as received, with its key, value, headers, timestamp, topic, partition and offset, to object storage or HDFS in a layout that can be listed by partition and time. Keeping the original bytes lets you re-parse them later if the ClickHouse schema changes. Use a file or table format you can read back with the tools you already run, and record the format version in the archive manifest. The step is done when a sample of archived files, read back by a separate process, returns the same offsets and payloads as the source topic.
Verify archive completeness against topic offsets
For each topic partition, compare the highest archived offset with the log end offset reported by the broker over the same period, and count offset gaps in the archive. Any gap must be explained, whether it comes from a consumer restart, compaction, retention expiry or a write failure, and then fixed by re-reading the affected range. Run this check on a schedule and keep the results. The step is done when every partition in scope shows contiguous offsets from the archive start point to the current position, with no unexplained gaps.
Build the rebuild procedure on a spare cluster
Write a procedure that creates target tables on a spare ClickHouse cluster with the same schema and engine definitions, reads the archived files for a chosen time range, and inserts them into those tables through the parsing and transformation logic used in production. Pin the ClickHouse version used for the rebuild to the one in production, and keep the procedure in version control. Include the steps for swapping rebuilt data into place after validation. The step is done when the procedure runs end to end on a non-production cluster and produces tables that can be queried.
Validate the rebuilt data against known values
Compare the rebuilt tables with a reference: row counts per partition and time bucket, checksums over key columns, and a sample of records checked against the archived source files. Compare against production only for periods that have not changed since the archive was taken, and document any expected differences. Keep the query text and results as evidence for later review. The step is done when the comparison passes within the differences you documented beforehand, and a failing comparison has been traced to a cause.
Run and time a full restore test
Run the complete procedure from an empty spare cluster, starting from the archive alone, and record the wall-clock time from the first read to the point where validation passes. Repeat the test on a schedule and after any change to the topic, archive format or ClickHouse version. Compare the measured time with the recovery time the data owners accepted in step one. The step is done when a timed run has been recorded, the measured time is documented alongside the agreed window, and any gap between them has a named owner and action.
Where replay falls short of BACKUP and RESTORE
Replay is only as complete as the archive behind it. A Kafka topic can be replayed only for the offsets that still exist, and the copy you keep in object storage or HDFS is the true recovery boundary. If the archive job lagged, failed silently, or skipped a partition, the rebuilt ClickHouse table will have holes that no amount of replay can fill. Before you rely on this path, confirm the archive's completeness with a per-partition offset comparison against the topic, and check how the archiver handles restarts, schema changes and dropped records. The stream retention setting governs how far back the live topic can serve, but the archive, not the topic, decides what you can actually rebuild.
Rebuilt data is not always identical to the original. Late-arriving events, deduplication, enrichment and any transformation applied before ClickHouse receives the data can all produce a table that differs from the one you lost. A replay that runs today through a newer parser or a changed materialized view may give different values from the first load, and that difference can surface as a failed reconciliation during an audit. Document which transformation version produced each archived segment, and compare row counts and sample checksums against a known good source after every rebuild. Rebuild time is also a planning constraint: replaying a long history through ingestion can take far longer than restoring a backup, so measure throughput on a spare cluster before you commit to a recovery time.
Use replay for what the stream can cover, and BACKUP and RESTORE for the rest. Replay fills the gap between your latest ClickHouse backup and the present, while the backup, taken with the ClickHouse backup and restore documentation as the reference, provides a known consistent point that does not depend on the stream's retention or the archive's completeness. Schedule BACKUP runs at an interval that keeps the replay window short enough to meet your recovery objective, and record the backup's end timestamp so the replay starts at the matching offset. Confirm with your own tests that a restore from backup followed by replay of the remaining offsets reproduces the production table.
Securing the archive and the replay credentials
The replay copy is only as trustworthy as the controls around it. The archive bucket or HDFS path, the Kafka principal that reads the stream, and the retention rules on both must be designed together, because a leaked archive or an over-long copy can create exposure that the recovery plan itself introduced.
Scope the archive bucket or HDFS path
Give the archive writer its own storage identity with write access to the archive prefix only, and give restore jobs read access on a separate identity. Deny deletes to the writer, and enable object versioning or the equivalent so a compromised writer cannot silently erase history. Confirm how your storage platform enforces these rules, because bucket policy, HDFS permissions and object lock behave differently.
Create a separate Kafka principal
Run the archive consumer under its own principal, distinct from application producers and consumers, so its reads appear in broker logs and can be revoked without affecting production. Choose an authentication mechanism from the options Kafka's security overview documents, such as SASL/SCRAM or SASL/GSSAPI over TLS, and record which principal holds which credential.
Grant read-only ACLs in KRaft mode
In KRaft clusters the StandardAuthorizer stores ACLs in the metadata log. Grant the archive principal Read on the replayed topics and the consumer group it uses, and nothing else. Remember that resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true, so confirm that setting is off or deliberately chosen before you rely on it.
Encrypt replay traffic in transit
Archive writes, Kafka reads and the ClickHouse restore path all carry event data across networks. Use SSL/TLS on the Kafka listener the archive consumer uses, and use TLS for the object storage or HDFS connection. Confirm that no listener used by the archive job is left unsecured, since Kafka allows clusters to mix secured and unsecured listeners.
Align retention with erasure duties
An archive copy of an event is still personal data if the event contains personal data. Set archive lifecycle rules, Kafka topic retention and ClickHouse TTL so that a deletion or erasure request reaches every copy within the window your obligations allow. Document the retention period for each copy and ask your compliance lead to confirm it before the archive starts.
Store and rotate credentials securely
Keep the archive and Kafka credentials in your secrets manager, not in job definitions or shell history. Rotate them on a fixed schedule and after any staff change, and test that the restore job still authenticates after each rotation. A replay that fails at authentication during an incident costs time the recovery window may not have.
Questions teams ask before replaying Kafka for ClickHouse recovery
What does a Kafka replay recovery setup cost, and what drives it?
Most of the cost is labor and storage, and the largest driver is how much stream history you must keep. The Kafka project is open source under the Apache License 2.0, so there is no license fee for the stream itself. You pay for the infrastructure that holds a second copy of the ingestion stream in object storage or HDFS, for the ClickHouse capacity needed to replay into a spare cluster, and for the engineering time to build and test the pipeline. Ask for a quote that separates one-time build work from recurring costs such as storage growth, Kafka retention tuning, and scheduled replay tests. Volume, event size, retention length, and the number of tables affected all move the total up or down.
How long does setup take, and what changes that timeline?
Duration depends on the number of topics, the schema complexity, and how much of the replay path already exists. Teams with a working Kafka cluster and a tested ClickHouse Kafka table usually need less time than teams building both from the start. The timeline also grows if you must add a second storage target, configure access for the replay job, or prove that replayed tables match production. Agree on a written scope and milestones before work starts, and ask for the phases to be listed separately so you can see where time goes.
What does our team need to provide before the work starts?
You need to provide access to the Kafka cluster and its topic configuration, the ClickHouse cluster layout, and the storage target where the second copy will live. Your team should also supply the topic retention settings, the schema definitions, and the list of tables that must be recoverable. Named owners for Kafka, ClickHouse, and storage, plus an approved change window for the spare cluster, prevent delays. Confirm which Kafka version and security settings are in use, since authentication and authorization affect how the replay consumer is granted read access.
Can our own team build and run this without a partner?
Yes, if your team already runs Kafka and ClickHouse in production and has time to build, test, and maintain a replay path. The work requires skills in Kafka consumer configuration, ClickHouse table engines, object storage, and access control. Teams without those skills often underestimate the testing effort, which is where most recovery gaps appear. A partner is useful when you need an outside build, a fixed-scope delivery, or an independent set of replay tests. Ask any partner to state in writing what they build, what they hand over, and what your team must operate afterward.
How often should we run replay tests?
Run a replay test on a schedule that matches how quickly your data must return after corruption, and repeat it after every significant change to Kafka retention, schema, or ClickHouse version. The ClickHouse backup documentation warns that a backup never test-restored will probably not restore properly when needed, and the same reasoning applies to replay. Test replays on a spare cluster, compare row counts and checksums with production, and record the results. Frequency is a decision for your recovery objectives and change rate, so set it in your recovery plan rather than treating it as a fixed rule.
How does Kafka retention set the recovery window?
Replay can only rebuild data that still exists in the stream copy, so the stream retention period sets the earliest point you can recover to. If Kafka keeps seven days of events in its topics but your object storage copy holds ninety days, the second copy defines the window. Check both the Kafka topic retention and the archive retention, and confirm that offsets, partitioning, and event ordering are preserved in the copy. Gaps caused by consumer lag, failed archive writes, or retention expiring before a corruption is detected will shrink the window, so monitor them.
Can Kafka replay replace BACKUP and RESTORE for ClickHouse?
No. Kafka replay rebuilds tables from events, which works for corruption and some deletion cases but not for everything a backup covers. ClickHouse documentation states that replication alone does not protect against human error or corruption, and BACKUP and RESTORE captures table state, including data that never passed through the stream, such as reference tables or corrected rows. Replay also depends on the stream having kept the events, and it may need extra steps to restore users, roles, or table definitions. Most teams combine the two methods and document which one covers each table.
How DeployOpen scopes and delivers Kafka replay recovery
Scope is agreed in writing before any work starts. For a Kafka replay recovery source, the written scope should name the topics that feed each ClickHouse table, the archive destination (object storage or HDFS), the retention settings on the Kafka side, the archive format and partitioning, the rebuild procedure and the owner of each step. DeployOpen plans this work against your existing Kafka cluster and ClickHouse tables, so the scope also records which Kafka version and access model are in place. Confirm with your Kafka operators the current retention.ms and retention.bytes values for each topic, because those values set the recovery window that the scope must state plainly. Deliverables, timelines and acceptance criteria are fixed in the same document, and a quote follows from that scope rather than being estimated in advance.
The archive and rebuild procedure is built inside your infrastructure. DeployOpen implements the consumer that copies messages from the topic into the archive in your cloud account, data center or restricted network, with the credentials, encryption and access roles you specify. The rebuild path is written as a runbook: how to select the offset or time range, how to replay into a staging table, how to validate row counts and checksums against the source, and how to swap the table into production. Kafka access is controlled through its SASL and ACL model, so the consumer identity receives only the read permissions it needs. Decisions such as archive format, partitioning and whether to replay into a new table or a spare cluster are recorded in the runbook so that your team can run the procedure without the original builders.
Restore tests and retainers keep the path working over time. An archive that has never been replayed into a spare environment is an assumption, not a recovery plan. DeployOpen runs restore tests against the spare environment on a schedule set in the scope, records the results as evidence, and documents any gaps found, such as a topic whose retention expired before the archive started. Retainers can cover recurring restore tests, monitoring of the archive consumer, and Kafka and ClickHouse upgrades that may change offsets, message formats or replay behavior. DeployOpen prepares these controls and records for your own assessments, but it does not issue audit reports or certifications.
