How the ClickHouse Flink sink connector fits a streaming pipeline
If your team runs Flink jobs that must land high-volume events in self-hosted ClickHouse, the supported method is the official Apache Flink sink connector for ClickHouse. It is built on Flink's AsyncSinkBase and the ClickHouse Java client, and it targets the DataStream API. Table API support is planned but not available, so any job written in Flink SQL or the Table API will need a different design or a DataStream rewrite before the connector can be used. The connector runs inside your own Flink cluster and writes to your own ClickHouse servers, so both sides of the pipeline stay in infrastructure you control.
Start the design by pinning versions on both sides. The connector requires Flink 1.17 or later with Java 11 or later, and Flink 2.0 requires Java 17 or later. Confirm the artifact that matches your Flink line in the ClickHouse Flink connector documentation before you build, and check that your build and runtime images use the Java version the chosen artifact needs. A mismatch here usually surfaces at job submission or at first class loading, which is cheaper to find in a pre-production cluster than in production.
Plan the sink as one stage in a longer chain. Flink's checkpointing decides how much work is replayed after a failure, while the sink's batching decides how many rows each insert carries. ClickHouse performs best with fewer, larger inserts, so batch size and flush interval should be tuned against your event rate and your target table's merge behavior, using the settings the connector exposes. Ask the connector's maintainers or your own tests to confirm how delivery guarantees interact with checkpoints for your job, since replays can produce duplicates unless the target table and keys are designed to absorb them.
Versions, runtimes and accounts needed before the first job
The Flink connector depends on three things lining up: the Flink release, the Java runtime on every task manager, and a ClickHouse server line that is still supported. Confirm these together before writing any job code, because a mismatch in any one of them shows up as a failed build or a stalled sink rather than a clear error.
Choose the Flink release line
The connector supports Flink 1.17 or later, and Flink 2.0 has its own artifacts. Pick the Flink line your platform team already operates, then pull the connector build published for that exact line. Mixing a 1.x connector into a 2.0 cluster, or the reverse, is a common cause of dependency conflicts at submission time. Check the connector docs for the artifact names that match your chosen line and confirm them with your build owner before the first job is packaged.
Java 11 for Flink 1.17
Flink 1.17 jobs need Java 11 or later on the client that builds the job and on every task manager that runs the sink. Check the runtime on the workers, not only on the build server, because a container image or a legacy host can carry an older JDK. Ask the platform team to confirm the JVM version that the task manager images actually start with, and record it in the deployment runbook.
Java 17 for Flink 2.0
Flink 2.0 requires Java 17 or later for the connector. Moving to Flink 2.0 is therefore a runtime change as well as a library change, and it affects every image, startup script and build plugin in the pipeline. Plan the JDK upgrade on the task managers, the job manager and the CI runners together, and test one full job submission on a non-production cluster before switching the production line.
Supported ClickHouse server line
Before choosing a ClickHouse line, compare it with the versions listed as supported for security updates in the ClickHouse security policy. Versions outside that list receive no security updates, so a cluster on an unsupported line should be scheduled for upgrade before a new ingestion path is added. Confirm the chosen server line against the current list and note which release is the long-term support line for your planned upgrade window.
Network reachability to the cluster
The Flink task managers must reach the ClickHouse HTTP interface on every node or load balancer address the sink will use. Check routes, firewall rules, DNS names and any TLS termination in front of ClickHouse from the subnet where Flink runs, not from a workstation. If the sink writes to several replicas, confirm that each replica address is reachable, since a single blocked host can make the job retry and fall behind.
A dedicated ingestion user
Create a separate ClickHouse user for the Flink job with insert rights only on the target database and tables. Do not reuse the default user, which starts with all permissions, or a human analyst account. Record who owns the credential, where it is stored and how it rotates. Ask your security team to confirm the authentication method, since ClickHouse supports several external authenticators in addition to local users.
Configuring the sink from dependency to first committed batch
This sequence takes a Flink job from a matching connector artifact to a bounded test run that proves rows arrive in ClickHouse once, with batching and metrics confirmed before any high-volume stream is connected.
Match the artifact to your Flink version
Confirm the Flink version your cluster runs before choosing a connector artifact. The official connector requires Flink 1.17 or later, with Java 11 or later for Flink 1.17 and Java 17 or later for Flink 2.0, so check the JDK on your build agents and task managers against that line. Pick the artifact built for your Flink major version and record the choice in your build file. The step is done when a dependency resolution on a clean build succeeds with the chosen version and the JDK matches the requirement.
Add the connector and ClickHouse Java client
Add the connector and the ClickHouse Java client to the job's dependencies, and keep the client version compatible with the connector release you selected. Mark Flink core dependencies as provided so the cluster's own runtime is used rather than a second copy packaged into the job jar. Build a fat jar or use your standard packaging, then confirm the artifact contains the sink classes. The step is done when the job compiles, the packaged jar is inspected, and the dependency tree shows one version of each library.
Create the target MergeTree table
Create the destination table in a MergeTree family engine on the ClickHouse cluster before the job runs, with column types that match the record fields your Flink job emits. Choose the ordering key and partitioning for your query pattern, and use a dedicated database user limited to inserts and the needed reads. For a replicated cluster, plan the replicated table definition with your ClickHouse team. The step is done when a manual test insert from the same user succeeds and the table schema matches the record class field by field.
Configure the sink batching settings
Set the sink's batching parameters, including the maximum batch size and the flush interval, in the job code or configuration, starting from conservative values for a first run. Batch size trades latency against insert efficiency, so note the expected event rate and the number of parallel sink subtasks, then test the values rather than copying them from another system. Keep the connection endpoint, credentials and TLS settings in your secrets store, not in source. The step is done when the job starts with the settings logged or visible in the job configuration and the target endpoint resolves from the Flink cluster.
Run a bounded test job against a known source
Run the job against a bounded, finite source such as a file or a fixed slice of a topic, so the input size is known in advance. Let the job finish, then compare the row count in the ClickHouse table with the number of records the source produced, and sample rows to confirm field values and types. If counts differ, check for dropped records in the job logs and failed inserts before changing the batching settings. The step is done when the destination count equals the source count for the test window.
Verify no duplicates after a restart
Stop the job during a bounded run, restart it from its last checkpoint or savepoint, and let it finish. Check the destination for duplicate rows by comparing row counts and a stable key with the source. Duplicates depend on how the sink and checkpointing are configured, so confirm the delivery guarantee you get with your Flink and ClickHouse versions in the connector documentation and write it down before you rely on it. The step is done when the restart test produces the expected count, or the duplicates are measured and the team has accepted a documented compensation.
Confirm sink metrics in Flink
Open the Flink web interface or your metrics reporter and confirm the sink operator shows records sent, throughput and any backpressure or failure counters for the test job. Watch the metrics while a batch flushes to confirm the flush interval behaves as configured. Record a baseline of these values so later changes can be compared. The step is done when the sink metrics are visible for every subtask and the baseline is saved with the job configuration.
Tuning batching for high-volume Flink streams
Batching is the setting that decides whether a Flink job writes ClickHouse efficiently or creates a stream of tiny inserts. The official connector is built on Flink's AsyncSinkBase, so its batching behavior is governed by a small set of shared controls: the maximum batch size, the limit on buffered requests, the flush timing that forces a write when a batch is not full, and the number of in-flight requests allowed at once. Larger batches and longer flush intervals produce fewer, bigger inserts, which means fewer parts for ClickHouse to create and merge in the background. The cost is latency, because events wait in the sink buffer longer before they are queryable. Smaller batches and shorter flush intervals reduce that delay but raise the insert rate and the part count, which can push a cluster toward merge backlog under sustained load.
Do not take the defaults on trust. Before going to production, open the connector documentation for the exact Flink and connector versions you run (1.17 or later, or 2.0 with Java 17), and confirm the parameter names, their defaults and whether each one is set on the sink builder or in configuration. Then benchmark with event rates and record sizes taken from your real streams, including the peak and burst patterns you expect, not a synthetic constant load. Watch the system.parts count per table and the merge queue on the ClickHouse side while you change one setting at a time, so you can see which control actually moves the part count and which only moves latency.
Plan for the failure cases as well as the steady state. A large in-flight limit can multiply the pressure on a cluster during a slow period, and a large buffer holds more unacknowledged data in Flink memory if the sink backs off. Agree with your data owners how much delay the business accepts for freshness, and record the chosen values with the benchmark results that justify them, so a future connector upgrade or a change in event volume starts from evidence rather than from memory.
Mapping Flink record types to ClickHouse columns
This table pairs common Flink record shapes with the ClickHouse column types they usually land in, and flags where the mapping needs checking before go-live. Read each row as a question to settle in the design phase, not as a guarantee of connector behavior. The main takeaway is that type mismatches and a weak ordering key are the two changes that are hardest to undo once data is loaded.
| Flink type or field | Likely ClickHouse type | What to verify or decide |
|---|---|---|
| Timestamps (Instant, LocalDateTime, epoch millis) | DateTime64 with explicit precision | Confirm how the connector serializes each Java time type and which time zone it assumes. Set the column time zone deliberately in the table definition. |
| Nullable fields (boxed types, null values) | Nullable(T) columns | Confirm nulls pass through the connector unchanged. Nullable columns add storage and query overhead, so mark only fields that are truly optional upstream. |
| Decimal values (BigDecimal) | Decimal(P, S) with fixed precision and scale | Match precision and scale to the source system. Test rounding and overflow on a sample stream, because a mismatch can fail or silently truncate. |
| Arrays (List or primitive arrays) | Array(T) columns | Confirm element types and whether nested arrays are supported by the connector version you pin. Check that array length does not drive large row sizes. |
| Maps (Map<K, V>) | Map(K, V) columns | Verify key and value type support in the connector documentation. Consider flattening high-cardinality keys into typed columns for query speed. |
| POJO versus Row field naming and order | Column names and positions in the target table | Confirm how the sink maps fields: by name or by position. Reordering POJO fields can break inserts if mapping is positional. |
| Ordering key fields (for example, event time and tenant or device ID) | ORDER BY key in MergeTree | Choose this deliberately before loading data. The ordering key sets storage layout and query pruning, and changing it later means rebuilding the table. |
| Event time and ingestion time fields | Partition key and TTL column | Decide which timestamp drives partitions and retention. Mixing late-arriving event time with ingestion time complicates TTL and backfills. |
Choosing failure and restart behavior for the sink
Some pipelines can absorb the same record arriving twice, as long as the table design makes a second write harmless. This split suits append-only event data where each row carries a stable identifier and occasional duplicates do not change the questions the business asks. Before you rely on it, confirm what the connector does on restart. After a failure, Flink restores from its last completed checkpoint, and records processed after that checkpoint can be sent again.
- Stable row identity. Carry an event ID or business key in every row so a replayed row can be recognized and filtered in queries.
- Duplicate-insensitive metrics. Confirm that dashboards and alerts either tolerate duplicates or read from a deduplicated view.
- Checkpoint interval as a bound. The interval between checkpoints limits how much data may be re-sent after a restart, so record the value you chose and the reason for it.
- Restart visibility. Log restart counts and reprocessed time ranges so duplicate volume is measured and reported.
Check how the sink behaves during restore for your exact Flink and connector versions before you accept duplicates in a production table.
When duplicates cannot reach reports, deduplication moves into ClickHouse. ReplacingMergeTree keeps one row per sorting key during background merges, so duplicates disappear over time, not at insert time. Keyed writes or a staging table with a deterministic key are other common options. Because merges run in the background, queries must account for copies that have not merged yet.
- Sorting key choice. The ORDER BY key defines which rows count as the same event, so include only the fields that uniquely identify it.
- Version column. Use a version column so the most recent copy wins when rows collapse during merges.
- Read path cost. Queries need FINAL or an aggregation pattern that handles unmerged rows, and that adds cost on large tables.
- Write mode support. If the job writes keyed updates, confirm the connector supports the write mode you need before you design around it.
Test the query path on a copy with representative volume, because deduplication timing changes what dashboards show between merges.
Some teams keep the source, such as a Kafka topic or a set of object storage files, with enough retention to re-run a time window. Replay can correct logic errors or lost sink writes when the source still holds the data. Its practicality depends on source retention and on whether you can limit the replay to a time range without loading again rows that are already correct.
- Retention that covers the window. Confirm the source keeps data longer than the longest period a bad write could go unnoticed.
- Partition-scoped reload. Delete or replace the affected time partitions before re-running, so the replay does not stack copies on good data.
- Savepoints and offsets. Save Flink savepoints and source offsets so the replay start point is known and recorded.
- Combined with deduplication. Pair replay with ReplacingMergeTree or keyed writes when the replay window overlaps live processing.
Rehearse a single partition replay on a spare cluster before you depend on it during an incident.
ReplicatedMergeTree keeps replicas in step, so a recovered node syncs missing data from a healthy replica without manual work. The same mechanism copies bad writes. A duplicate burst, a faulty transform or an accidental delete reaches every replica. Backups made with BACKUP and RESTORE to separate storage protect against that, but only once restores have been tested. The backup documentation also states that replication alone does not protect against human error or corruption.
- Replicas copy mistakes. A job that writes wrong values updates every replica in the same way.
- Backups before large replays. Take a backup or freeze the affected partitions before a replay so the change can be reversed.
- Scheduled restore tests. The ClickHouse backup documentation warns that a backup never test-restored will probably not restore when needed, so schedule restore tests on spare clusters.
- Documented connector guarantees. Read the connector documentation and confirm with the maintainers what delivery behavior applies when a checkpoint fails, then record it in the runbook.
Write down the recovery path for each job before production, including who runs it and what the steps involve.
Securing the connection between Flink and ClickHouse
A Flink job that writes to ClickHouse holds a database identity with write rights, so the connection needs the same care as any other service account. The four controls below cover who the job logs in as, how that identity is protected in transit and at rest, and which network paths are open. Check each one before the first production stream starts.
Dedicated insert-only user
Create a separate ClickHouse user for the Flink job and grant it INSERT rights only on the target tables, not on whole databases. Use the SQL-driven access control workflow with roles so that the grant can be reviewed and revoked without touching other applications. Confirm that the job never needs to read, alter or drop data, because a sink that only writes cannot leak existing rows if its credentials are exposed. Record which tables the role covers so reviews stay concrete.
Restricting the default user
ClickHouse ships with a default user that starts with all permissions, so leaving it open means any client that reaches the server can bypass the Flink role model. Restrict or disable the default user in your configuration, then confirm that the Flink connection cannot fall back to it if a setting is missing. Test this on a staging cluster by deliberately omitting the credentials from the job and checking that the write fails.
TLS on the client connection
Encrypt the connection between the Flink task managers and ClickHouse with TLS so that credentials and inserted rows are not readable on the network. Decide which port the sink uses, then confirm the certificate chain is trusted by the Java runtime that Flink runs on, since a custom certificate authority often needs an explicit truststore. Ask your security team whether mutual TLS with client certificates is required, and confirm with the connector documentation how your chosen client settings expose these options.
Credentials in Flink configuration
Keep the ClickHouse password out of job code, container images and plain text in flink-conf.yaml. Pass it from a secret store or an orchestrator secret mounted at runtime, and rotate it through the same process as other service accounts. Check whether job logs, the Flink web UI and savepoints can display configuration values, because a password that appears there is effectively exposed. Use a dedicated secret per environment so a staging credential cannot reach production.
Network rules between Flink and ClickHouse
Allow traffic only from the Flink task managers to the ClickHouse HTTP or native ports, and block all other sources at the firewall, security group or Kubernetes network policy. Include every ClickHouse node that the sink can reach, because a sink writing through a load balancer or directly to individual replicas needs rules for each path. Also restrict Keeper ports so that only ClickHouse nodes can reach them. Verify the rules with a connection test from a host outside the Flink cluster.
External authenticators for Flink
ClickHouse can delegate authentication to LDAP, Kerberos, SSL X.509 certificates, an HTTP authenticator or JWT tokens. Choose one only if your identity team already operates it, because a shared directory can add a dependency that stops streaming jobs when it fails. Ask your identity team how service accounts are issued, renewed and disabled before you configure the job. Confirm which of these methods the connector and your ClickHouse version support for the sink user.
Testing the pipeline before production traffic
Test the Flink to ClickHouse pipeline in a pre-production ClickHouse cluster before any production stream is pointed at it. The ClickHouse production guidance recommends a pre-production environment with representative data and queries, so the test cluster should receive records with the same schema, key cardinality, batch sizes and peak event rate you expect in production. Ask your team to agree these inputs in writing before the test starts, because a pipeline that passes at a fraction of the real rate tells you little about how the AsyncSinkBase batching behaves under load. Confirm the Flink version (1.17 or later, or 2.0 with Java 17 or later) and the connector artifact on the test cluster match what production will run.
Run a restart and failover test while checkpointing is active. Kill a TaskManager, then restart the job from its latest checkpoint, and repeat with a ClickHouse node taken out of service if your topology has replicas behind the sink. Watch for duplicate rows, gaps and sink errors during recovery. Then compare source and target counts for a bounded window of events, grouped by a business key and by event time, so that missing or doubled records show up as specific mismatches rather than a single total. Because the sink writes asynchronously, record how many events were in flight when the failure was injected, and confirm whether your job's delivery guarantee holds after recovery. Confirm this behavior with the connector documentation for your version rather than assuming it.
Finish with a restore test of the target tables using the backup approach your team already runs. The ClickHouse backup documentation warns that a backup never test-restored will probably not restore properly when needed, and recommends regular restore tests on spare clusters. Take a BACKUP of the tables the Flink job writes, restore them to a separate cluster, and run the same count and key comparisons against the restored data. Record how long the restore took and whether the restored tables match row for row. If your team has no backup routine for these tables yet, treat that gap as a blocker for production traffic, since replication alone does not protect against human error or corruption.
Common failures when Flink writes to ClickHouse
This table lists the failures teams most often hit when a Flink job writes into a self-hosted ClickHouse cluster, with the likely cause and the first fix for each. Read it before the first load test, and use the first column to match an observed symptom to a row. The main takeaway is that most failures trace back to version pairing, runtime levels, or sink batching rather than to ClickHouse itself.
| Failure | Likely cause | First fix |
|---|---|---|
| Job fails at startup with class or method errors | Connector artifact built for a different Flink minor line than the cluster runs, for example a 1.17 artifact on a 2.0 runtime | Match the connector artifact to the Flink version in use, rebuild the fat JAR, and redeploy the job from a clean dependency tree |
| Job refuses to start or fails on class version errors | Flink runtime or build JVM is below the required level: Java 11 or later for Flink 1.17, Java 17 or later for Flink 2.0 | Confirm the JDK on task managers and build agents, then upgrade the runtime before changing connector settings |
| Many small parts and rising merge load on ClickHouse | Sink flushes too frequently, so each flush creates a separate small insert and a new data part | Raise the batch size and flush interval in the sink settings, then check part counts in system tables during a test run |
| Insert errors or wrong values on timestamp columns | Flink timestamp type or time zone does not match the ClickHouse DateTime or DateTime64 column definition | Map Flink timestamps to an explicit ClickHouse type with a stated time zone, and test with values near midnight and daylight saving changes |
| Duplicate rows after a job restart | Sink retries or checkpoint replay re-send records that ClickHouse already stored, and MergeTree does not deduplicate by default | Make the target table idempotent with a deduplication strategy such as ReplacingMergeTree keys or a batch-level token, and test a forced restart before go-live |
| Growing backpressure and rising checkpoint times | Inserts queue behind slow background merges or an overloaded replica, so the sink cannot drain the stream | Check merge and insert metrics on the target cluster, reduce insert concurrency, and widen batches before adding more Flink parallelism |
| Data missing after a ClickHouse node failure | Writes were sent to a single replica or without retry handling while that node was down | Point the sink at a load balancer or several nodes, and confirm the target table is replicated before relying on it |
| Schema change breaks the running job | Column added or renamed in ClickHouse without matching Flink record types | Change the Flink record type and ClickHouse table in a planned order, and stop the job or test the change in a pre-production environment first |
Questions teams ask before building a Flink to ClickHouse pipeline
What drives the cost of a Flink to ClickHouse pipeline?
The main drivers are the number of Flink jobs, the number of target tables and schemas, the delivery guarantees the business needs, and the scope of testing. Each additional job adds its own source handling, sink configuration, batching tuning and load testing. Each additional table adds schema mapping and type checks. Stronger delivery expectations add failure and replay testing, which is often the largest share of effort. Cost is mostly one-time build work plus any recurring retainer for upgrades and support. Ask for a quote that lists jobs, tables, delivery expectations and test cases separately, so you can see which items move the price.
How long does a Flink to ClickHouse integration take?
Duration depends on the same inputs as cost: job count, schema count, source complexity and how much load testing the target volume requires. A single job with one well-understood stream and a cluster that already runs is a much smaller piece of work than several jobs feeding a replicated cluster with strict validation. DeployOpen does not publish a timeline. Your written scope sets the schedule, so ask for milestones tied to deliverables such as a working sink, a tested batching profile and a documented runbook.
What does our team need to provide before the work starts?
Three things make the largest difference. You need access to the ClickHouse cluster, including network paths from the Flink environment and a non-default user with the privileges the sink requires. You need representative sample data or a representative event stream, plus the target table definitions or the intended schema. You also need to confirm who owns the Flink cluster, its job deployment process and its upgrade schedule, since the connector is a dependency that must match your Flink version. Agree these items in the scope document so delays do not come from missing access.
Can our in-house team build this without a partner?
Yes, if your team already runs Flink in production and has Java experience, the connector is a documented, supported path. The work that usually needs experience is batching tuning under real volume, failure and replay testing, and sizing the ClickHouse side so inserts do not overwhelm merges. Ask whether your engineers have run high-volume sinks before and whether they can reproduce production-like load in a pre-production environment. A partner is most useful when those skills are missing or the deadline is tight.
Does the ClickHouse Flink connector support Table API or SQL-only jobs?
No, not today. The official connector supports the DataStream API, and Table API support is planned but not available. If your Flink jobs are written in SQL or the Table API, you will need to either rewrite the sink portion in DataStream code or choose a different ingestion path. Confirm the current status with the connector documentation before you commit to a design, because planned features can change.
Which Flink and Java versions does the connector require?
The connector requires Flink 1.17 or later and Java 11 or later. For Flink 2.0, the requirement is Java 17 or later. Check these before you choose a Flink version, because a runtime upgrade can force a JDK change across your job build and cluster. Confirm the exact artifact for your Flink line in the connector documentation, and test the build on the JDK your cluster actually runs.
What breaks most often in these pipelines, and what should we check?
The most common problems are batches that are too small, which create many small inserts and heavy merge work on ClickHouse, and batches that are too large, which increase memory use and latency. Check that the Flink job restarts cleanly, that replays do not create unexpected duplicates in your target model, and that schema changes on either side are caught before deployment. Measure throughput and ClickHouse insert health at your expected peak volume, not only at sample volume, before you call the pipeline finished.
What DeployOpen delivers for a ClickHouse Flink integration
The engagement produces a written integration design for the Flink to ClickHouse path, followed by configured sink code, dependency setup, batching validation under load, restore and restart test evidence, and runbook notes your operators can use after handover. The design fixes which Flink line you run (the connector supports Flink 1.17 or later, and Flink 2.0 requires Java 17 or later), which sink and table layout you write to, how the sink's batching and flush settings were chosen for your message sizes and rates, and how a failed write is retried or routed. Dependency setup covers the connector artifact that matches your Flink version, the ClickHouse Java client version it pulls in, and the build files that keep both reproducible in your CI pipeline.
Batching validation runs your representative stream against a pre-production cluster that matches your ClickHouse topology, so the batch size and flush interval are measured rather than guessed. Restart testing stops and restarts the Flink job from a checkpoint and confirms that rows are neither lost nor duplicated at the ClickHouse side, and restore testing confirms that data written during the test can be recovered with the backup procedure described in the backup and disaster recovery work. The evidence pack records the test inputs, settings, throughput figures, and pass or fail results, and it names the open risks, such as any duplicate window the sink design allows. Runbook notes cover the alerts to watch, the Flink and ClickHouse metrics that show backpressure, and the steps for a stuck job.
The work is agreed as a fixed written scope before it starts, with deliverables and timelines set out in that scope. Ask for the Flink and ClickHouse versions in scope, the expected event rate and record size, the deduplication and delivery guarantees your downstream reports need, and the environment where load tests will run. Ongoing support, upgrades when Flink or ClickHouse releases change the connector, and recurring restore tests are available as a retainer, so the integration stays current after handover.
How DeployOpen works on this
DeployOpen plans, implements, tests, documents and supports self-hosted open-source platforms inside your own infrastructure, including the ClickHouse and Apache Flink integration described here. Work starts from a written scope, so the Flink jobs, sink settings and ClickHouse tables are agreed before any code is written.
How we scope
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
- The scope names the Flink version (1.17 or 2.0), the Java runtime, and the DataStream jobs to be delivered.
- We confirm event volumes, record shape and target MergeTree table design so batching settings can be sized.
How we build
- We build the Flink sink connector pipeline in a non-production environment with representative event data.
- Batching, flush and retry settings are tuned against measured throughput and ClickHouse insert behavior.
- We test failure cases such as job restarts, ClickHouse node outages and malformed records before release.
How we hand over
- You receive job code, connector configuration, table definitions and a runbook covering deployment and rollback.
- Documentation records the version pairings, Java requirements and the limits of DataStream-only support.
- Handover includes a walkthrough with your platform and data engineering teams, who then run the pipeline.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- Support can cover Flink and connector version upgrades, reviewed against the changelog before each move.
- Ask any provider to confirm response expectations and upgrade responsibilities in the written agreement.
