How ClickHouse subscribes to NATS subjects with the NATS table engine
If your team runs self-hosted ClickHouse and needs messages from NATS in analytical tables, the supported method is the NATS table engine. It subscribes ClickHouse to one or more NATS subjects, including wildcard subjects such as foo.*.bar, and reads the messages into a table that queries can scan. The engine handles the subscription. Your work is in deciding which subjects feed which tables, how the rows are shaped, and where the data goes next.
The common pattern pairs the NATS table engine with a *materialized view*. The engine table acts as a queue-like source that is read continuously, and the materialized view moves each batch of parsed rows into a MergeTree table that holds the history. Because the source table is only a read path, the destination table is where you set the ordering key, partitioning, retention and access rules. Plan the destination schema before you write the view, since changing it later means rebuilding the pipeline. Confirm the exact engine settings for consumer counts and flush intervals against the ClickHouse NATS engine documentation for the release you deploy.
This section covers the ClickHouse engine only. It does not cover operating NATS itself, such as configuring JetStream streams, consumers, replicas or retention on the NATS side. If you depend on replay, durable delivery or stream retention, those guarantees come from how NATS is configured, so have the NATS owner confirm them before you design the ClickHouse side. Treat the engine as one consumer in your NATS topology and test how it behaves when the subject has no traffic, when ClickHouse restarts and when a message cannot be parsed.
What you need before creating a NATS table in ClickHouse
A NATS table engine link depends on four things lining up: a supported ClickHouse server, a current NATS server, a network path between them, and a secure way to hand over credentials. Work through these before any CREATE TABLE statement, because each one is cheap to check now and expensive to discover during a failed ingest.
Supported ClickHouse server version
The ClickHouse security policy lists 26.9, 26.8, 26.7 and 26.3 as versions that receive security updates, and all other versions are unsupported. Pick a line from that list, preferably a stable release or an LTS line, and run the same version in your pre-production environment first. Confirm in the NATS table engine documentation which ClickHouse releases include the engine and its current options, because support can differ between versions.
Current NATS server release
Check the nats-server releases page before you choose a version, since v2.15.1 and v2.14.8 were both published on October 9, 2026, after v2.15.0 on September 17, 2026. Record the exact server version you will connect to, because client behavior around subjects and authentication is tied to the server line. If you plan to use JetStream streams as the source, confirm that the stream and consumer you subscribe to exist on that same server version.
Network path from ClickHouse to NATS
Every ClickHouse node that will run a NATS table must reach the NATS client port on the NATS server or cluster member it connects to. Ask your network team to open that path from each replica, not only from the node you test on, and to confirm any firewall, proxy or Kubernetes network policy between the two. Also check how NATS cluster members are addressed, so that a failed server does not leave the table pointing at a host that no longer answers.
TLS and credentials files
Decide how the connection is encrypted and authenticated before you write the table definition. Agree which NATS user or account the table will use, and create a dedicated credentials file for it with only the subject permissions it needs. Keep that file out of SQL text, since the engine reads the credentials file path from a named collection in server configuration, and the query-level nats_credentials setting is the other route you should understand before choosing.
Access to server configuration
The nats_credential_file value can be defined only in a named collection in the ClickHouse server configuration, so someone with write access to that configuration must create it. Confirm who owns those files on your side, how changes are deployed to every node, and whether your change process can ship configuration without a full restart. Without this access, the team will be stuck passing credentials through queries, which is a weaker pattern for shared clusters.
Licensing and stewardship context
NATS server is licensed under Apache License 2.0 and is part of the Cloud Native Computing Foundation. In May 2025, CNCF and Synadia announced that the NATS trademarks would move to the Linux Foundation while the NATS.io domain and GitHub repositories stayed with CNCF and development continued under Apache-2.0. Record this in your platform decision, and have your legal or procurement team confirm that Apache-2.0 terms fit your internal use.
Configuring the NATS engine table in order
This sequence takes a NATS subject into a ClickHouse MergeTree table through a NATS engine table and a materialized view, with a check after each step so you know the pipeline is live before you rely on it.
Store NATS credentials in a named collection
Your ClickHouse administrator adds a named collection to the server configuration that holds the path to the NATS credentials file in nats_credential_file. The file stays on the ClickHouse host with restricted permissions, and only the server configuration can reference it. Confirm the collection loads by restarting or reloading the server as your change process requires, then checking that the named collection appears in the system tables for named collections. Note that queries never supply a file path: they pass credentials through nats_credentials, so the file location is not exposed to query authors.
Create the NATS engine table
Create a table that uses the NATS engine and lists the subjects to subscribe to, including wildcard patterns such as foo.*.bar where your subject design supports them. Set the consumer count and the flush settings to match your message rate and the batch sizes you want to write, and reference the named collection rather than a credentials path. Confirm the statement succeeds and that the table reports its settings back when you inspect its definition. Check the NATS engine reference for the exact setting names in your ClickHouse version, because they are version-dependent.
Create the target MergeTree table
Create a MergeTree table that stores the decoded messages with typed columns, an ordering key that fits your common filters, and any retention or TTL you need. The NATS engine table is only a stream of incoming data and keeps no history of its own, so this table is where the data is kept and queried. Confirm the table exists and accepts a test insert with the expected column types before you connect the pipeline.
Attach a materialized view
Create a materialized view that selects from the NATS engine table and writes into the MergeTree table, mapping each message field to a target column. The view is what moves rows from the subscription into storage, so it must exist before consumption begins for the data to be stored. Confirm the view appears in the system tables with the correct source and target, and that its select list matches the target column types.
Publish a test message to a matching subject
From a NATS client that your security team has authorized, publish a small test message to a subject that matches one of the subscribed patterns. Use a known payload so you can identify it in the target table. This step confirms that network access, credentials and subject matching all work together before real traffic depends on the pipeline. Record the subject and payload you used so you can remove the test rows afterward.
Verify the subscription and row arrival
Query the target MergeTree table for the test payload and confirm the row arrived with the expected values. Then check that the subscription is active by reviewing the NATS engine table's status in the system tables and the server logs for connection or authentication errors. The sequence is complete when test rows arrive in the target table and no errors appear for the subscription. Ask your ClickHouse operator to confirm where these checks belong in your monitoring, because the exact system tables and log messages depend on your version.
Designing subjects, consumer counts and flush intervals
The NATS table engine subscribes to the subjects you name, and the subject layout decides how many tables and materialized views you need. Subjects such as foo.*.bar use wildcards, so one table can receive every message whose middle token varies, which suits a stream of events that share a schema. When event types need different column sets or different downstream tables, give each type its own subject family and its own engine table rather than filtering inside a single wide table. Before you settle the layout, list the subjects your publishers actually use, confirm which ones carry the same payload shape, and check that a wildcard does not also capture subjects you meant to exclude. Confirm the wildcard syntax against the NATS table engine documentation for your ClickHouse release.
The engine defaults to one consumer per table. Adding consumers lets several threads read the same subscription in parallel, which helps when message rates outpace a single reader, but it also adds load on the NATS side and can change how messages spread across consumers. Raise the count only after you have measured a backlog or a lag that a single consumer cannot clear, and change one variable at a time so the cause of any improvement stays clear. Your team should own this decision, because the right count depends on your message size, the cluster sizing and the number of tables sharing the server.
The flush interval and the batch size set the trade-off between latency and insert efficiency. Small, frequent flushes make new rows visible sooner but create many small parts that ClickHouse must merge later, while larger batches insert more efficiently and wait longer before data becomes queryable. Pick the latency your dashboards and alerts truly need, then tune batch size and flush timing together against representative message rates. Run this in a pre-production environment with a replay of real traffic, watch part counts and merge activity, and record the settings you chose with the reasons behind them. Ask your NATS and ClickHouse operators to confirm the values that match your peak load, since no single setting fits every workload.
Mapping NATS message fields to ClickHouse columns
This table compares each part of a NATS message with the ClickHouse column or setting that receives it, so you can see where values come from and where they can be lost. Read it before writing the materialized view, since most mapping errors appear as empty or defaulted columns rather than failed inserts. The main takeaway is that the subject, payload and consumer identity must each be mapped on purpose, and malformed payloads need a stated handling rule.
| Message part | Mapping approach | What to confirm or decide |
|---|---|---|
| Subject tokens | Extract tokens from the subject with string functions in the materialized view, such as splitting on the dot separator, into a LowCardinality String column | Wildcard subscriptions like foo.*.bar match many subjects, so decide whether the token is stored as a column or dropped |
| Payload to typed columns | Parse the payload in the materialized view using the data format set on the table, then cast each field to its target type | Confirm the supported data formats in the NATS engine docs before choosing the payload format |
| Timestamps | Use a message field if the publisher sets one, or a server-side default such as now() in the view if it does not | Pick one timezone and precision, and store it as DateTime64 so late or replayed messages sort correctly |
| Malformed payloads | Parse failures can stop a consumer or drop rows depending on settings, so test bad input before go-live | Confirm how the engine reports parse errors and whether a dead-letter subject or staging table is needed |
| Consumer identity | Each table defaults to one consumer, so record the table or consumer name in a column to trace which subscription wrote a row | Confirm how consumers are named and whether multiple tables share a queue group |
| Credentials | Keep credentials out of queries by referencing a named collection that holds the credentials file path | Schema changes to the view do not alter stored rows, so plan a backfill strategy if a field mapping changes |
Securing credentials and subject access
The NATS engine puts two things at risk: the credentials that let ClickHouse read a subject, and the SQL users who can read what the engine receives. Handle both in the server configuration and in ClickHouse access control, not in ad hoc query text. The cards below cover each control in the order a reviewer is likely to check them.
Keep the credentials file in a named collection
The engine accepts nats_credential_file only from a named collection defined in the server configuration. Put the path there, not in a table definition, so the file location is set by an administrator and not by whoever can run CREATE TABLE. Confirm that the collection file is readable only by the ClickHouse service account and that it is not copied into backups or shared configuration repositories without review.
Pass query credentials with care
Queries can pass credentials through nats_credentials, but any value written in SQL can end up in query logs, client history and monitoring output. Prefer the named collection for standing subscriptions and reserve query-level credentials for short, supervised tests. Ask your team to confirm which query logs capture the statement text and to mask or exclude those fields.
Give the subscriber its own NATS identity
Create a dedicated NATS user for ClickHouse, scoped to the subjects it must read, such as one wildcard pattern that matches the events in scope. Do not reuse an operator or application identity. If several teams share a server, place ClickHouse in its own NATS account so that its subscriptions cannot see other messaging domains. Review the permissions whenever a new subject is added.
Restrict which ClickHouse users can read the engine
A NATS table is a live view of messages in transit, so granting SELECT on it is granting access to that data. Create ClickHouse roles for the engine table and the materialized views that write to MergeTree, and grant only those roles to analysts. Use row policies where a subset of messages must stay hidden from some users.
Restrict the default user
A fresh ClickHouse server starts with a default user that has all permissions. Before connecting NATS, disable or restrict that account, create named administrative users and assign roles through SQL-driven access control. Confirm the change by connecting as a non-administrative user and attempting to create a NATS table, which should fail.
Encrypt the NATS connection and plan authentication
Enable TLS between ClickHouse and the NATS server so that credentials and message payloads are not sent in clear text across the network. Decide whether ClickHouse's own users authenticate through LDAP, Kerberos, X.509 certificates, an HTTP authenticator or JWT, and confirm which of these your environment supports before you design around them. Verify the TLS settings and the NATS server certificate chain with your platform team.
Where delivery guarantees depend on JetStream
The delivery guarantee you get from the NATS table engine depends on whether the subject is backed by core NATS or by a JetStream stream, and that difference is the first thing to settle. Core NATS delivers messages to subscribers that are connected at the moment of publication. If no subscriber is listening, the message is not stored for later, so a ClickHouse server that is restarting, upgrading or unreachable can miss traffic it never saw. JetStream adds a persistence layer that stores messages for streams bound to subjects, gives at-least-once delivery, and lets consumers replay from the beginning, the latest message, a sequence number or a point in time. Confirm with the NATS table engine documentation for your ClickHouse version whether the engine consumes core subjects, JetStream consumers, or both, and whether it acknowledges messages in a way that gives at-least-once behavior end to end. Do not assume exactly-once semantics without a written test result from your environment.
If messages must survive subscriber downtime, the durability plan sits mostly on the NATS side and should be agreed before ClickHouse is configured. Decide how long each stream retains messages, which limits apply, and how much disk the stream needs at your peak message rate, then test a planned ClickHouse outage against that retention window. For replication, the JetStream clustering guidance recommends an odd number of servers, typically three or five, treats R3 replication as the production floor, and caps a stream at five copies. Ask your NATS owner to confirm the replica count and the server count for the environment that will hold the stream, because a stream with fewer replicas can lose data during a node failure even when ClickHouse behaves correctly. Record these values in the runbook next to the ClickHouse consumer settings so that an on-call engineer can tell whether a gap came from the stream or the table.
Common NATS engine failures and how to fix them
This table lists the failures teams most often hit when a ClickHouse NATS table engine goes live, with the likely cause and the fix for each. Read it as a triage guide: match the symptom you see, check the cause column, then apply the fix in the order given.
| Failure | Likely cause | Fix |
|---|---|---|
| Authentication errors on connect | The credentials file path was passed in the query, or nats_credential_file was set outside a named collection in the server configuration | Move the path into a named collection in the server configuration as nats_credential_file, and pass only nats_credentials from the query. Confirm the file is readable by the ClickHouse service account. |
| Engine cannot reach the NATS server | Wrong host or port, a firewall or network policy blocking the client port, or TLS settings that do not match the server | Test connectivity from the ClickHouse host to the NATS client port first. Then check the server address, firewall rules and any TLS configuration on both sides. |
| Wildcard subject receives no messages | The pattern token count does not match the published subject, for example foo.*.bar will not match foo.a.b.c | Print a sample of the published subjects from the producer side and compare token by token. Adjust the wildcard or the subject list until each published subject matches one pattern. |
| Table exists but no rows arrive | No materialized view reads from the NATS engine table into a MergeTree table, so messages are consumed but not stored | Create a materialized view that selects from the NATS table and inserts into the target MergeTree table. Check that the column names and types in the view match the target table. |
| Duplicate rows after a restart | Core NATS delivery is not guaranteed to be exactly once, and messages redelivered or replayed around a restart can be inserted twice | Design the target table to tolerate duplicates, for example with a deduplicating engine or a unique event key, and test restarts in a pre-production environment with representative traffic before go-live. |
| Messages missing after a restart | Messages published while no consumer was subscribed were not stored, because core NATS does not persist messages for absent subscribers | Confirm with the NATS team whether the subjects are backed by JetStream streams. Plan the ingestion path and expected loss window before relying on the data for analytics. |
| Several tables consume the same subject unevenly | Each table creates its own consumer by default, so duplicate table definitions or unexpected consumer counts split or repeat the load | List the NATS table engines defined on each node and remove duplicates. Set the consumer count deliberately and check the monitoring view of subscriptions on the NATS server. |
| Ingestion slows or stalls under load | Flush interval or block size set too high for the message rate, or too few consumers for the volume | Review the flush settings and consumer count against observed throughput in a test environment. Change one setting at a time and watch insert latency and merge activity. |
Questions teams ask before connecting ClickHouse to NATS
How much effort does a ClickHouse and NATS integration take?
Effort depends on how many subjects you consume, how cleanly those subjects are designed, and how many tables and materialized views sit between the NATS engine and your MergeTree targets. A single subject with a stable JSON payload is a small build. A wildcard design such as foo.*.bar across several producer teams, with mixed message formats, takes longer because each format needs a parsing and typing decision. Security review adds time when credentials must be issued per table and stored in a named collection in the server configuration. Ask any partner to break the estimate into subject design, table and view build, load testing and handover, and to show which items depend on your teams.
Can our in-house team build this without a partner?
Yes, if your team already runs ClickHouse and NATS in production and has someone who can write materialized views and read NATS subject hierarchies. The integration itself is a table definition, a materialized view and configuration, so the core skill is not unusual. The gaps usually appear around delivery behavior, flush settings, consumer counts under load, and credential handling in named collections. A partner is most useful when the team lacks one of those skills, needs the work done before a fixed date, or wants an independent review of the design before it reaches production.
What do we need to provide for the work?
Plan on providing the following before the build starts:
- NATS access, including the subject list, the accounts involved and a credentials file for a test user with the minimum permissions needed
- ClickHouse administrator access to a non-production cluster and the rights to edit the server configuration that holds named collections
- Sample messages from each producer, covering normal, malformed and oversized payloads
- A test environment with representative message rates, so flush intervals and consumer counts can be tuned before go-live
Ask the partner to confirm this list against your architecture, because access to the production NATS cluster is usually limited to a read-only path.
How long does a pilot take, and what changes that timeline?
The pilot length depends on the number of subjects in scope, how far the message formats are already documented, and how quickly your teams can approve test credentials. A pilot on one subject with clean JSON is shorter than a pilot covering several wildcard subjects from different producers. Timing also shifts when the test environment needs its own NATS server, when security review requires a formal sign-off, or when load testing reveals that flush intervals must change. Ask for a written timeline with named dependencies, so each delay has an owner.
Does DeployOpen work on a fixed scope with written deliverables?
Most DeployOpen engagements are fixed-scope. Scope, deliverables and timelines are agreed in writing before work starts, and a quote follows from that written scope. For this integration, the scope typically names the subjects to consume, the target tables and materialized views, the credential approach, the load and failure tests, and the handover documentation. DeployOpen does not publish a price list. Ongoing support, upgrades and continuous testing are available as retainers, which are separate from the initial build.
Should we use NATS or Kafka for this data path?
Choose based on your existing platform, delivery requirements and operating skills. NATS is a small single binary that runs on premises, in the cloud or at the edge, and JetStream adds persistence with at-least-once delivery. Kafka has its own consumer group model, retention behavior and tooling. Teams that already run Kafka usually gain more from the Kafka engine, while teams that want a lighter messaging layer may prefer NATS. The ClickHouse and Kafka integration comparison in the sibling topic covers durability, fsync and lag monitoring, which should inform the choice.
What breaks most often in a NATS table engine setup?
The most common problems come from unclear subject design, credentials handled inconsistently between queries and the server, and flush or consumer settings left at defaults. A wildcard subject that matches more traffic than expected can overload a single table. Credentials passed in a query are not a substitute for the named collection approach, because the NATS engine accepts the credentials file only from a named collection in the server configuration. Test each failure path, including broker restarts and malformed payloads, before go-live.
How DeployOpen scopes a ClickHouse and NATS engagement
A ClickHouse and NATS engagement starts with a written scope that names every subject pattern the cluster will consume, the stream or subject owners for each, the target tables and their ordering keys, and who signs off on the consumer design. Scope also records the decisions that change the build: whether each NATS table has its own consumer or shares one, which flush interval the business can tolerate for late data, and whether the subjects are core NATS messages or sit on a JetStream stream with replay. Deliverables and timelines are agreed in that document before work begins, so the customer knows what is built, what is handed over and what is out of scope.
The inputs the customer provides carry most of the scoping risk. Ask for the NATS server version and topology, the account and user model that will issue the credentials, sample messages for each subject family, and the expected message rate and message size at peak and at normal load. The test environment should replay representative traffic at those rates into a ClickHouse cluster sized like production, because a flush interval or consumer count that looks correct at low volume can behave differently under load. Confirm where the credentials file will live on each ClickHouse host and how the nats_credential_file named collection will be created, since the query-level nats_credentials setting is not a substitute for it in server configuration.
Handover documentation should let the operations team run the integration without the build team. Expect a written description of each NATS table, its materialized view, the named collection and credential rotation steps, the expected subject wildcards, and the monitoring checks for consumer lag and failed inserts. The document should also state which NATS and ClickHouse versions were tested together and what to re-verify before an upgrade on either side. Ask any prospective partner to show a sample handover pack and to state, in the quote, which items are included and which the customer's team must own.
How DeployOpen works on this
DeployOpen plans, builds, documents and supports self-hosted open-source platforms inside your own infrastructure, including the ClickHouse NATS table engine. Work starts from a written scope, and each stage ends with artifacts your team can review and keep.
How we scope
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- The written scope names the NATS subjects, wildcard patterns and consumer counts that ClickHouse will subscribe to.
- Scope also records where NATS and ClickHouse run, which credentials model is used and which teams consume the tables.
How we build
- We configure NATS table engines, flush intervals and materialized views against a pre-production ClickHouse cluster first.
- Credentials files are placed in named collections in the server configuration, never passed in query text.
- Replay, duplicate and message-loss behavior is tested with representative subject volumes before any production cutover.
How we hand over
- Handover includes documented table definitions, subject mappings, named collection layout and the consumer configuration used.
- Runbooks cover restarting a NATS table consumer, rotating credentials files and checking consumer counts after changes.
- Your team receives test results and a list of the NATS and ClickHouse versions validated together.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, separate from the initial build.
- Support covers reviewing NATS and ClickHouse release notes before upgrades, since both projects publish frequent releases.
- Confirm during scoping which support hours, response expectations and upgrade cadence the retainer includes.
