Shipping logs to ClickHouse with Vector

How Vector ships Nginx logs to ClickHouse

Web access logs pile up on application servers as plain files, and teams usually want to ask questions of them with SQL: which paths return errors, which clients generate the most traffic, how latency changes after a release. ClickHouse's documented Vector guide covers exactly this path, tailing an Nginx log file with Vector and sending the events into ClickHouse. ClickHouse also lists Vector as a partner integration, so the pattern is documented by the database vendor rather than assembled from community scripts. The Vector integration guide is the natural starting point for the reference steps.

This topic covers three stages of one pipeline. The first is file tailing: Vector reads the Nginx access log as it grows, tracks its position so restarts do not replay the whole file, and handles log rotation. The second is parsing: raw lines become structured fields such as timestamp, status code, request path and byte count, which must match the column types of the target table. The third is the sink: Vector batches those events and writes them to a ClickHouse table over HTTP. Each stage has settings that change data quality, so the guide is worth reading against your own Nginx log_format before anything reaches production.

Out of scope are the ClickHouse cluster itself (sizing, replication with ClickHouse Keeper, table engines and retention policy), Nginx tuning, and the Kafka or OpenTelemetry alternatives covered in sibling topics. Before you start, confirm which ClickHouse version and Vector release your environment will run, since both are maintained on their own release schedules, and check that the database user Vector writes with has insert rights on one table only. A pre-production test with a copy of representative logs will show whether your parsing rules hold for the malformed or unusual request lines your servers actually produce.

What you need before Vector can write to ClickHouse

Vector needs a supported ClickHouse server, a database and table it can write to, and an ingest identity with narrow rights. On the web host, it needs readable Nginx logs, a network path to the ClickHouse HTTP port, and a planned install location. Confirm each item below before you write the sink configuration.

Supported ClickHouse release line

Choose a target version on a line the security policy lists as supported: 26.9, 26.8, 26.7 or 26.3. Lines outside that list receive no security updates, so an ingest pipeline built on them carries an unpatched server into production. Stable releases ship roughly monthly, and LTS releases are supported for a year after release, so pick the line that matches how often your team plans to upgrade. Confirm the current list before you build, because the policy changes over time.

Target database and table

Create a dedicated database and a MergeTree table for the Nginx access log before Vector starts. Decide the column types now, since Vector maps fields to whatever columns exist and a string-only table pushes parsing cost into every query. Pick an ordering key that matches your most common filters, such as host and time. Keep the table name and database stable, because they appear in the sink configuration and in any dashboards built on top of it.

Dedicated ingest user

Create a separate ClickHouse user for Vector with insert rights only on the target table and nothing else. Do not reuse the default user, which starts with all permissions, and do not share a human analyst account. Use the SQL-driven access control workflow to create the user and a role, then grant the insert privilege on the one table. Store the password in the Vector environment or a secrets store, not in the configuration file committed to version control.

Nginx log format and read access

Confirm the Nginx log_format directive produces a predictable, delimited line for each request, and record that format before writing parsing rules. Vector must be able to read the access log file, which usually means adding its service account to the group that owns the log, not running Vector as root. Check that log rotation keeps Vector following the current file so entries are not lost at rotation time.

Network path to the HTTP port

Vector sends rows to ClickHouse over HTTP or HTTPS, so the web host must reach the ClickHouse HTTP interface on its configured port. Confirm firewall rules, security groups and any proxy in between allow outbound traffic from the Vector host to that address. If the cluster sits behind a load balancer, confirm the path to each node you intend to write to, and plan TLS before the first production write.

Vector placement and update plan

Decide which hosts run Vector, usually each web server that writes an access log, and how Vector is installed and updated across them. Pick one install method, such as the operating system package or a container image, and use the same method on every host so version drift is visible. Record who owns updates and how a configuration change is rolled out and rolled back, because the sink configuration is shared by all those hosts.

Vector sink configuration steps

This sequence takes a team from an empty ClickHouse table to Nginx access log lines arriving in it through Vector, with a least-privilege ingest user and a check at each stage. Each step can be verified before the next one starts, which keeps faults easy to isolate.

Create the target table with an explicit schema
Create a database and a MergeTree table whose columns match the fields you intend to extract from the Nginx access log, such as timestamp, client address, request method, path, status code and response size. Declare each column with a concrete type rather than a catch-all string, and choose an ORDER BY key that matches how you will filter the data. Confirm the table exists by running a DESCRIBE TABLE query and checking that every column and type appears as you planned.
Create a dedicated ingest user with insert rights only
Create a ClickHouse user that Vector will authenticate as, and grant it INSERT on the target table only, with no rights to read other tables or to change settings. Restrict the built-in default user as well so that the ingest path does not depend on it. Confirm the grant by logging in as the new user and attempting an INSERT into the target table, which should succeed, then a SELECT from it, which should be denied. The ClickHouse access control documentation describes the SQL-driven workflow for these grants.
Configure the file source to tail the Nginx log
In the Vector configuration file, add a file source that points at the Nginx access log path and reads from the end of the file on first start, so that historical lines are not loaded by accident. Set a data directory that Vector can write its file checkpoints to, because those checkpoints let Vector resume after a restart without duplicating lines. Start Vector in a validation mode, if your version provides one, and confirm it reports no configuration errors. Then generate a request against the Nginx site and check that Vector's own log shows the file source reading a new line.
Add a parser transform for the Nginx log format
Add a remap transform that reads the raw line from the file source and parses it into the fields defined in your table, using the combined log format or the custom log_format your Nginx configuration declares. Convert the timestamp and numeric fields to the types your columns expect, and drop or route lines that do not match the pattern so that malformed input does not reach the sink. Send a known test request, then confirm that the transform output contains the expected field values by temporarily adding a console sink. Remove the console sink once the parsed fields look correct.
Configure the ClickHouse sink with batching limits
Add a clickhouse sink that takes the output of the parser transform and sets the endpoint for your ClickHouse HTTP interface, the database and table names, and the ingest user credentials, which should come from an environment variable or secret store rather than the file itself. Set batch size and timeout limits so that Vector groups rows into inserts of a practical size, and enable acknowledgements if you want the source to wait for ClickHouse to confirm each write. The exact option names change between Vector releases, so take them from the ClickHouse Vector guide and check them against the Vector version you run. Validate the configuration before starting the service.
Start Vector and confirm rows arrive
Start the Vector service and send a set of test requests to the Nginx site. Within the batch timeout you configured, run SELECT count() on the target table and check that the count rises by the number of test requests, then query the most recent rows and confirm that the parsed columns hold the values you expect. Check Vector's internal log for sink errors such as authentication failures or type mismatches, which appear there before they show up as missing data. Once the count and sample rows match, the pipeline is running end to end.

How Nginx log fields map to typed ClickHouse columns

This table pairs common Nginx access log fields with the ClickHouse column types a Vector parse step should produce. Treat each row as a starting point to check against your own log_format, since custom directives and proxy headers change what appears in each line. The main takeaway is that every parsed value must match its declared type, so the rows that can be missing or malformed need explicit defaults or nullable columns.

Nginx field	Suggested ClickHouse type	Parse and fallback notes
Client address ($remote_addr)	IPv4 or IPv6 (or String if mixed with proxy chains)	Parse as an IP type. If the address comes from an upstream proxy header, confirm which field is trusted before storing it.
Timestamp ($time_local)	DateTime or DateTime64	Parse with the timezone shown in your log_format. Confirm the timezone with the team before loading historical files.
Request method ($request_method)	LowCardinality(String)	Few distinct values, so LowCardinality keeps storage and filtering efficient. Malformed requests may yield an unexpected method, so set a default for empty values.
Request path ($request_uri)	String	Keep the full path as String. Query strings can carry personal data, so confirm whether they should be masked before storage.
HTTP status ($status)	UInt16	Always numeric in standard formats. A line that fails to parse should go to a dead-letter path rather than receive a guessed default.
Bytes sent ($body_bytes_sent)	UInt64	Can be zero for some responses. Use a default of 0 when the field is missing or contains a dash.
Referrer ($http_referer)	Nullable(String) or String with empty default	A dash or empty value is common. Choose one convention and apply it consistently across all sources.
User agent ($http_user_agent)	String	Often long and variable, so do not use LowCardinality unless you have measured few distinct values. Empty values need a default.

How batching and parsing choices affect ingestion

Batch size and flush interval decide how many insert parts ClickHouse has to create and merge. The ClickHouse sink in Vector buffers events and sends them as batches, so each flush becomes one insert. A large batch with a long flush interval keeps the part count low and merges cheap, but log lines reach queries later. A small batch with a short interval shortens the delay but produces many small parts, and the background merge process then spends CPU and disk I/O combining them. Tune these two values against your real log volume and the freshness your investigators need, not against defaults copied from another environment. Measure the part count and merge activity on a pre-production cluster with a representative Nginx log stream before you settle on numbers, and confirm the batch and timeout option names against the Vector version you deploy.

Parsing decides whether a line becomes a typed row or a silent loss. Typed columns such as a DateTime for the request time or a UInt16 for the status code only accept values of that type, so a malformed timestamp, an unexpected status or a truncated line will fail the insert or be coerced if the pipeline does not check it first. Handle this before the sink: use a parse transform that tags each event as parsed or failed, route failed events to a dead-letter file or a separate sink, and count them with a metric that you alert on. A parse failure rate that rises after a change in the Nginx log_format is the most common early warning. Decide who owns the dead-letter path and how long failed lines are kept, because an unmonitored discard path is where log data disappears without anyone noticing.

When to choose Vector over the OpenTelemetry Collector

Vector suits a team whose requirement is narrow: tail existing log files such as Nginx access logs, parse them, and write them into one ClickHouse table. The configuration stays small, one source feeds one sink, and the team can read the whole pipeline in a single file. This is also the right fit when you already run Vector for other work or want a lightweight agent on web servers that does not need a wider telemetry framework.

Pipeline count: One source, one transform and one ClickHouse sink usually cover the requirement, so there are few moving parts to review.
Existing skills: If your operators already maintain Vector configurations, the marginal learning cost is low.
Application logs: Logs from applications are collected only if they are written to files that Vector can read, so confirm how each service writes its output.
Schema work: Parsing into typed columns happens in Vector transforms, so someone must own those rules as the log format changes.
Growth: Adding traces or metrics later means a second tool or a rewrite, so confirm that the log-only scope is stable.

Check whether the log formats you need are fixed and whether any traces or metrics requirement is planned within the next upgrade cycle of your platform.

Securing the log path from Nginx to ClickHouse

The log path has three exposure points: the credentials Vector uses to write, the transport between Vector and ClickHouse, and the content of the access logs themselves. Each card below covers one of these decisions so the pipeline is secured before the first production load.

Dedicated insert-only ingest user

Create a separate ClickHouse user for Vector and grant it only the rights needed to insert into the target log table. Vector never needs to read, alter or drop data, so a leaked ingest credential should not be able to query other tables or change schema. Check the grants by logging in as that user and attempting a select on the same table, which should fail. Use roles so the same grant pattern can be reused for other shippers.

Restricting the default user

ClickHouse's default user starts with all permissions, so a fresh node exposes full administrative access to anyone who can authenticate as that account. Disable or restrict the default user, set a strong password or remove network access to it, and confirm that the ingest user is the only account Vector connects with. Review this during the rollout checklist, not after the first ingest run.

TLS on the HTTP endpoint

Vector should send logs to the ClickHouse HTTP interface over HTTPS, with the server certificate issued by an internal or public CA. Configure Vector to verify the certificate chain and hostname rather than disabling verification to get a connection working. Confirm the CA bundle path on the Vector host and test a rejected certificate to prove verification is active.

Credentials outside plain config files

Keep the ingest password out of the Vector configuration file, which is often readable by every operator with access to the host and is copied into configuration management repositories. Pass the secret through an environment variable or a secrets store the host can read at start-up, and restrict file permissions on any remaining config. Rotate the ingest password on a schedule and after any staff change.

Dropping or masking sensitive fields in parsing

Nginx access logs often include query strings that carry session tokens, reset codes, email addresses or other personal data, and client IP addresses may count as personal data under some regimes. Decide in the parse step which fields are stored raw, which are truncated, which are hashed and which are dropped, and document that decision with the data owner. Test the transform on a sample of real traffic before enabling the sink.

Limiting network exposure to the ingest path

Allow the ClickHouse HTTP and native ports only from the Vector hosts and the operators who administer the cluster. Enforce this with firewall rules or security groups and verify it from a host outside the allow list. Keep ClickHouse off public addresses, since a log store reachable from the internet exposes both the data and the endpoint to scanning.

Common failures when Vector writes to ClickHouse

This table lists the failures most often seen when Vector tails Nginx logs and writes to ClickHouse. Read each row as cause, then the check that confirms it, then the fix. Confirm Vector's retry and delivery behavior in its documentation before relying on any of these safeguards.

Failure	Likely cause and check	Fix
Vector cannot read the Nginx log	After rotation the new log file is owned by a different user or has tighter permissions than the one Vector's process can read. Check the Vector logs for permission denied errors and compare file ownership with the user Vector runs as.	Grant the Vector service user read access through group membership or a logrotate create directive that keeps the original mode and owner, then restart or reload Vector and confirm new lines arrive.
Type mismatch on insert	A parsed field, such as a status code or response time, contains a value like a dash or an empty string that does not fit its typed column. Check the ClickHouse server log and the rejected event sample for the failing field.	Map empty or malformed values to defaults in the Vector transform, or declare the column as Nullable where the data truly can be absent. Test the parser against a sample of real access log lines.
Table or database not found	The sink points to a database or table name that differs from what was created, or the table was never created on this cluster node. Check the sink configuration against the output of the ClickHouse table list for the target database.	Correct the database and table names in the sink, create the target table with the expected schema, and confirm the ingest user can see that database.
Authentication rejected for the ingest user	The username, password or authentication method in the sink does not match the ClickHouse user, or the user lacks insert rights on the target table. Check the ClickHouse server log for authentication failure messages and the user's grants.	Create a dedicated ingest user with insert permission on only the log table, store its credentials in a secret store rather than the config file, and retest the connection.
Duplicate rows after a Vector restart	Vector re-reads log lines it has already delivered, or retries a batch whose acknowledgment was lost, so the same events are inserted twice. Check whether row counts for a time window exceed the line count in the source files.	Confirm Vector's checkpointing and retry settings in its documentation, then use a deduplication approach suited to your schema, such as a stable event identifier and a ReplacingMergeTree design or query-time deduplication.
Gaps during log rotation	Vector may not follow a file that has been renamed and replaced, or lines written between rotation and reopening are missed. Check for missing time ranges around rotation timestamps in the logrotate schedule and Vector's file source behavior.	Confirm how the file source handles rotation in the Vector documentation, align the logrotate postrotate step with Vector's reload behavior, and compare line counts for the rotation window after each change.

Questions teams ask before shipping logs to ClickHouse

What drives the effort of shipping logs to ClickHouse with Vector?

The effort is driven mainly by the log formats, the number of hosts sending logs, how much parsing each format needs, and the retention and access model the business requires. A single Nginx access log with a stable format is a small job. Many services with custom formats, multiline stack traces, and different retention rules for each source take much longer to design and verify. Ask for the count of log sources, a sample of each format, the target table layout, and the retention and role requirements before comparing quotes.

Can our in-house team set this up using the ClickHouse Vector guide?

Yes, a team with Linux and ClickHouse experience can follow the ClickHouse Vector guide to tail an Nginx log file and send it to ClickHouse. The guide covers the basic pipeline, which is a good starting point for a proof of concept. Production work usually adds more: parsing into typed columns, batching tuned to your insert volume, handling file rotation, least-privilege ClickHouse users for the sink, TLS, monitoring of the pipeline, and a tested way to backfill after failures. Teams that have owners for those areas can often do it in-house. Teams without them should weigh the time against the risk of silent data gaps.

What does the work need from our side?

The work needs four things from your organization. First, representative log samples, including malformed lines and rotated files, so parsing is designed against real data. Second, access to the target ClickHouse cluster and to the hosts that run Vector, with a change window agreed in advance. Third, a test environment with a copy of representative data and queries, since ClickHouse recommends a pre-production environment before production changes. Fourth, a named owner who can decide on field names, retention and who may query which columns.

How long does a Vector to ClickHouse engagement take?

Duration depends on scope rather than a fixed figure. The main variables are the number of log sources, how many distinct formats exist, how many parsing rules must be written and tested, and how much access control and monitoring must be added. Data volume matters for batching and for the time needed to validate loads. We agree the timeline in the written scope before work starts, and it reflects the items above. Ask for a scope document that lists each source, its format, and the acceptance checks.

What is involved in running the pipeline after go-live?

Operation recurs. Vector configurations need changes when log formats change, new hosts are added, or a field is renamed. Vector itself needs upgrades, and each upgrade should be tested against the same sample logs before it reaches production. On the ClickHouse side, the cluster has its own release cycle, and ClickHouse advises reading the changelog before upgrading because minor backward-incompatible changes can occur. Plan for monitoring of ingestion lag and failed inserts, a log of configuration changes, and a named person who receives alerts.

When should we choose a fixed-scope engagement and when a retainer?

Choose a fixed-scope engagement when the sources, formats and target tables are known and can be written down, such as the initial build of a defined set of log pipelines. Choose a retainer when you expect continuing change: new services each quarter, regular Vector upgrades, schema evolution, or on-call support for ingestion problems. Most organizations start with a fixed-scope build and then decide whether ongoing support fits their operating model. DeployOpen does not publish prices, so ask for a written quote based on the scope you define.

Should we use Vector or the OpenTelemetry Collector for logs?

Choose Vector when your main need is reliable file tailing and direct delivery of logs such as Nginx access logs into ClickHouse, and your team wants a focused pipeline. Choose the OpenTelemetry Collector when you are standardizing traces, metrics and logs across applications and infrastructure, or when your services already emit OpenTelemetry data. The decision also depends on your existing tooling and the people who will maintain the configuration. Confirm the current ClickHouse exporter and sink options against the vendor documentation for each tool before you commit.

How DeployOpen scopes and delivers this integration

DeployOpen plans, builds, tests, documents and supports Vector pipelines that ship log files into a ClickHouse cluster you run in your own infrastructure. Each engagement starts from a written scope that names your log sources, target tables and the operating model your team will keep after handover.

How we scope

Inventory the log sources, such as Nginx access logs, and the fields your analysts need as typed columns.
Agree the target database, table names, ordering keys and retention window in the written scope.
Confirm which ClickHouse endpoint, credentials and network path Vector will use, and who owns each.

How we build

Configure Vector file sources, parsing transforms and ClickHouse sinks in version-controlled configuration files.
Tune batch size and flush settings against sample traffic, then test insert behavior with representative log volumes.
Verify row counts, column types and timestamps in a pre-production ClickHouse environment before any production traffic flows.

How we hand over

Deliver the Vector configuration, table definitions and parsing rules with a runbook for changing them safely.
Document how to recover from a restarted Vector agent, a rotated log file or an unavailable ClickHouse node.
Walk your operations team through monitoring Vector health and checking that ingested rows match what was written.

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, configuration changes and Vector upgrades are available as a retainer.
Ask for the support boundary, response arrangements and upgrade approach in the quote before signing.