Kubernetes logs to Graylog with Fluent Bit

Kubernetes container logs to Graylog with Fluent Bit: how the pipeline works

Fluent Bit runs as a DaemonSet, so one collector pod sits on every node, reads the container log files that the kubelet writes there, enriches each record with Kubernetes metadata and sends it to a Graylog GELF input. Cluster logs are otherwise scattered across nodes, and a single pod's output disappears when the pod is rescheduled. A node-level collector keeps the collection point stable as workloads move, and Graylog then holds one searchable copy for investigation and alerting. Fluent Bit is an Apache-2.0 licensed log processor from the CNCF graduated Fluentd family, and it is lightweight enough to run on every node without much of the node's capacity. The Kubernetes metadata filter adds fields such as namespace, pod name, container name and labels, which are what make later searches and stream rules useful.

The integration method is the GELF protocol over a Graylog input. Fluent Bit's output plugin sends structured JSON records to a GELF input that your team creates in Graylog, typically over TCP or UDP, and Graylog parses and indexes them into the index sets you configure. The pipeline therefore has four stages you can test separately: file tailing on the node, metadata enrichment in the cluster, transport to the input, and indexing in Graylog. Graylog Open is licensed under SSPL-1.0, which is not an OSI-approved open source license, so confirm with your legal and procurement teams that this license fits your use before you standardize on it. Also confirm the exact Fluent Bit output and filter options against the version you deploy, since the Fluent Bit release line moves quickly.

What you need before deploying Fluent Bit to Graylog

Most delays in this deployment come from versions and network paths that were never checked before the DaemonSet went out. The cards below cover the Graylog stack, the Kubernetes cluster, the Fluent Bit release, access rights and the trust chain, in the order you should confirm them.

Graylog version and backends

Start from the Graylog release you run, not the one you intend to run later. The current documentation covers Graylog 7.2, and the compatibility matrix lists which MongoDB and search backend versions each Graylog line supports. Confirm that your MongoDB replica set and your log storage are inside those ranges. Graylog recommends Data Node as the search backend, and OpenSearch 1.x is deprecated since Graylog 7.0, so a cluster still on 1.x should be planned as a separate migration before the Kubernetes input goes live.

Kubernetes version and support window

Check which Kubernetes minor version your clusters run and whether the project still patches it. The project maintains the three most recent minor releases, so a cluster on an older minor has no upstream fixes. Verify the end-of-life date for your line and plan the cluster upgrade before the DaemonSet work, because the Fluent Bit manifests and RBAC you write will be tested against that cluster. The version skew policy also limits how far kubelets may lag behind the API server during upgrades.

Cluster-admin access for DaemonSet and RBAC

Deploying the log collector needs rights that most application teams do not hold. You will create a DaemonSet that runs on every node, a ServiceAccount, a ClusterRole and a ClusterRoleBinding so the Kubernetes filter can read pod metadata, and a ConfigMap for the pipeline. Agree in advance who approves cluster-scoped objects, and confirm that your Pod Security Standards level allows the host log path mounts the collector needs. Without this agreement, the rollout stalls at the first admission error.

Network path from nodes to Graylog

Every node that runs the DaemonSet must reach the GELF input port on the Graylog nodes or load balancer, and the path must allow the protocol you chose. Check security group or firewall rules between the pod network, node subnets and the Graylog tier, and confirm which source addresses Graylog will see, since pods leave through node addresses in many setups. Test from a node before deploying. A blocked path usually shows up as a growing buffer in Fluent Bit, not as a clear error.

Fluent Bit version to pin

Pin a specific Fluent Bit image tag in the DaemonSet rather than using a floating tag. Fluent Bit 5.1 is the current line in the project's releases, with patch releases on the same line, and an older 5.0 patch is still published for the previous line. Pick one tag, record it in your change record, and test it in a non-production cluster first. Upgrades should follow the release notes for the GELF output and the Kubernetes filter, since changes there affect field names in Graylog searches.

Certificate and trust chain for TLS

If the GELF input uses TLS, the collector must trust the certificate chain that Graylog presents. Confirm who issues the certificate, whether an internal CA bundle must be mounted into the pods, and how the chain is renewed before expiry. A certificate that expires silently stops log delivery, and the collector may keep retrying without a clear alarm. Record the CA bundle source, the expiry date and the owner so the rotation is handled like any other cluster secret.

Configuring Fluent Bit DaemonSet and the Graylog GELF input

This sequence connects Kubernetes nodes to Graylog by creating a GELF input, granting Fluent Bit read access to pod metadata, and running a pinned DaemonSet that tails container logs, enriches them and sends them to Graylog. Each step ends with a check so the team knows records are flowing before adding parsing and routing.

Create a GELF input in Graylog
A Graylog administrator opens the system inputs page, selects GELF (TCP or UDP, according to the transport your network policy allows), and launches the input on a node that the cluster can reach. Record the bind address, the port and the protocol, and set the input to the Graylog nodes you want receiving traffic. Confirm the input shows as running in the input list. Ask your Graylog team which transport they prefer for cluster traffic, since the choice affects delivery guarantees and whether a load balancer sits in front of the input. Confirm the port is open from the node network to Graylog before continuing.
Create the namespace, service account and RBAC
A platform engineer creates a dedicated namespace for the logging agent, a service account for Fluent Bit, and a ClusterRole that allows reading pods and namespaces, plus the matching ClusterRoleBinding. The Kubernetes filter uses these reads to attach pod name, namespace, labels and annotations to each record. Grant only get, list and watch on those resources, and avoid broader verbs. Check the result by running a kubectl auth can-i query for the service account against pods and namespaces, and confirm that secrets and write verbs return no.
Write the ConfigMap with tail, filter and output
The platform engineer writes a ConfigMap that holds the Fluent Bit configuration: a tail input reading container log files under the node's container log path, a kubernetes filter for metadata enrichment, and a gelf output pointing at the input host and port from step one. Set the tail input to a database file on a host path so offsets survive restarts, and keep the Kubernetes filter's API and metadata options as documented for the Fluent Bit version you pin. Validate the file syntax with a dry run in a test pod before rolling it out to the cluster.
Deploy the DaemonSet with a pinned image
The engineer applies a DaemonSet that mounts the ConfigMap, the node container log directories and the offset database path, and runs the service account from step two. The image tag is pinned to a specific Fluent Bit release that the team has tested, not to a floating tag. Confirm the DaemonSet reports a pod on every node that should send logs, including control plane or tainted nodes if they are in scope and tolerations are set. Check the pod status is Running on each node and that the agent logs show no startup or output errors.
Verify records arrive in Graylog search
Deploy a short test workload in a non-production namespace that writes a known string to standard output. In Graylog, open search over the default stream and filter on that string together with the Kubernetes namespace or pod name field added by the filter. The check is complete when the test message appears with the expected metadata fields. If nothing arrives, check the Fluent Bit pod logs for output retries, then confirm the GELF input is running and that the port is reachable from the node. Record the field names your team sees, because later routing rules depend on them.
Add multiline parsing for stack traces
Applications that print stack traces or multi-line exceptions split one event across many lines, so the engineer adds a multiline parser to the tail input for the affected container runtime format and for the application patterns the team has identified. Test against a sample workload that prints a known multi-line exception, then search Graylog for that exception and confirm it appears as one message rather than many fragments. Keep the parser scoped to the namespaces that need it, because broad multiline rules can merge unrelated lines.
Route namespaces to Graylog streams
Graylog streams separate messages by rule. The team adds a field that Fluent Bit sets from the Kubernetes namespace, then creates one stream per namespace group with a rule matching that field, and assigns index sets that fit each group's retention needs. Confirm in Graylog that test messages from each namespace land in the correct stream, and that messages from unmatched namespaces stay in the default stream. Ask Graylog administrators to confirm how stream rules interact with index retention before choosing groups.

How Kubernetes metadata enrichment and multiline parsing shape searchable fields

The Kubernetes metadata enrichment step decides which fields exist in Graylog and how many distinct values each one can take, so it should be designed before the first DaemonSet rolls out. The Fluent Bit kubernetes filter reads the pod's identity from the API server and attaches the namespace, pod name, container name, node name and pod labels to every record. In Graylog these arrive as fields that you will search, group and build streams on, so their names and values matter. Before you go live, decide which labels you keep (for example, a stable app or team label) and which you drop with the filter's exclusion options. Pod names and some labels change with every deployment or rollout, and each unique value adds to the cardinality of the index. Confirm your field naming and label policy with the Fluent Bit filter documentation for the version you deploy, and check how Graylog's index sets handle the resulting field count in your environment.

Multiline handling is where most parsing errors appear. Java, Python and Go stack traces span several lines, and without a multiline parser each line becomes a separate message, so a single exception produces dozens of unrelated records. Fluent Bit's multiline parsers can join those lines, but the start-of-event pattern must match what your applications actually print. JSON application logs need a different approach: a JSON parser can promote the application's own keys into fields, which can collide with the Kubernetes metadata names if both use namespace or container. Decide a prefix or a nesting convention before parsing, because Graylog field types are set when values are first indexed and changing them later usually means reindexing.

Parser choice must match the container runtime log format. The node's container runtime writes each line in a specific layout, such as the CRI format used by containerd, and the CRI parser expects that layout. If the parser does not match, lines are passed through unparsed or dropped, and the failures can look like missing data rather than an error. Confirm the runtime on your nodes and the log path in your cluster, then test each parser against captured samples from your own workloads, including large stack traces, JSON with nested fields, and lines that contain unusual characters. A short sample set run through a local Fluent Bit configuration, with the output compared to what Graylog displays, catches these problems before they reach production and before any stream rules depend on the fields.

Mapping Kubernetes fields to Graylog message fields

This table compares the Kubernetes metadata keys Fluent Bit can attach to each container log, what each one tells an investigator, and where it fits in Graylog stream rules or index design. Key names depend on the Kubernetes filter settings your team chooses, so confirm them against your Fluent Bit output before writing stream rules. Read the last column first, because labels and pod-level identifiers are the usual source of index growth.

Kubernetes metadata	What it identifies	Stream rule or search use	Index load risk
Namespace	Team or environment boundary for the workload	Strong candidate for routing into a per-namespace or per-environment stream	Low, few distinct values in most clusters
Pod name	Individual pod instance, including replica suffixes	Useful for drilling into one failing instance during an incident	High, names change on every rollout and restart
Container name	Container within a pod, such as a sidecar or init container	Good for separating app logs from proxy or sidecar logs	Low to moderate, bounded by the images you run
Node name	Host running the pod	Helpful for finding node-level problems such as disk pressure or kernel errors	Moderate, grows with cluster size
Namespace and workload labels	Team, application and tier labels set by your teams	Preferred for stream rules when labels are enforced by policy	Moderate, depends on how freely teams add label values
Pod UID or container ID	Unique runtime identifier	Rarely needed in searches, use only for exact correlation	High, unique per container, so avoid indexing it as a filterable field
Free-form annotations	Metadata that teams write per object	Generally not suitable for stream rules	High, values are unbounded and often contain long strings
Container image and tag	Software version running in the container	Useful for finding what changed after a deployment	Moderate, tags vary with each build

Routing namespaces to streams and handling back pressure

Fluent Bit assigns a tag to each record when it reads it, and filters and outputs match on those tags. For Kubernetes container logs, the tag usually encodes the pod, namespace and container, so routing decisions can be made before anything reaches Graylog. Graylog then uses stream rules on fields it receives to place messages into streams, which controls who can search them and how long they are kept in the index sets behind each stream.

Route early, match late: Use the tag or a record_modifier or modify filter to add a field such as namespace or environment before the GELF output, so Graylog stream rules have a stable field to match.
Separate by environment first: Production and non-production logs often need different access and retention, so decide that split before splitting by team.
Watch for field drift: Pod labels and namespace names can change. Stream rules that match on a field that a Fluent Bit filter no longer adds will silently send messages to the default stream.
Limit stream count: Every stream adds configuration to maintain. Group namespaces by ownership rather than creating one stream per namespace unless auditors require it.
Confirm field names: Check which Kubernetes metadata keys reach Graylog in your build before writing rules.

Test by generating a known message in each namespace and confirming it lands in the expected stream and nowhere else.

Securing the log path from pods to Graylog

A Fluent Bit DaemonSet reads node log files, sees pod metadata and sends data to a Graylog input, so it touches credentials, host paths and the network at once. The cards below cover the controls to set on each of those surfaces before the pipeline carries production logs.

Scope the Fluent Bit service account

Fluent Bit needs to read pod and namespace metadata from the Kubernetes API to enrich logs. Bind a dedicated ServiceAccount to a ClusterRole that allows only get, list and watch on pods and namespaces, and avoid wildcard verbs or resources. Check with kubectl auth can-i --list that the account cannot read Secrets or write workloads. Keep the binding separate from any application service account.

Keep secrets out of the ConfigMap

Fluent Bit configuration is often stored in a ConfigMap, which any principal with read access can view. Keep Graylog tokens, TLS keys and passwords out of that file and mount them from a Secret instead. Kubernetes stores Secrets unencrypted in etcd by default, so confirm that encryption at rest is enabled and that list and watch access to Secrets is restricted to the people and workloads that need it.

Apply Pod Security Standards to the DaemonSet

A logging DaemonSet usually needs host path mounts such as /var/log and /var/lib/docker/containers and may run as root to read them. Label the namespace with the Pod Security Standards level it can meet, usually baseline with the exceptions documented, and record each exception as a deliberate decision. Drop unneeded Linux capabilities, disable privilege escalation and keep the root filesystem read-only where the image allows it.

Limit host mounts to read-only log paths

Host path volumes give a container access to the node filesystem, so mount only the log directories the tail input reads and mark them read-only. Do not mount the whole host root or container runtime sockets for convenience. Confirm the exact log paths with your container runtime, since they differ between runtimes and distributions, and review the mount list whenever the DaemonSet image or runtime changes.

Restrict the GELF input to cluster ranges

A GELF input in Graylog accepts messages from any host that can reach its port unless the network is restricted. Bind the input to the interface reachable from cluster nodes and limit the allowed sources with firewall rules or network policy to the node and pod CIDRs used by the cluster. Confirm in your Graylog version how the input's listen address and access settings are configured, and test that a host outside the range is refused.

Protect transport between nodes and Graylog

Decide whether log traffic from nodes crosses only a trusted cluster network or a wider network, and encrypt it where it does. Check which transports your chosen Fluent Bit output and Graylog input support, including TLS options, before committing to a design. Where TLS is not available on the path, keep the traffic on a segregated network and record that as an accepted risk for the reviewing team.

Common failures when Kubernetes logs stop reaching Graylog

This table lists the failures teams most often hit when Fluent Bit logs from a cluster do not reach Graylog, with the likely cause, the check to run and the fix for each. Work through the rows in order, starting with the Fluent Bit pods, because most symptoms trace back to the agent before they reach the Graylog input.

Symptom	Likely cause and check	Fix
No records from some nodes	The DaemonSet pod on those nodes is not scheduled, is pending, or carries taints that its tolerations do not match. Check with kubectl get pods -o wide and compare the node list against the DaemonSet.	Add the tolerations the node taints require, or confirm the nodes are in scope. Redeploy the DaemonSet and confirm one pod per node.
Duplicate or split multiline events	Stack traces or JSON spanning several lines are parsed as separate records, or the multiline parser is applied twice (once in the container runtime format and again in Fluent Bit). Compare a sample event in Graylog with the raw container log.	Use one multiline parser on the tail input, matched to the application's first-line pattern. Remove duplicate parsers and test with a known multi-line sample before rollout.
Missing Kubernetes metadata	The Fluent Bit service account cannot read pods or namespaces from the API server, so the kubernetes filter cannot add labels. Check the filter logs for forbidden responses and test with kubectl auth can-i against the service account.	Bind a ClusterRole that grants get, list and watch on pods and namespaces, then restart the DaemonSet. Keep the grant read-only and scoped to what the filter needs.
GELF input receives no traffic	The Graylog GELF input listens on a different port or protocol than the output in Fluent Bit, or a NetworkPolicy blocks traffic from the logging namespace to the Graylog nodes. Check the input status in Graylog, then test connectivity from a pod in the logging namespace.	Align host, port and protocol (UDP or TCP) on both sides. Add an egress rule for the logging namespace and an ingress rule on the Graylog side, and confirm the rule with a test message.
Back pressure drops records or buffers grow	Graylog or its journal cannot keep up, the output is slow, or Fluent Bit's memory buffer fills and it discards chunks. Check Fluent Bit retry and error counters, the Graylog journal and the output queue depth.	Enable filesystem buffering with a bounded size, tune retries, and size the Graylog journal and nodes for peak ingest. Confirm the fix by watching buffer size and dropped-record counts under load.
Records appear in the wrong stream	Namespace routing rules in Graylog do not match the field names that the Kubernetes filter adds, or the field is named differently after a filter change. Check a sample message for the namespace field and the stream rule conditions.	Match stream rules to the exact field name in a sample message. Retest after any filter or Fluent Bit version change, and document the field names the rules rely on.

Questions teams ask before connecting Kubernetes logs to Graylog

What drives the effort and cost of connecting Kubernetes logs to Graylog?

The main drivers are the number of clusters, the number of namespaces and workloads that produce logs, how much of that output is noisy or needs parsing, and how many environments must be covered. Each cluster adds a Fluent Bit DaemonSet, RBAC for Kubernetes metadata lookups, network paths to Graylog inputs and a routing design. Parsing work grows with the number of distinct log formats, especially multiline stack traces. Costs split into one-time design and build work, paid for in a fixed-scope engagement or by your own engineers, and recurring work such as Fluent Bit upgrades and pipeline changes. Ask any quote to itemize clusters, log formats, environments, and whether testing and handover documentation are included.

How long does a Kubernetes to Graylog logging integration take?

Duration depends mostly on cluster count, log volume and how many formats need parsing, not on the Fluent Bit installation itself, which is quick on a single cluster. A single cluster with a few well-understood applications is a much smaller job than a multi-cluster estate with several teams, each needing its own streams and retention. Storage and Graylog sizing decisions can also add time if the ingest rate is unknown. Confirm the schedule against the written scope, and ask which steps depend on other teams, such as network approvals or Graylog administration access.

What does our internal team need to provide before the work starts?

Three groups of access and decisions are needed before work starts:

Cluster access: a kubeconfig or equivalent with rights to create a namespace, a DaemonSet, a ServiceAccount and the RBAC that lets Fluent Bit read pod metadata, plus a named contact for each cluster.
Graylog access: an administrator who can create GELF inputs, streams, index sets and routing pipelines, and who can confirm the target Graylog version.
Network approvals: firewall rules from cluster nodes to the Graylog input port, and any TLS certificate or trust chain the input requires.

Also confirm log retention requirements, data classification and who approves stream names.

Can our team set up Kubernetes log shipping to Graylog without outside help?

Yes, a team with Kubernetes and Graylog experience can build a basic pipeline in-house, since Fluent Bit is open source under Apache-2.0 and Graylog accepts GELF input. The difficult parts are rarely the first DaemonSet. They are multiline handling for stack traces, consistent Kubernetes metadata fields, back pressure behavior when Graylog slows down, and stream routing that stays correct as namespaces change. Those tasks benefit from someone who has built them before. An outside partner is most useful for a fixed-scope design, a repeatable deployment across several clusters, and documented operations your team can keep running.

How does Graylog licensing affect this decision?

Graylog Open is licensed under SSPL-1.0, which is not an OSI-approved open source license, so confirm that this license suits your legal and procurement policy before building on it. Graylog Open supports GELF input and streams, which is enough for the pipeline described here. Graylog's pricing page says Graylog Open is free with no volume limits and community support, while Graylog Enterprise starts at 15,000 US dollars per year and Graylog Security at 18,000 US dollars per year, each with a minimum of 10 GB per day or 100 Graylog Compute Units. Enterprise and Security also add 24/7 support and other features. Confirm with Graylog which features your design needs before you commit to an edition.

Does Fluent Bit add risk to a production Kubernetes cluster?

Fluent Bit runs on every node as a DaemonSet, so its resource use, its configuration and its upgrade path affect every workload on the node. Set memory and CPU limits, test configuration changes in a non-production cluster, and watch for a growing backlog when Graylog is unavailable. Fluent Bit is written in C and documented as lightweight, but the actual footprint depends on your plugins and volume. Kubernetes itself moves quickly: its three most recent minor releases are maintained, so plan Fluent Bit and cluster upgrades together, and confirm compatibility with your Kubernetes version before each change.

Which security controls should we check before shipping container logs?

Logs often contain tokens, personal data and internal hostnames, so check what leaves the cluster before it reaches Graylog. Restrict who can read the Fluent Bit configuration, since it may hold Graylog endpoints or credentials, and store those values as Kubernetes Secrets. Note that Kubernetes Secrets are stored unencrypted in etcd by default, so confirm encryption at rest is configured. Apply least-privilege RBAC to the Fluent Bit ServiceAccount, decide whether transport to Graylog uses TLS, and restrict which Graylog users can search each stream. Document these choices for your security review.

Keeping the pipeline healthy after go-live

The ongoing work starts with upgrade planning tied to Kubernetes version skew. The Kubernetes version skew policy allows kubelets to be up to three minor versions older than kube-apiserver, so a Fluent Bit DaemonSet can keep running through a control plane upgrade, but it cannot stay put indefinitely. Each cluster upgrade should trigger a check of the Fluent Bit image against the Kubernetes versions still in use, and the Fluent Bit release notes should be reviewed before the image tag changes. Confirm the current Fluent Bit line and its support window on the Fluent Bit GitHub releases page before scheduling the change, and pin the image to a specific tag rather than a floating one.

Monitoring has two halves. On the node side, watch Fluent Bit's own health: restarts on each DaemonSet pod, error and retry counts in its logs, and whether buffer or filesystem storage is filling on hosts. On the Graylog side, watch the message journal, the input throughput and the processing backlog for the GELF input receiving Fluent Bit traffic. A growing journal means Graylog is accepting data faster than it can index it, and the cause is usually index capacity, not the agent. Use Graylog's own system pages to track journal utilization over time, and set an alert on sustained growth so the team hears about it before the node disk fills.

Index growth is the cost that rises quietly. Every new namespace routed to its own stream adds volume, and the index model applies retention by rotating indices and deleting the oldest ones, so review the size of each index set against the sizing formula in Graylog's system architecture guidance (daily ingest times retention days times 120 percent) each time a namespace is added. Test every change to parsing, labels, multiline rules or routing in a non-production cluster that mirrors the production DaemonSet and GELF input, and confirm that messages reach the intended stream before the change is promoted. Keep the Fluent Bit configuration in version control so every rollout can be reversed.

How DeployOpen scopes and delivers Kubernetes log integrations

DeployOpen plans, builds, documents and supports Kubernetes container log collection into Graylog with Fluent Bit inside infrastructure you control. Each engagement starts from a written scope covering your clusters, namespaces, Graylog backend and routing rules, so the delivered pipeline matches what your teams need to investigate events.

How we scope

We confirm which clusters, node pools and namespaces send logs and which of them need separate streams.
We record the Graylog version, Data Node or OpenSearch backend and the GELF input endpoints the collectors will reach.
We agree in writing how multiline, metadata enrichment and back pressure handling will be tested before build starts.

How we build

We deploy the Fluent Bit DaemonSet with pinned versions, resource limits and tolerations for each node pool.
We configure Kubernetes metadata enrichment, GELF output and namespace-to-stream routing in version-controlled configuration.
We test back pressure behavior, multiline parsing and log loss under node restarts before handing the pipeline over.

How we hand over

You receive the manifests, Fluent Bit configuration, Graylog input and stream definitions, and a runbook for each.
Documentation covers how to add a namespace, change a parser and verify that logs arrive in the expected stream.
We walk your platform and security teams through troubleshooting steps and the checks to run after a cluster upgrade.

How we support

Ongoing support, upgrades of Fluent Bit and Graylog-side changes are available as a retainer under a separate written agreement.
Security testing of the collection path, including the Kubernetes RBAC and Secrets handling, can be scoped as a separate engagement.
Any change to the written scope, such as new clusters or a different Graylog edition, is agreed in writing before work begins.

Kubernetes container logs to Graylog with Fluent Bit