Kubernetes observability with OpenTelemetry

The placement decision comes down to which signals need to be collected close to the node and which need to pass through a central point before they leave the cluster. A daemonset Collector runs one pod on every node, reads node-local data such as container log files and kubelet metrics, and adds Kubernetes metadata (namespace, pod, deployment and node names) while the telemetry is still tied to its source. A gateway Collector runs as a central Deployment behind a Kubernetes Service. It receives OTLP from applications, from daemonset agents and from other clusters, applies batching and sampling, and exports to the backends. The two roles are different, and most clusters need both.

The usual combined pattern sends node-local signals through the daemonset and application traces and metrics straight to a gateway. Daemonset pods read logs from the node and collect kubelet and host metrics, then forward them over OTLP to a gateway. Applications instrumented with OpenTelemetry SDKs export OTLP to the gateway, or to the daemonset on the same node if you want the node-local hop. Each daemonset pod needs CPU and memory on every node, including nodes running system or batch workloads, so its requests and limits should be set from measured usage on your busiest nodes, not from defaults. The gateway's resource use scales with total throughput, so it needs its own sizing, horizontal scaling and a memory limiter processor.

The trade-offs fall in three areas. Node resource use grows with node count for daemonsets and with telemetry volume for gateways. Failure blast radius is local for a daemonset, since a crashed agent affects one node's logs and metrics, while a gateway outage can stall telemetry from the whole cluster unless its senders have retry and queue settings in place. Scaling is also different: gateways scale horizontally behind a load balancer, but tail-based sampling needs all spans of a trace to reach the same gateway instance, which affects how you route traffic. Decide on the failure mode you can tolerate for each signal before you pick replica counts.

Use a few questions to decide where each signal goes. Does the data exist only on the node, as container logs and kubelet stats do? If so, it needs a daemonset. Does it need sampling, filtering across the whole cluster, or fan-out to several backends? Then it should pass through a gateway. Is the data already in OTLP from an instrumented application? It can go straight to the gateway. Before you choose any receiver or processor, confirm its current component list and stability level in the component README in the OpenTelemetry Collector repository, because stability differs across components and the Collector distribution you pin determines which ones are available.

Auto-instrumentation moves much of the telemetry work from application code into the cluster. The OpenTelemetry Operator can inject language agents into pods when a workload carries the right annotation, so the Operator's admission webhook adds the agent when the pod is created. The agent depends on the application's language and runtime, so a Java service, a Node.js service and a Python worker each need their own agent. One shared configuration cannot serve all of them without a clear mapping from namespace or workload to language. Before enabling injection, inventory the languages and runtime versions in each namespace, then confirm in the Operator documentation which languages it supports and which runtime versions each agent covers.

Compared with manual SDK instrumentation, auto-instrumentation trades developer effort for operational dependencies. Injection happens at pod creation, so enabling, changing or removing an agent takes effect only when workloads roll out, and a configuration change can restart many pods at once. The agent also loads before the application starts, which adds work to each pod's startup and can slow readiness during scaling or failover. Measure startup times in staging with and without injection. Injected components usually add an init container or a mounted volume, which changes the pod spec but not the application image. Check that these additions comply with the Pod Security Admission level set on each namespace, because a restricted namespace may reject them. Manual SDKs give finer control over span names and attributes, but developers must change and release code to get it.

Agents must be versioned together with the application runtime. An agent built for one runtime line may behave differently on another, and an agent upgrade can change span names, attributes or default behavior that dashboards and alerts depend on. Decide who owns each version. In most setups the platform team pins the agent version in the Instrumentation resource, while application teams own runtime upgrades. Ask the Operator project which agent versions it bundles, and test a runtime upgrade in a non-production namespace before the platform default changes. Record the agent version each workload runs, ideally in the same deployment inventory that tracks runtime versions, so an incident can be traced to a specific agent change.

Maturity differs by signal and by language. Trace APIs and SDKs are stable in most languages, including C++, .NET, Go, Java, JavaScript, PHP, Python, Ruby and Swift, which makes traces the lowest-risk starting point for application telemetry. Log SDK maturity varies more, and several languages are still in development. Before relying on logs from applications through OpenTelemetry, check the status of each runtime in use on the OpenTelemetry status page, and confirm the support level for the specific signal. Where a language's log support is not mature enough for your standards, keep the existing logging path for that workload and forward those logs through the Collector while the gap closes.

Every OpenTelemetry pipeline should pass a written test plan in a non-production cluster before any application sends it production data. Run the plan separately for each pipeline: the node-level agent Collectors deployed as a daemonset, the gateway Collectors, and each path that carries traces, metrics and logs. Use synthetic telemetry with known identifiers so every test has an expected result. For each signal, record the number of items sent, the number received at the backend, and the attributes present on arrival. Agree the pass criteria with the platform and application owners before testing starts, so the results cannot be reinterpreted after the fact.

Kubernetes attribute enrichment is the check most likely to fail silently. Send synthetic spans and logs from a test workload, then confirm the k8sattributes processor adds the pod, namespace, workload and node fields your design specifies. Repeat the check after deleting the test pod so its replacement starts with a new name, and again after a rolling update of its Deployment. The Collector's service account needs read access to the pod and namespace metadata it looks up, so confirm the RBAC role is bound as designed and that the association rules match the labels your teams use in dashboards. Missing attributes often surface weeks later as blank panels, so treat any gap as a blocking defect.

Test the exporter path separately from the receivers. Confirm the exporter authenticates to the backend with the method in the design, whether that is an authenticator extension such as oauth2client or another mechanism your backend requires. Then rotate or expire the test credential and check that the Collector reports the failure in its logs and metrics instead of losing data quietly. Next, restart a gateway Collector while synthetic load is running and measure how much data is lost. If the design relies on persistent queues through a storage extension, verify that the storage volume is mounted and that queued data replays after the restart. Write down the loss figure the team accepts, because the result only means something against an agreed number.

Backpressure testing shows what the pipeline does when a backend slows down or stops. Point an exporter at a deliberately slow or unreachable endpoint, then watch queue occupancy, retry behavior and the exporter failure counters in the Collector's own metrics. Confirm that the memory_limiter processor refuses data before the pod reaches its memory limit, and that every dropped item is counted. Then run the expected volume for a sustained period and record CPU and memory for each Collector pod. Use those peaks, with headroom agreed with the platform team, to set requests and limits. Check the exact metric names against the Collector version you deploy, since they can change between releases.

Compare the sampling and filtering rules in the running configuration with the design document, line by line. This covers tail sampling decisions, filters that drop health checks or noisy logs, and any attribute redaction. Send traces that should be kept and traces that should be dropped, and confirm the outcome for each. Confirm also that the Collector's own telemetry settings are enabled and exported, so the pipeline reports its health to your monitoring. Finish with a sign-off record that lists the Collector and contrib versions, the configuration revision, test results, accepted loss figures and open defects. The security team should review this record, including exposed receiver ports and how credentials are handled, before production approval.