Zabbix Kubernetes monitoring with the Helm chart

The Helm chart places a Zabbix proxy inside the cluster so that collection traffic stays local, and only the proxy's outbound connection crosses to the Zabbix server. The proxy authenticates to the Kubernetes API with the monitoring token from the chart, queries the API for node, pod and workload objects, and polls the kubelets for node-level metrics. It stores what it collects and forwards that data to the server on its own schedule. For the reader, the practical point is where failures appear: an API token problem or a kubelet that cannot be reached shows up in the proxy's logs and in host availability, while a broken path to the server shows up as delayed data. Check each of these paths separately during deployment.

A single proxy is the simpler design. It has one deployment to version, one set of logs to watch and one place where cluster host definitions live. Its weakness is that it is also the only collection point for that cluster, so its outage stops all Kubernetes data until it recovers. Several proxies spread polling load and split failure domains, which helps when clusters are large or span zones, but each proxy adds a component to upgrade, monitor and keep on a matching Zabbix version. The right count depends on cluster size, the number of hosts and items the templates create, and how long your team accepts a gap in monitoring data. Ask the team that will run the platform to write those thresholds down before choosing.

From Zabbix 7.0 onward, proxy groups let Zabbix redistribute monitored hosts between proxies in the group for load balancing and high availability when one proxy goes offline. This is the feature that makes a group of in-cluster proxies more than a set of independent collectors. The protection depends on the group design. Proxies that share one node, one zone or one API endpoint may fail together, and a proxy with no spare capacity cannot absorb another proxy's hosts without falling behind. Confirm the exact redistribution timing, the requirements for reassignment and the limits that apply to your version against the Zabbix proxy load balancing and high availability documentation before you rely on it in a design.

Test the design before go-live. Stop one proxy pod on purpose, confirm that the cluster's hosts move to another proxy in the group, and check that node, kubelet and API items resume without unexplained gaps in history. Repeat the test with the outbound path to the Zabbix server blocked, since data then queues in the proxy and the dashboards show delay rather than an obvious error. Also rotate or expire the monitoring token in a test cluster to confirm that the alert reaches your team. Keep proxy high availability separate from server high availability. The Zabbix server cluster uses a shared database and a configurable failover delay of 10 seconds to 15 minutes, and that setting does not change how a proxy group behaves.

Unfiltered discovery on a large cluster turns every namespace, workload and node object the Kubernetes API exposes into a candidate Zabbix host, with items and triggers attached to each one. That multiplies configuration, polling and stored data before anyone has decided what matters. Discovery filters narrow this at the point of creation. You can limit scope by namespace, by label selectors on workloads or by object type, so that only a team's workloads, or only node and control plane objects, enter the inventory. Confirm in the chart version you deploy which discovery rules exist, which filter fields each rule accepts, and whether a filter acts on the discovered object or on the host it generates. Most over-collection starts with filter logic, so treat the filters as a design artifact that someone owns, not as a default setting left in place.

The central trade-off is visibility against capacity. Broad discovery gives platform teams a view across every namespace, which helps when a fault crosses team boundaries. The cost falls on the in-cluster proxy, the Zabbix server, the database and the notification path. Each additional host and item adds collection work, and a noisy namespace can push alert volume into the same queues that carry alerts for critical nodes. Proxy groups, available from Zabbix 7.0, can redistribute monitored hosts between proxies and keep monitoring running when one proxy goes offline, but they do not decide what is worth collecting. Sizing must be validated in your own environment with your own object counts, polling intervals and retention needs. Do not plan capacity from any general figure, including those in this section.

Roll discovery out in stages. In the first stage, enable discovery for one non-production namespace or one team's labels. Observe the number of discovered hosts, the item counts, the proxy queue and server performance across several ordinary days, including at least one deployment window. In the second stage, review the filter output with the owners of those workloads. Ask which discovered objects they would act on, which would only generate noise and which namespaces should be excluded entirely. In the third stage, widen the scope one group at a time and repeat the review before each step. Record every filter change in version control with its reason, so that later questions about why a host exists can be answered from the history.

Before enabling full discovery, complete a filter review against a short checklist. Confirm that the Zabbix API token's Kubernetes-side permissions match what the filters expect, so discovery sees the intended objects and no more. Check that excluded objects do not remove signals that the node, control plane and kubelet templates depend on, because a filter that hides a control plane component can silence the very health data you need. Agree who owns the resulting host list and who approves changes. Plan for drift, since namespaces created after rollout will match or miss the filters without anyone changing them, so schedule a recurring review. Also ask the implementation team to show how a filter change applies to hosts that discovery has already created, and confirm that behavior in your Zabbix version before you rely on it.

Run these checks in order before handover, because each step depends on the one before it. Start with proxy connectivity. Confirm that the in-cluster Zabbix proxy can reach the Zabbix server or the upstream proxy it reports to, and that its host record shows as online with recent data. Then test API token reads. Run the queries that the node, control plane and kubelet templates depend on, using the token itself and not an administrator kubeconfig. Confirm that the token can read the namespaces, nodes and endpoints in scope and that writes and Secret reads are refused. A token that works only because it is over-privileged will pass this step and fail the access review later.

Next, compare discovery results with the expected inventory. Before the test, write down the nodes, control plane components, kubelets and namespaces that should appear, and the filters that should include or exclude them. Missing hosts usually point to an expression error or an RBAC gap. Unexpected hosts usually point to a filter that is too broad, which matters in large clusters because every extra host adds items and load. Then check item collection on nodes and control plane components. Items should carry recent values, and any unsupported items should be recorded with their error text so the team can see whether the cause is the token, the network path or the chart configuration.

Trigger behavior comes after collection is stable. In a non-production cluster, induce a test condition such as scaling a test workload to zero, draining a test node or stopping a control plane component on a disposable cluster. Confirm that the problem opens, that its severity matches the agreed mapping, that it suppresses duplicates as designed and that it closes when the condition clears. Confirm that notifications reach the intended media type and recipients. Record the time between the induced condition and the problem, and agree with the operations team which delay is acceptable for each severity. Do not run induced failures against production clusters.

If you use a proxy group, restart one proxy in the test environment while data is flowing. Zabbix documents that proxy groups redistribute monitored hosts when a proxy goes offline, so confirm that the hosts move, that data keeps arriving and that no template items stop for longer than the team accepts. If the Zabbix server runs as a high availability cluster, test failover separately. Its failover delay is a server setting that can be configured from 10 seconds to 15 minutes, so confirm the value with your team and verify that the standby node takes over. Finally, run a Kubernetes upgrade check. The version skew policy requires kube-apiserver to be upgraded one minor version at a time, ahead of the controller manager, scheduler and kubelets, and allows kubelets to be up to three minor versions older. Upgrade the test cluster one minor version, then repeat the token, discovery and item checks. Confirm the kubelet lag you plan to allow in production, because monitoring must keep working during that window.