Kubernetes audit logging

Start the policy with the resources that answer access and privilege questions, because those are the events an investigation needs first. Put RBAC objects (Roles, ClusterRoles, RoleBindings and ClusterRoleBindings) at the top, followed by ServiceAccounts, authentication-related requests such as TokenReview and certificate signing requests, and workloads that can run privileged or reach host namespaces. For these resources, a Request level records who made the change and what they asked for, which is usually the detail a reviewer needs when a binding grants more access than expected. Before writing any rule, ask your security and compliance leads which privilege questions they must be able to answer from the logs, then map each question to a rule.

Broad read traffic is the main source of audit volume, so give it a lighter rule. Requests such as get, list and watch against common resources can produce large numbers of events from controllers and monitoring agents, and most of those events add little to an access investigation. A Metadata level records who made the request, when, against which resource and with which verb, without storing the body. Use it as a catch-all for reads once the privilege rules are in place. Confirm the expected volume on your own cluster before you commit, because the controllers and agents in your environment determine how many events you receive.

Secrets need the most care. Kubernetes stores Secrets unencrypted in etcd by default, and a request or response body for a Secret contains the credential itself. Write a rule that records Secrets at the Metadata level, so the log shows who read or changed a Secret without storing its value. Do not use Request or RequestResponse for Secrets unless a documented need exists and your log store is restricted to the same people who can read the Secrets. Encryption at rest and least-privilege RBAC for list and watch access reduce exposure in etcd and through the API, but they do not stop credentials from being copied into audit logs, so the audit policy has to handle that separately.

Rule order decides what happens to each event. The API server applies the first rule that matches an event, so a broad read rule placed above a Secrets rule will override the Secrets behavior you designed. Put the most specific rules first, end with a catch-all, and review the resource, verb and group fields each rule matches. Test the policy against the API server version you run, and check field names and behavior against the documentation for that version, because they can change between releases. Then generate test requests for each rule, such as reading a Secret, creating a RoleBinding and listing pods, and confirm that each one is recorded at the level you intended.

The first decision is how audit events leave the control plane. With the log backend, the API server writes events to a file on each control plane node, and a collector agent on the same node reads that file and forwards it to the SIEM. This keeps the request path of the API server simple and lets the collector handle transport, compression and retries. The cost is that the file becomes a second buffer on control plane disk. If the collector stops or falls behind, log rotation can delete events before they are shipped, and a full disk on a control plane node can affect the API server itself. Confirm the rotation settings for your Kubernetes version, size the disk for a collector outage, and alert on gaps in event delivery, not only on whether the agent process is running.

The webhook backend sends batches of events from the API server to an external HTTP endpoint, usually a log receiver or a SIEM ingest service. No file or collector is needed on the nodes, and forwarding does not depend on local disk. The trade-off moves into the API server. Delivery behavior depends on the buffering and retry settings that the webhook configuration exposes, and the availability and latency of the endpoint become part of control plane behavior. If the endpoint stalls and buffers fill, you need to know in advance whether events are dropped, retried or allowed to slow requests. Confirm the failure modes for your version in the Kubernetes documentation, then test them in a non-production cluster by blocking the endpoint and measuring what happens to both the events and API latency.

Parsing effort matters more than the choice of transport suggests. Neither backend hands the SIEM a parsed Kubernetes model, so every pipeline needs field mapping from the audit event structure, including the user, verb, object reference, response status and source addresses, into the schema the SIEM expects. Do not assume any product ships a Kubernetes audit parser. Confirm what each candidate provides, then write tests for the events that carry the most risk, such as reads of Secrets and changes to RBAC bindings. Wazuh is distributed under the GNU General Public License version 2, with an exception that permits linking with the OpenSSL library, so have legal review that exception against your own obligations. Its indexer stores alerts, and the Wazuh documentation should be checked for the supported path to ingest Kubernetes audit events before you design around a particular collector or file location.

OpenSearch, Graylog and Grafana Loki are other common choices for storing and searching these events, and the right one depends on the tooling your team already runs. If you operate a search cluster, dashboards or label-based log storage today, the audit pipeline should feed the tools your analysts already use. Storage sizing follows event volume, which depends on the audit policy you write, so measure volume in a staging cluster before you commit to capacity. Ask each candidate how it handles a burst of events when ingestion slows, what happens to events it has already accepted, and how access controls apply to audit data. Set audit retention as a separate decision from the SIEM choice, because the required period is set by your compliance and investigation obligations.

The audit policy file and the kube-apiserver audit flags must be identical on every control plane node. The kubeadm high availability guide describes a stacked topology, where etcd runs on the control plane nodes, or an external etcd topology, with at least three control plane nodes behind a TCP load balancer on port 6443. Each API request lands on whichever kube-apiserver the load balancer selects, so a node with a different policy or a missing flag records different events for the same action. Investigators then find gaps that depend on the node that handled the request, and a detection rule written against one node can miss activity handled by another. Keep the audit policy and flags in version control, apply them to all nodes from the same source, and after each change confirm that every node's kube-apiserver static pod manifest carries the same audit flags and mounts the same policy file.

The Kubernetes version skew policy requires kube-apiserver to be upgraded one minor version at a time, and before kube-controller-manager, kube-scheduler and kubelets. Because kube-apiserver moves first, it is the component where audit behavior changes earliest in an upgrade. Policy rules that name specific resources, API groups or stages can behave differently on the target minor version, so check the release notes for that version before the upgrade. Then run the existing policy against a test cluster on the target version, confirm that kube-apiserver starts cleanly with the flags, and verify that audit events still appear for the resources your detections depend on. Record this check in the change record for every minor step, not only in the initial setup.

The project maintains the three most recent minor releases. As of September 2026 these were 1.37 (end of life October 28, 2027), 1.36 (June 28, 2027) and 1.35 (February 28, 2027), according to the Kubernetes releases page. A cluster that falls outside this window stops receiving upstream fixes, so plan audit configuration into every upgrade window scheduled before that date. Clusters still on 1.35 have the earliest deadline. Add the audit validation step to the upgrade runbook with a named owner, and if a cluster is several minor versions behind, repeat the check at each intermediate step rather than only at the final target. Confirm with your distribution vendor whether its support dates match the upstream dates, since a vendor may ship and support builds on a different schedule.

Roll the change out one control plane node at a time, and keep a rollback path for each node. Before editing, copy the current audit policy file and the kube-apiserver static pod manifest on that node to a dated location outside the manifests directory. Apply the change to one node, wait for its kube-apiserver to restart, and generate a known test event, such as creating and deleting a labeled ConfigMap in a non-production namespace. Confirm that the event appears in the audit log or webhook backend at the expected level before moving to the next node. If an API server fails to start or expected events are missing, restore the saved manifest and policy file on that node. The load balancer continues to route to the remaining nodes while you do this. Ask your team to document and test these rollback steps on a non-production cluster before the first production change.