Kubernetes on OpenStack with cloud-provider-openstack

The first design decision is which component owns which resource. The OpenStack Cloud Controller Manager runs inside the cluster and calls OpenStack APIs to keep Kubernetes objects aligned with cloud resources. It reads server records from Nova so that nodes receive labels and addresses, consults Neutron for network addresses, and asks Octavia to create load balancers when a Service of type LoadBalancer is created. Volumes and shared file systems are a separate concern. The Cinder CSI driver provisions block volumes and the Manila CSI driver provisions shares, and both are reached through StorageClasses that your platform team defines. Keeping these responsibilities apart lets you review the permissions of each component on its own terms.

The controller needs its own OpenStack identity. Configure it with a dedicated Keystone credential rather than a personal user account, so that its actions are attributable and the credential can be rotated without affecting people. Application credentials are commonly used for this purpose, but confirm the current recommendation for your release in the cloud-provider-openstack documentation. Scope the credential to the project that holds the cluster and grant only the roles the controller and CSI drivers need. Store it in a Kubernetes Secret with tightly restricted read access, because Secrets are stored unencrypted in etcd by default unless you configure encryption at rest.

Controller authentication to Keystone is a machine-to-cloud relationship, and it is separate from how people reach the Kubernetes API server. Kubernetes does not represent normal users as API objects. It authenticates them with X.509 client certificates, bearer tokens such as OpenID Connect and JWT, webhook token authentication or an authenticating proxy. Keystone is not the default identity source for engineers running kubectl. Decide how staff authenticate to the API server, typically through your corporate directory using OIDC, and confirm whether Keystone federation will serve only the cloud-side identity or also the cluster front door. Documenting both paths separately makes access reviews easier to run and explain.

The storage backend is a separate decision that affects both drivers. Ceph block devices can back Glance images, Cinder volumes and backups, and Nova guest disks, but whether your cloud uses Ceph, another Cinder backend or a mix is a choice to confirm with your OpenStack design, not a requirement of the CSI drivers. Ask which backend each Cinder volume type maps to, because StorageClass parameters point at those types. Check the Manila share backend too, since it determines whether ReadWriteMany claims can be supported. Agree on capacity and failure domains with the storage team before you define StorageClasses that application teams will depend on.

Kubernetes stores Secrets unencrypted in etcd by default, so the first decision is whether your cluster will encrypt them and where the key will live. The Kubernetes good practices guidance asks administrators to do three things: configure encryption at rest, restrict list and watch access with least-privilege RBAC, or keep sensitive values in an external store reached through the Secrets Store CSI Driver. These controls cover different risks. Encryption at rest protects etcd data and its backups, RBAC limits who can read Secrets through the API, and an external store keeps some values out of etcd entirely. Most clusters on OpenStack need more than one of these. Decide which ones your team will run before any workload that uses Secrets is migrated.

Encryption at rest is configured on kube-apiserver through an encryption configuration file that lists providers for each resource type. The first provider in the list encrypts new writes, and the others allow existing data to be read during a change. Local key providers keep the key material on control plane nodes, so anyone who can read that file and the etcd data can decrypt Secrets. A KMS provider moves the key out of the API server and calls an external key service. The Barbican-backed option for cloud-provider-openstack is a design choice, not a default. Confirm it against your exact Kubernetes and cloud-provider-openstack releases, including which provider is supported, what Keystone credentials the API server needs, and how the key lifecycle is handled. Get that version matrix in writing before you commit to the design.

Key rotation and key ownership are where this design usually breaks down. Rotating a local key means adding a new key entry, restarting the API servers, and then rewriting existing Secrets so they are encrypted with the new key before the old one is removed. Skipping the rewrite leaves data protected by a key you have retired, and removing the old key too early makes that data unreadable. Name an owner for each key: the platform team that runs the control plane, a security team that controls the key service, or the OpenStack operators who administer Barbican. Document the owner alongside the rotation procedure, the interval your policy requires, and the approval path for emergency rotation. Ask the project whether Barbican rotation can be automated and how it audits key use.

The most serious risk is a backup that cannot be restored. An etcd snapshot taken with etcdctl snapshot save contains encrypted Secret values, and restoring it with etcdutl into a cluster whose encryption key is missing or different leaves those Secrets unreadable. The snapshot can be intact while the cluster is unusable for every workload that depends on those values. Store encryption keys as a separate backup with separate access control, outside the failure domain of the etcd backups. Record which key version each snapshot requires so a restore after a rotation does not depend on memory. Then test a restore into a non-production cluster that uses the same key, and include that check in the restore runbook before go-live.

The integration is not ready for production until a written verification plan exists and every test in it has an acceptance criterion agreed before the build starts. Write the pass and fail conditions first, then ask the build team to show evidence against them. Otherwise, the team tends to define success as whatever the cluster happens to do during the demo. The plan should cover the OpenStack Cloud Controller Manager, each CSI driver in use, the load balancer path, Keystone authentication, the kubeadm control plane and etcd. Assign an owner to each test, name the environment it runs in, and keep the results with the change record so an auditor or a later operator can see what was proven and when.

Start with the node and storage path, because failures there surface everywhere else. After the controller starts, confirm that every node registers with the expected name, provider ID, and zone and region labels, and that each node maps to a Nova instance in the right project. Then provision a PersistentVolumeClaim through the Cinder CSI driver and another through the Manila CSI driver, attach or mount each to a pod, and write a marker file. Delete the pod so the scheduler places it on a different node. The acceptance criterion is that the same data reads back after the reschedule and that the volume detaches cleanly from the old node. Repeat the test with the storage class parameters your platform team plans to use in production.

Test ingress from outside the cluster with a LoadBalancer Service and an Ingress backed by Octavia. Record the external address, the listener and the health monitor status in OpenStack. A successful HTTP response from a host on the corporate network is the minimum, and the test should also confirm that security group rules permit only the ports you planned. Next, rotate the Keystone credential that the controller and CSI drivers use, following your organization's rotation procedure. Confirm that workloads keep running, that the old credential is rejected, and that the rotation is recorded. Ask your identity team beforehand which credential type the integration uses, because the rotation steps depend on it.

Fail a kubeadm control plane node and verify that the TCP load balancer on port 6443 keeps serving the API, that the etcd members elect a leader if you use the stacked topology, and that kubectl, the controller manager and the scheduler recover. The kubeadm high availability guide calls for at least three control plane nodes, so run the failover test at that count. Finally, rehearse an etcd snapshot save with etcdctl snapshot save, then restore it into a scratch environment with etcdutl. Time the procedure and record every manual step, because the restore runbook is the document operators will reach for during an incident. Store the rehearsal results with the acceptance criteria.