Kubernetes deprecated API migration before upgrades

Build the inventory of deprecated API usage in a fixed order, and finish it before you choose an upgrade path. Start with the deprecation guide and the release notes for each target minor version, since these name the APIs being removed, the version where each removal takes effect, and the replacement API. Take the target versions from your upgrade plan and confirm them on kubernetes.io/releases. Kubernetes maintains only the three most recent minor releases, so a cluster several minors behind may cross more removals than the single step you planned for. Record each removed API in a working list with its target version and the replacement group and version, so every later scan checks against the same list.

Next, run kubectl convert against exported manifests where your kubectl version supports it. Export from Git first, and render Helm charts with helm template so you scan the output the cluster would receive, not the template source. Before relying on the command, confirm in your own environment that kubectl convert is available, check its flags with the help output for your kubectl version, and test it on a sample file you already know uses an old API. Treat its output as a strong signal, not a complete answer. It can only convert manifests it understands, so files it skips or rejects still need a manual review against your working list.

Then check what is running, not only what is stored in Git. Query the API server for live objects of each removed kind across all namespaces, and compare the results with the repository. Manifests applied by hand, by a controller or by a pipeline that has since changed can drift from the Git history. For Helm, list releases in every namespace, read the stored manifest for each release, and map each release back to a chart version and a Git path. Releases with no chart in Git are the highest risk, because nobody can change their templates before the upgrade. Confirm with your Helm version where release records are stored, and whether your control plane exposes any signal, such as audit events or metrics, that shows requests to deprecated APIs.

Finish with operators and custom resource definitions. Each operator and CRD bundles the API versions it serves and may depend on removed APIs internally. Review the CRD versions served and stored, any conversion webhooks, and the operator release notes for compatibility with your target minor version. Decide per operator whether to upgrade it before or after the Kubernetes step, and confirm that order with the vendor. Keep one findings register with the resource, namespace, owner, target version, fix and verification status for each item, and repeat the scan at every minor step, since the version skew policy requires control plane components to move one minor version at a time.

The upgrade path for a self-managed cluster has to pass through every minor version between the current release and the target, and the reason is the Kubernetes version skew policy. That policy requires kube-apiserver to be upgraded one minor version at a time, and to be upgraded before kube-controller-manager, kube-scheduler and kubelets. Kubelets may run up to three minor versions behind the API server, so a node pool can lag behind the control plane for a while, but the control plane itself cannot jump. For the operator, this turns deprecated API cleanup into a sequence problem: the order in which API fixes are made has to follow the order in which the control plane will move.

Because each control plane step is one minor version, every manifest, Helm chart and operator that depends on an API has to work on each intermediate target, not only on the final one. A workload that is corrected for the release you are upgrading to may still break on an earlier intermediate release if it uses an API that is removed there. The practical check is to build a table with one row per intermediate target, listing the APIs that are removed or deprecated in that release, and then assign each workload fix to the step where it first becomes necessary. Confirm the removal list for each release in that release's own notes, not from memory or from a summary of a later release.

Intermediate releases can remove APIs in their own right, so the removal list is not simply the difference between your starting and final versions. An API that was deprecated long ago may be removed on a middle step, and a cluster that only checks the final target can pass that check and still fail during the upgrade. Plan for this by asking your platform team and any chart or operator vendors which intermediate releases they have tested, and by reviewing the release notes for every minor version in the path. Record the outcome of each review so that an auditor or a later operator can see why each step was approved.

The version skew policy also sets limits on how long mixed-version states can last. Kubelets can lag the API server by up to three minor versions, but that allowance does not mean the lag should be used as a buffer. Each step still needs its own API check, its own workload validation and a rollback point, and the team should decide in advance how many minor versions they will allow kubelets to trail during the upgrade window. Confirm with your distribution vendor whether it supports the same skew allowance, because a packaged distribution may set its own upgrade rules on top of the upstream policy.

A rollback restores the state you captured, and nothing outside the cluster returns with it. Before you change any API version in a manifest, chart or operator, decide which restore points you will rely on and what each one leaves untouched. The most important distinction is between cluster state held in etcd and everything the cluster depends on outside it: container images in your registry, DNS records, cloud load balancers, storage volumes, certificates issued by external authorities, and databases your workloads write to. An etcd snapshot restore brings back the objects the API server knew about at snapshot time. It does not delete images pushed afterward, roll back DNS changes, recreate cloud resources that were removed, or undo writes that applications made to their own databases after the snapshot.

Restoring etcd is a cluster-wide operation and must be planned as one. The Kubernetes documentation uses etcdctl snapshot save to take backups and etcdutl for restores. Confirm your topology first. In a stacked control plane, etcd members run on the control plane nodes, so the restore involves stopping kube-apiserver and the other control plane components on every node, restoring the snapshot consistently, and bringing members back in the same membership. With an external etcd cluster, the etcd members are separate hosts, and the restore has to be coordinated with the API server nodes that point at them. Plan for an odd number of etcd members, because quorum depends on a majority. The etcd guidance recommends a five-member cluster for production. Confirm with your own runbook and the upstream operations guide how many members you run and which restore steps apply to that layout before an upgrade window opens.

Helm releases and manifests in Git can usually be rolled back more precisely, but only for the objects they describe. Reverting a Git commit that restores an older API version will not work on a cluster where the old version has already been removed by the target release, so check the target version before treating a revert as a safe fallback. For Helm, a previous revision can be reinstated only if the chart version and its rendered objects still deploy on the current API server. Store the rendered output of each release in your change record, not just the chart version, so you know exactly what a revert will apply. Remember that rollback of a Helm release does not revert resources that a chart created outside its manifests, such as custom resources managed by an operator that has since been upgraded.

Decide in advance the point after which a fix cannot be reverted cleanly. This usually occurs once the target control plane version has written objects in the new storage version, once an operator has migrated custom resources, or once workloads have written data that the old version cannot read. Mark that boundary in your upgrade plan, with the checks that confirm you have passed it and the owner who authorizes the next step. Ask your platform team to confirm which fixes are one-way for your installed operators and charts, and test the restore path on a non-production copy of the cluster that matches your topology before you rely on it.