Proxmox VE 8 to 9 upgrade

The failures most likely to appear after a Proxmox VE 8 to 9 move are LXC containers built on assumptions the new host no longer meets, and host network configuration that depends on interface names the new kernel may present differently. Both can leave a node running but services unreachable, so they deserve inventory and testing before any production node is touched. Containers share the host kernel, which means anything in a container's configuration that depends on how the host handles resource control, devices, or older init and distribution behavior is exposed when the host changes. The removal of cgroup v1 support is the item to plan around, but confirm in the official Proxmox VE 9 release notes exactly which container types, distribution releases and settings it affects.

Finding those containers before the upgrade is inventory work. List every container on every node with its OS type and distribution release, whether it is unprivileged, its feature and nesting settings, mount points, device passthrough, and any hand-written tuning in its configuration file. Run the pve8to9 checker on each node and treat its output as a worklist, not as a pass or fail stamp. Ask your team or the vendor which checks the tool performs in the version you are targeting, because a clean run does not prove every container will start. Record which containers run business-critical services or have no named owner, since those need a decision about upgrade order or replacement before the change window opens.

Network configuration is the second group. Host interface names are assigned from the naming scheme in place at install and from udev rules, and a change in kernel, firmware or naming policy can rename a physical NIC. When that happens, bridges such as vmbr0, bonds and VLAN subinterfaces that reference the old names fail to come up, and cluster traffic on corosync links can end up on the wrong path or nowhere. On each node, compare the interface definitions in the network configuration file with the current MAC addresses and names, and write down which interface carries management, VM traffic, storage and cluster links. Check the release notes for the recommended way to pin names before the upgrade, and plan console or out-of-band access before rebooting any node.

Test all of this on a non-production copy. Build a node or small cluster that matches production in NIC models, bond modes, VLAN layout and storage type, and use the no-subscription repository for that copy, which Proxmox describes as suited to testing and non-production use. Upgrade the copy, then confirm interface names, bridges, bonds, cluster membership and a representative sample of containers from each distribution family you run. Check container start, service health inside each container, network reachability and scheduled backups. Keep production on the enterprise repository with a valid subscription key, and do not let test-only repository settings drift into production configuration.

Some items must be confirmed in the official release notes rather than assumed: the exact scope of the cgroup v1 removal, which container distributions and settings are affected, the interface naming behavior and any required pre-upgrade step, and the current pve8to9 check list for your target version. If Ceph is part of the cluster, check the Ceph upgrade order against the Proxmox notes as well, and note that the Ceph Squid 19.2.x line has an estimated end of life of October 31, 2026 according to the Ceph releases page. Write down each confirmed item with its source and date so the plan can be reviewed and signed off.

Choose the in-place upgrade when every node can be taken out of service in turn, the cluster is healthy and you can show that each node's running configuration matches what you intend to keep. An in-place upgrade keeps the existing hosts, their network and storage settings, cluster membership and the guests on them, so there is no guest migration and no rebuilt configuration. The cost is that every node carries forward whatever drift has built up since it was installed: packages changed by hand, leftover kernel parameters, unused repositories and settings nobody remembers making. Each node is a separate risk. A fault on node three is still a fault on node three after the upgrade, and it may only surface once the new release is running.

A fresh install on new or reimaged hardware gives you a clean Proxmox VE 9 base and forces you to write down what each node is for. The price is a larger project. Guests must be moved off each node before it is reimaged, using live migration or backup and restore, and the disk and network checks have to be repeated. Nodes must leave the cluster cleanly and rejoin, which means checking membership, quorum and the cluster file system before any guest moves back. Storage definitions, bridges and bonds, firewall rules, backup jobs, users and authentication realms, HA resources and repository subscriptions all need to be rebuilt and checked against your records, not reconstructed from memory.

Several factors usually decide the path. Spare hardware comes first: if you can add nodes without touching production, a fresh install is easier to roll back. Node count matters because Proxmox VE high availability needs at least three cluster nodes for reliable quorum, so a small cluster at that minimum has little room to take a node out for reimaging. Your Ceph layout matters as well, since monitors and OSD replicas must stay healthy while nodes restart, and you should confirm in the official documentation which Ceph release your target Proxmox VE version supports. Downtime tolerance sets how long guests can be moved or stopped. Finally, count the undocumented configuration. The more settings that exist only on the node, the stronger the case for an in-place upgrade preceded by a full backup, or for a fresh build preceded by a written inventory.

Many clusters end up with a mix: in-place upgrades on nodes that are healthy and well documented, and fresh installs on hardware that is being replaced anyway. Whichever path you take, run the pve8to9 checker on each node and treat its output as a starting list that your own inventory extends. Check interface naming before the reboot, because changed names can leave bridges and bonds without their member ports. Take verified backups of guests and copy the node configuration off the host before anything changes. Confirm the exact sequence, prerequisites and any version-specific steps in the official Proxmox VE upgrade documentation for the version you are moving to, and agree that procedure with your change board before the first node is touched.

Rehearse the full upgrade on a copy of the cluster before you touch production. A rehearsal only tells you something useful if the copy matches production in the places that cause upgrade failures: node count, storage types (ZFS, LVM, NFS or Ceph RBD), Ceph release, network bonds, VLAN tags, and the interface naming that the node will see after the move to the new release. Separate hardware gives the most reliable result. A nested setup can prove the procedure and the order of steps, but it will not show how real NIC drivers or disks behave under load. If you rely on high availability, the copy needs at least three nodes and shared storage, or the HA tests will not mean anything. Use the same authentication realms, and restore a recent copy of the production configuration so guest definitions, firewall rules and storage entries are the ones you will actually upgrade. Confirm that the Ceph release on the copy is the same release line you will run in production before you begin.

Run the same test cases before and after the upgrade so you can compare results. Start with guest boot: power on a sample of virtual machines and confirm they reach a usable state, including a guest that was already running before the upgrade and a guest that boots from a cold start. Test containers separately, because older containers are the most likely to fail after the removal of cgroup v1, so start each container class you run in production and check its services. Confirm network reachability from inside guests to gateways, storage networks and the management network. Measure storage I/O against your recorded baseline, and check Ceph health if you use it. Run a backup job and restore a guest to an alternate ID, then compare the restored data with the source. For HA, power off a node in a controlled window and confirm that watchdog fencing completes before guests are recovered on another node. Finally, log in with each authentication method your users depend on, including any second factor.

Set go and no-go criteria in writing before the rehearsal starts, so the decision is not made under pressure during the production window. A go decision should require that every test case passes, that the pve8to9 checker reports no blocking findings, that the cluster has quorum with all expected nodes, that Ceph reports a healthy state if it is in use, and that restored data matches the source. Agree on the performance thresholds for storage and network tests in advance, and name the person who signs off each criterion. Define stop conditions too: any failed restore, any container class that will not start without a fix you have not tested, or any authentication path that fails for an administrator account should halt the upgrade. Record the outcome of each test and each fix you applied during the rehearsal, because the production runbook should repeat only the steps that worked. If the rehearsal forces a change to the procedure, rehearse again before production.

Plan rollback for each stage, because the options are different at each point. Before any package change, the rollback is a restore: keep a verified backup of guests, a copy of the cluster configuration and a record of network and storage settings. Once a node has been upgraded in place, the usual route is to reinstall that node and restore its configuration, after moving its guests off first. A fresh install on new hardware can keep the old node running until the new one is verified, which gives you the simplest rollback, but it adds migration work. Ceph has its own rollback limits, since monitor and OSD daemon upgrades cannot always be reversed on the same hosts. The hardest case is mixed-version operation, when some nodes run the new release and others still run the old one. Rollback after that point may be limited, and the cluster may need to move forward rather than back. Confirm the exact rollback procedure in the vendor upgrade documentation before you start, and ask your partner to put that procedure in the runbook with its prerequisites.