Waking a Server After Two Years

I built a small Proxmox server in 2024, installed it two times, and then left it off for two years. This Post is the technical record of the day I started it again: each fault with its error text, the commands that found the cause, and the config that runs now.

I did the work with an AI agent that had SSH access to the host. I name its errors too, because they are part of the record.

The network addresses in this Series are examples. The structure is real, and the numbers are not mine.

The hardware

The host ran Proxmox VE 8.2.2 on Debian 12, kernel 6.8.4, with the enterprise package sources and no subscription.

Fault one: Destination Net Unreachable

apt update could not resolve a name. The network config looked correct, and the gateway answered ping. But a ping to the internet got this, from the gateway itself:

$ ping -c2 1.1.1.1
From 192.168.50.1 icmp_seq=1 Destination Net Unreachable
From 192.168.50.1 icmp_seq=2 Destination Net Unreachable

A router that says "net unreachable" has no route out. The laptop on the same subnet told the rest of the story:

$ route -n get default | grep gateway
    gateway: 192.168.50.254

In 2024 the router held .1. Today a Wi-Fi access point with OpenWRT holds .1, with its WAN port not connected and its DHCP server still on. The hardware address in the ARP table matched my own inventory note for that access point.

The agent tested the fix on the running system first, then saved it:

ip route replace default via 192.168.50.254 dev vmbr0
ping -c2 1.1.1.1                      # 0% packet loss
sed -i 's|gateway 192.168.50.1$|gateway 192.168.50.254|' /etc/network/interfaces

The upgrade, 8.2 to 9.2

The path is two steps: the last 8.4 release first, then 9.

Without a subscription the enterprise sources answer 401, so they were replaced:

# /etc/apt/sources.list.d/pve-no-subscription.list
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

apt dist-upgrade took 270 packages to 8.4.21. The checker for the major upgrade then reported 33 passed, 0 failed, and 1 warning for the missing amd64-microcode package:

pve8to9 --full

For the major step, the sources change from bookworm to trixie, and the Ceph source from ceph-quincy to ceph-squid. Two details made a remote upgrade safer.

A snapshot of the root dataset first:

zfs snapshot -r rpool/ROOT@pre-pve9

And the upgrade as a detached systemd unit with a log, so a dropped SSH session cannot stop it in the middle:

systemd-run --unit=pve9-upgrade \
  --setenv=DEBIAN_FRONTEND=noninteractive --setenv=APT_LISTCHANGES_FRONTEND=none \
  --property=StandardOutput=file:/root/upgrade-9.log \
  apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold dist-upgrade

591 packages upgraded, 160 new, 59 removed. The removals were all libraries with new names in Debian 13. The result: Proxmox VE 9.2.21, Debian 13.7, kernel 7.0.14, and ZFS 2.4.4.

Three things needed attention after it, and the checker finds the first two:

• The systemd-boot meta-package must be removed on a host that boots through proxmox-boot-tool.
• amd64-microcode needs the non-free-firmware component.
• The upgrade adds a new file, pve-enterprise.sources, which brings the 401 back. One line, Enabled: false, turns it off.

Fault two: more than one matching pool

The host did not come back. The console showed this:

Command: /usr/sbin/zpool import -N 'rpool'
Message: cannot import 'rpool': more than one matching pool
import by numeric ID instead
Error: 1

Failed to import pool 'rpool'.
Manually import the pool and exit.
(initramfs)

The warning was there before the upgrade. A service had failed at each boot:

zfs-import-scan.service: Failed with result 'exit-code'.
zpool[813]: Last accessed by (none) (hostid=82a14d48) at Wed Dec 31 19:00:00 1969
zpool[813]: The pool can be imported, use 'zpool import -f' to import the pool.

The agent saw it and called it harmless, because zpool import listed no other pool at that moment. That was wrong. ZFS 2.2 ignored the stale pool at boot. ZFS 2.4 does not guess.

At the prompt, zpool import lists both pools with a numeric id. The one on the Samsung partition is the real one:

zpool import -N 17496142453079483950
exit

Where the second pool came from

The NVMe drive showed no partitions, but ZFS labels were on it:

$ zdb -l /dev/nvme0n1
failed to unpack label 0
failed to unpack label 1
    name: 'rpool'
    path: '/dev/disk/by-id/nvme-CT2000P3SSD8_...-part3'
    labels = 2 3

ZFS writes four labels to each device: two at the start and two at the end. Labels 2 and 3, at the end, were intact. Their path says the pool lived on partition 3, and there was no partition 3. So the partition table was gone and the data probably was not.

The Proxmox installer puts partition 3 after a 1 GB EFI partition, at sector 2099200. A read-only loop device at that offset showed all four labels:

L=$(losetup -r -f --show -o $((2099200*512)) /dev/nvme0n1)
zdb -l $L        # labels = 0 1 2 3

A read-only loop device cannot be written, so it is safe to import the pool from it under a temporary name:

zpool import -f -N -o readonly=on -o cachefile=none -R /mnt/old \
  -d $L -t 3607148325270156473 oldnvme
zfs list -r oldnvme

NAME                         USED  CREATION
oldnvme/ROOT/pve-1          1.43G  Sun Jul  7 14:34 2024
oldnvme/data/vm-100-disk-1  3.30G  Sun Aug 11 12:22 2024
oldnvme/data/vm-101-disk-1  3.97G  Sun Jul  7 16:42 2024

It was my first install from July 2024, complete, with both VMs. Even their definitions were readable, from the SQLite file behind /etc/pve:

sqlite3 -readonly "file:/mnt/old/var/lib/pve-cluster/config.db?immutable=1" \
  "select name, data from tree where name like '%.conf'"

In August 2024 I had installed again on the SATA drive and wiped the NVMe partition table. A wipe of the table leaves the pool.

I did not need the old VMs. A full discard removes each label, then the drive becomes its own pool with a name that cannot collide:

blkdiscard -f /dev/nvme0n1
zpool create -o ashift=12 -o autotrim=on \
  -O compression=lz4 -O atime=off -O xattr=sa -O acltype=posixacl \
  erebus /dev/disk/by-id/nvme-CT2000P3SSD8_...
pvesm add zfspool erebus --pool erebus --content images,rootdir --sparse 1

Fault three: SVM disabled

One line in the kernel log explained why my VMs were slow in 2024:

SVM disabled (by BIOS) in MSR_VM_CR

There was no /dev/kvm. The old VM definitions had kvm: 0, so they ran in full software emulation. Two BIOS settings fixed it: SVM Mode, and IOMMU for the passthrough later. After the restart the kernel listed 29 IOMMU groups, and the GPU sat alone in one with its audio function, which is the best case.

A sandbox network with no path home

Code that agents write must not reach my lights or my laptop. The switch has no VLAN support, so the separation lives on the host.

Proxmox SDN gives a bridge with DHCP and source NAT: a simple zone, one VNet named chaos, and the subnet 10.20.0.0/24. SDN does the routing out. It does not block the path to the home LAN. A small nftables table does that:

table inet chaos_guard {
  chain forward {
    type filter hook forward priority -10; policy accept;
    iifname "chaos" ct state established,related accept
    iifname "chaos" ip daddr 10.20.0.0/24 accept
    iifname "chaos" ip daddr { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16,
                               100.64.0.0/10, 169.254.0.0/16 } drop
    iifname "chaos" meta nfproto ipv6 drop
  }
  chain input {
    type filter hook input priority -10; policy accept;
    iifname "chaos" ct state established,related accept
    iifname "chaos" udp dport 67 accept
    iifname "chaos" icmp type echo-request accept
    iifname "chaos" drop
  }
}

The 100.64.0.0/10 range is the Tailscale network, so a sandbox cannot reach my other machines through the host. The input chain keeps guests off the web UI and SSH of the host.

The test needs no VM. A network namespace with one end of a veth pair on the bridge is a guest:

ip netns add t
ip link add vt0 type veth peer name vt1
ip link set vt0 master chaos up && ip link set vt1 netns t
ip netns exec t sh -c 'ip addr add 10.20.0.9/24 dev vt1; ip link set vt1 up; ip route add default via 10.20.0.1'
ip netns exec t ping -c1 1.1.1.1                       # works
ip netns exec t nc -z -w2 192.168.50.20 8123           # blocked

A console for a host with one GPU

The GPU goes to a VM, and the CPU has no graphics. So the console goes dark when that VM starts. For recovery, the boot menu has a second entry that masks guest autostart:

title    Proxmox VE rescue (no guest autostart)
options  root=ZFS=rpool/ROOT/pve-1 boot=zfs systemd.mask=pve-guests.service

proxmox-boot-tool writes over its own entries at each kernel update, so a small script writes this one again from the newest entry, at each boot and after each kernel install. To use it one time from a shell: bootctl set-oneshot nyx-rescue.conf.

GPU passthrough for a GTX 980

The host must not hold the card. One line in /etc/modprobe.d/ does it: blacklist nouveau. Proxmox binds vfio-pci at VM start:

qm set 110 --hostpci0 0000:09:00,pcie=1

The guest is Ubuntu 24.04, for one reason. The GTX 980 needs the legacy 580 driver branch, and Ubuntu ships that branch with ready-built modules for its kernel, so nothing compiles:

apt install linux-modules-nvidia-580-server-$(uname -r) \
  nvidia-headless-no-dkms-580-server nvidia-utils-580-server

Ollama in Docker found the card, with one notable log line:

msg="skipping CUDA device — compute capability not in compiled architectures"
  device="NVIDIA GeForce GTX 980" cc=520 archs="[750 800 860 ... 1210]"
msg="inference compute" library=CUDA compute=5.2 libdirs=ollama,cuda_v12

The CUDA 13 build skips a card with compute capability 5.2. The CUDA 12 build in the same image accepts it. A later image can drop that build, so the image version is pinned.

The first response

The first prompt to a local model on the server was about its own name:

Who is Nyx in Greek mythology? Answer in two sentences.

llama3.2:3b answered at 44.3 tokens each second, with all 2.72 GB in video memory:

In Greek mythology, Nyx (also known as Nox) was the goddess of night, born from the void of chaos at the beginning of the universe. She was the mother of the twelve Titans and the personification of darkness, shadow, and the underworld, often depicted as the sister and consort of Uranus, the sky god.

The second sentence is wrong two times. Gaia is the mother of the Titans, and the consort of Nyx is Erebus. A model of three billion parameters is fast on old hardware, and it is not a source of facts.

What runs now

What I take from this

• A failed unit is not harmless because the system boots. zfs-import-scan failed for two years. It was the one warning of the boot failure.
• A wiped partition table is not a wiped disk. zdb -l on the whole device shows what is left.
• Compare two machines on the same network before you debug one. The wrong gateway was visible in one command on the laptop.
• Run a remote upgrade as a unit, not in the SSH session.
• Read what the agent says is fine. It was right about most things and wrong about the one that mattered. It also made seven commits to the main branch before I set a rule against it.