Downgrading the kernel and fixing kernel modules after an incompatible new version in Fedora/Centos/RHEL
It happens quite often that a kernel module such as ZFS isnât quite up to speed with the latest Linux kernels, and a distro such as Fedora decides to go ahead and release the latest kernel version without waiting for ZFS to be ready.
The obvious symptom of this, is when you boot your server and your ZFS array is gone, your ZFS utils may or may not be gone too, and if they arenât, theyâll complain the ZFS kernel modules arenât loaded, and when you try modprobe zfs Linux will tell you thereâs no such thing.
Fret not! the dnf package manager allows you to downgrade the kernel to a previous version that ZFS is still compatible with. The command dnf downgrade kernel is pretty self explanatory, however, there are some caveats that need to be documented, and how to work around them.
Caveat 1: The ZFS module doesnât get recompiled
The first thing youâll probably notice is that, despite running dnf downgrade kernel and rebooting, your ZFS kernel modules still arenât back.
The reason for this, is because dkms doesnât always compile for older kernel versions, if there are multiple installed at the same time.
To fix this, after rebooting into the older kernel, simply run dnf reinstall zfs, this should be enough to fix it, although sometimes it may require a dnf remove zfs; dnf install zfs instead (donât ask me why).
Caveat 2: âOldest version already installedâ
If you run into this problem more often with the same kernel version, it may happen that a new version of the newer kernel gets released, and dnf automatically upgrades to it, while not removing the older version.
When this happens, dnf downgrade wonât work, because the older kernel is already installed.
You CAN try and mess with Grub so that it automatically boots the correct older version, but your mileage may vary, especially as dnf overwrites it anyway if another new kernel arrives, so I donât recommend this method.
To fix this, use dnf find kernel to find the exact version of the older kernel, copy the entire version string (e.g kernel-5.17.13-400.fc36.x86_64), and then run dnf remove 5.17.13-400.fc36.x86_64 to remove it. Afterwards, run dnf downgrade kernel again; it should work this time.
Update: You can also run dnf autoremove kernel and it should remove all the older versions:
[anthropy@stitch ~]$ sudo dnf autoremove kernel
Dependencies resolved.
============================================================================================================================================================================================================================================
Package Architecture Version Repository Size
============================================================================================================================================================================================================================================
Removing:
kernel x86_64 5.19.6-200.fc36 @updates 0
kernel x86_64 5.19.14-200.fc36 @updates 0
kernel x86_64 5.19.16-200.fc36 @updates 0