zfsutils-linux and HWE Kernels
By hernil
If you are setting up an Ubuntu server on new-ish hardware a few good months after a normal Ubuntu LTS release you might run into the need (or just want) to use a so called HWE Linux kernel with your installation. For the record the LTS releases are always in april of even numbered years. At the time of writing the current one is 22.04 and the next is coming in a few months and will be 24.04.
Ubuntu 22.04 Jammy Jellyfish shipped with the 5.15 kernel and 22.04.3 is now bundling the 6.5 kernel. The most common use case for a new kernel is better support for newer hardware. That said, if the original kernel shipped with the LTS works for you I believe that is what will be supported until that release’s EOL.
But let’s say you want the new shiny - but still stable experience that is the Ubuntu HWE kernel.
The problem
Ubuntu is one of the few distros that ship ZFS out of the box. A convenience that is very much welcome for someone setting up a primarily storage focused server. And if you are configuring everything with Sanoid and Syncoid replication you might run into something resembling this in your logs
Jan 13 17:57:17 chewie syncoid[3324896]: pigz: skipping: <stdin>: corrupted -- incomplete deflate data
Jan 13 17:57:17 chewie syncoid[3324892]: cannot receive resume stream: checksum mismatch or incomplete stream.
Jan 13 17:57:17 chewie syncoid[3324892]: Partially received snapshot is saved.
Jan 13 17:57:17 chewie syncoid[3324892]: A resuming stream can be generated on the sending system by running:
Jan 13 17:57:17 chewie syncoid[3324892]: zfs send -t 1-12002e2707-110-789c636064000310a501c49c50360710a715e5e7a69766a6304081e1d2e485bc99de0b14806c762475f94959a9c9250c0ce5fb99c0ea30e4d3d28a534b40326570793624f9a4ca92d46220edb067ce34112cfa4bf221ae98f2ecdfbdefaaa17f2390e439c1f27989b9a9403a312f5bbf3823b128b5583f3d3137373547b738b5a82cb5c821b1b424bf382fb120dec8c0c8
44d7c050d7d028dec0c00a848ce3531233732a617e04002dea3207
Jan 13 17:57:17 chewie syncoid[3322880]: CRITICAL ERROR: ssh -S /tmp/syncoid-zfssend@192.168.1.5-1705163866 zfssend@192.168.1.5 ' zfs send -t 1-162420c769-110-789c636064000310a501c49c50360710a715e5e7a69766a6304081e1d2e485bc99de0b14806c762475f94959a9c9250c0c592b98c0ea30e4d3d28a534b4032758c30793624f9a4ca92d46220ddb1fcf814212cfa4bf221ae98f2ecdfbdefaaa17f2390e4
39c1f27989b9a9403a312f5bbf3823b128b5583f3d3137373547b738b5a82cb5c821b1b424bf382fb120dec8c0c844d7c050d7d028dec0c00a848ce3531233732a617e040046573246 | pigz -9 | mbuffer -q -s 128k -m 16M 2>/dev/null' | mbuffer -q -s 128k -m 16M 2>/dev/null | pigz -dc | pv -p -t -e -r -b -s 239246617880 | zfs receive -s -F 'tank/backup/zfs/client/tank/shares/data' 2>&1
failed: 256 at /usr/sbin/syncoid line 580.
The cause
Turns out this issue can be caused by a mismatch of the zfsutils-linux package in the Ubuntu repo and the bundled OpenZFS in the Ubuntu HWE kernel.
This can be verified by running zfs version which might output something like this:
zfs-2.1.5-1ubuntu6
zfs-kmod-2.2.0-0ubuntu1~23.10
(I’ve since corrected this on my system so this particular version combo might not be possible)
The problem here is first and foremost the mismatch of the minor versions ie 2.1 and 2.2. Although a major version mismatch (like some experienced between version 0.8 and 2) would probably be even worse.
The solution
The easiest solution is to not use the HWE kernel.
The other solution is to selectively install the zfsutils-linux package from the Ubuntu repos matching the kernel you are using with what is known as apt pinning. In our case the 6.5 kernel was shipped with Ubuntu 23.10 Mantic Minotaur so logically the correct version if zfsutils-linux should be in those repos.
In short we add the mantic repos to our installation but tell it to only apply it as a source for a single package zfsutils-linux. We do this by creating a new file with
sudo vim /etc/apt/preferences.d/zfsutils-mantic-backport
This file should contain the following
Package: zfsutils-linux
Pin: release n=mantic
Pin-Priority: 990
# Don't do anything with packages that are not from jammy
Package: *
Pin: release n=jammy
Pin-Priority: 900
Package: *
Pin: release o=Ubuntu
Pin-Priority: -10
And in your /etc/apt/sources.list file you append the following to the end
# For zfsutils-linux userspace tools
deb http://archive.ubuntu.com/ubuntu mantic main restricted universe multiverse
Now type the following commands
sudo apt update
sudo apt-cache policy
sudo apt install -t mantic zfsutils-linux
Now zfs version should look something like this
zfs-2.2.0~rc3-0ubuntu4
zfs-kmod-2.2.0-0ubuntu1~23.10
Sources
https://douglasrumbaugh.com/post/apt-pinning/
https://github.com/jimsalterjrs/sanoid/issues/683
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1939210
Reply via email!