zfsutils-linux and HWE Kernels

By hernil

If you are setting up an Ubuntu server on new-ish hardware a few good months after a normal Ubuntu LTS release you might run into the need (or just want) to use a so called HWE Linux kernel with your installation. For the record the LTS releases are always in april of even numbered years. At the time of writing the current one is 22.04 and the next is coming in a few months and will be 24.04.

Ubuntu 22.04 Jammy Jellyfish shipped with the 5.15 kernel and 22.04.3 is now bundling the 6.5 kernel. The most common use case for a new kernel is better support for newer hardware. That said, if the original kernel shipped with the LTS works for you I believe that is what will be supported until that release’s EOL.

But let’s say you want the new shiny - but still stable experience that is the Ubuntu HWE kernel.

The problem

Ubuntu is one of the few distros that ship ZFS out of the box. A convenience that is very much welcome for someone setting up a primarily storage focused server. And if you are configuring everything with Sanoid and Syncoid replication you might run into something resembling this in your logs

Jan 13 17:57:17 chewie syncoid[3324896]: pigz: skipping: <stdin>: corrupted -- incomplete deflate data
Jan 13 17:57:17 chewie syncoid[3324892]: cannot receive resume stream: checksum mismatch or incomplete stream.
Jan 13 17:57:17 chewie syncoid[3324892]: Partially received snapshot is saved.
Jan 13 17:57:17 chewie syncoid[3324892]: A resuming stream can be generated on the sending system by running:
Jan 13 17:57:17 chewie syncoid[3324892]:     zfs send -t 1-12002e2707-110-789c636064000310a501c49c50360710a715e5e7a69766a6304081e1d2e485bc99de0b14806c762475f94959a9c9250c0ce5fb99c0ea30e4d3d28a534b40326570793624f9a4ca92d46220edb067ce34112cfa4bf221ae98f2ecdfbdefaaa17f2390e439c1f27989b9a9403a312f5bbf3823b128b5583f3d3137373547b738b5a82cb5c821b1b424bf382fb120dec8c0c8
Jan 13 17:57:17 chewie syncoid[3322880]: CRITICAL ERROR: ssh     -S /tmp/syncoid-zfssend@ zfssend@ ' zfs send  -t 1-162420c769-110-789c636064000310a501c49c50360710a715e5e7a69766a6304081e1d2e485bc99de0b14806c762475f94959a9c9250c0c592b98c0ea30e4d3d28a534b4032758c30793624f9a4ca92d46220ddb1fcf814212cfa4bf221ae98f2ecdfbdefaaa17f2390e4
39c1f27989b9a9403a312f5bbf3823b128b5583f3d3137373547b738b5a82cb5c821b1b424bf382fb120dec8c0c844d7c050d7d028dec0c00a848ce3531233732a617e040046573246 | pigz -9 | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | pigz -dc | pv -p -t -e -r -b -s 239246617880 |  zfs receive  -s -F 'tank/backup/zfs/client/tank/shares/data' 2>&1
 failed: 256 at /usr/sbin/syncoid line 580.

The cause

Turns out this issue can be caused by a mismatch of the zfsutils-linux package in the Ubuntu repo and the bundled OpenZFS in the Ubuntu HWE kernel.

This can be verified by running zfs version which might output something like this:


(I’ve since corrected this on my system so this particular version combo might not be possible)

The problem here is first and foremost the mismatch of the minor versions ie 2.1 and 2.2. Although a major version mismatch (like some experienced between version 0.8 and 2) would probably be even worse.

The solution

The easiest solution is to not use the HWE kernel.

The other solution is to selectively install the zfsutils-linux package from the Ubuntu repos matching the kernel you are using with what is known as apt pinning. In our case the 6.5 kernel was shipped with Ubuntu 23.10 Mantic Minotaur so logically the correct version if zfsutils-linux should be in those repos.

In short we add the mantic repos to our installation but tell it to only apply it as a source for a single package zfsutils-linux. We do this by creating a new file with

sudo vim /etc/apt/preferences.d/zfsutils-mantic-backport

This file should contain the following

Package: zfsutils-linux
Pin: release n=mantic
Pin-Priority: 990

# Don't do anything with packages that are not from jammy
Package: *
Pin: release n=jammy
Pin-Priority: 900

Package: *
Pin: release o=Ubuntu
Pin-Priority: -10

And in your /etc/apt/sources.list file you append the following to the end

# For zfsutils-linux userspace tools
deb http://archive.ubuntu.com/ubuntu mantic main restricted universe multiverse

Now type the following commands

sudo apt update 
sudo apt-cache policy
sudo apt install -t mantic zfsutils-linux

Now zfs version should look something like this







Input or feedback to this content?
Reply via email!
Related Articles