Unfortunately, after the migration from ext4 to ZFS, I discovered that k3s was crashing due to the missing support for overlayfs in ZFS.

Just before tackling this problem, I upgraded to Debian bullseye. By doing so, I encountered another problem: cgroup v2.

Cgroups v2

I’m running k3s version v1.19.5+k3s1, and apparently k3s doesn’t support cgroupv2 until v1.20 (I think). I’ll upgrade my k3s later. So, for now, I’ll just revert to cgroupsv1 (until the upgrade of k3s) by adding this line to the GRUB_CMDLINE_LINUX in /etc/default/grub:

systemd.unified_cgroup_hierarchy=0

Don’t forget to run update-grub.

k3s, ZFS and overlayfs

k3s doesn’t support ZFS snapshotter currently. There are three ways to work around this problem:

• Use an external containerd and the ZFS snapshotter (as containerd supports ZFS). However, I wasn’t so lucky with that, and I gave up after few hours.
• Use the native snapshotter, which is slower and consume more disk (you’ll have a lot of duplicated files), but it works.
  • Just edit /etc/systemd/system/k3s.service and add --snapshotter native to the k3s server options.
• Create a ZVOL with ext4 for k3s “agent”. I ended up with this solution as it’s faster and uses less space.

Create a ZVOL

We will exploit the ZVOL feature of ZFS for creating an ext4 volume to use as backend for overlayfs. ZVOL are block devices that ZFS can expose to the system (in contrast to datasets, which are ZFS filesystems). These ZVOLs can be used like a normal block device: we will create one of them, format it with ext4 and use it to host the k3s agent directory.

# Create a sparse (-s) zvol (-V).
# "Sparse" means that the volume will be allocated as data is written into the
# volume. So this ZVOLs will grow over time, when needed, until 50G
zfs create -s -V 50GB zpool/k3s/agent

# Format the ZVOL as ext4
mkfs.ext4 /dev/zvol/zpool/k3s/agent

# Now you need to use /dev/zvol/zpool/k3s/agent as block device for the k3s
# agent directory.
# _netdev here means that this mount will be delayed until the network is
# ready: this is an hack to mount the ZVOL after the import of zpools
# (no network is needed)
echo "/dev/zvol/zpool/k3s/agent /var/lib/rancher/k3s/agent ext4 defaults,_netdev 0 0" >> /etc/fstab

# Mount the directory manually
mount /var/lib/rancher/k3s/agent

Note: for my ZFS layout see the previous post about the homeserver migration to ZFS.

Shutdown

I experienced some issues shutting down my homeserver with k3s. I ended up using this systemd snippet:

# source https://github.com/k3s-io/k3s/issues/2400#issuecomment-1013798094
# $ sudo systemctl enable cgroup-kill-on-shutdown@k3s-agent.service.service
[Unit]
Description=Kill cgroup procs on shutdown for %i
DefaultDependencies=false
Before=shutdown.target umount.target
[Service]
# Instanced units are not part of system.slice for some reason
# without this, the service isn't started at shutdown
Slice=system.slice
ExecStart=/bin/bash -c "/usr/local/bin/k3s-killall.sh"
Type=oneshot
[Install]
WantedBy=shutdown.target