K0s in LXC and ZFS issues

Question

I'm trying to get K0s working in LXC (LXD) containers on Ubuntu 20.04 using ZFS and I've managed to get past the apparmor challenges (for now at least) but stuck on overlayfs issues during the install phase.

kernel: [1617529.987422] overlayfs: filesystem on '/var/lib/k0s/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/6371/fs' not supported as upperdir

Has anyone hit on this issues before and has a solution/workaround?

Regards, Matt

Answer 1

containerd needs to be told to use the zfs snapshotter:

• k0s will not start on ZFS-based systems
• mount a zfs dataset inside LXD

This approach will probably require the LXD container to run privileged & require the zfs modules passed through to the container.

With k0s it is also useful during the configuration above so containerd is found to add to your bash profile:

• export PATH=$PATH:/var/lib/k0s/bin

Alternative Approach (unprivileged LXD)

Adding an ext4 zvol to the container for containerd will probably be less problematic & is better from a security perspective as the container can run unprivileged . This approach works with k3s & /var/lib/rancher/k3s/agent/containerd running on LXD with zfs :

• /var/lib/kubelet seems to mount itself under zfs ok on k3s version v1.22.7+k3s1

• Using rootless overlayfs for the snapshotter works now on Ubuntu 22.04 thanks to this patch . It uses less space ( 639MB versus 1.9GB ) & gives better performance than the native snapshotter.