Mart Roosmaa - zfsZola2024-09-21T00:00:00+00:00https://www.roosmaa.net/tags/zfs/atom.xmlSetting up ZFS on Talos2024-09-21T00:00:00+00:002024-09-21T00:00:00+00:00
Mart Roosmaa
https://www.roosmaa.net/blog/2024/setting-up-zfs-on-talos/<p>Talos, being an immutable distro, is amazing, but it does come with a caveat. When it's time to deviate from the defaults, it involves some additional steps. Talos has extension support for those cases. There's also a bunch of official/community extensions ready to go. Rolling your own is also possible, but involves a bit of a learning curve.</p>
<p>I wanted to convert one of my Kubernetes nodes into a ZFS-based storage box. Luckily for me, Talos has a community-maintained <a href="https://github.com/siderolabs/extensions/tree/main/storage/zfs" class="external">ZFS extension</a> to get the ZFS kernel module and userland tools installed on the node. Unluckily, I wasn't able to locate sufficient documentation about it to get it working on first try. After a bunch of trial and error (and head banging), I was able to get things figured out.</p>
<h2 id="configuring-the-talos-node">Configuring the Talos node</h2>
<p>To customize the extensions included in the Talos node, one needs use a purpose built install images. There's two options - using <a href="https://factory.talos.dev/" class="external">Talos Factory</a> (the easy way), or using <a href="https://www.talos.dev/v1.8/talos-guides/install/boot-assets/#imager" class="external">imager</a> to do it locally.</p>
<p>Using Talos Factory web interface is very straightforward, just need to select the <code>siderolabs/zfs</code> under the system extensions. Doing it in Yaml and using the HTTP API is as easy:</p>
<pre class="giallo" style="color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);"><code data-lang="yaml"><span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">c</span><span style="color: light-dark(#22863A, #85E89D);">ustomization</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> s</span><span style="color: light-dark(#22863A, #85E89D);">ystemExtensions</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> o</span><span style="color: light-dark(#22863A, #85E89D);">fficialExtensions</span><span>:</span></span>
<span class="giallo-l"><span> -</span><span style="color: light-dark(#032F62, #9ECBFF);"> s</span><span style="color: light-dark(#032F62, #9ECBFF);">iderolabs/zfs</span></span></code></pre>
<p>Once you have the installation image, you can slot it into your Talos node configuration, like so:</p>
<pre class="giallo" style="color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);"><code data-lang="yaml"><span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">m</span><span style="color: light-dark(#22863A, #85E89D);">achine</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> i</span><span style="color: light-dark(#22863A, #85E89D);">nstall</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> i</span><span style="color: light-dark(#22863A, #85E89D);">mage</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> h</span><span style="color: light-dark(#032F62, #9ECBFF);">ttps://factory.talos.dev/image/4dd8e3a8b6203d3c14f049da8db4d3bb0d6d3e70c5e89dfcc1e709e81914f63c/v1.8.3/metal-amd64.iso</span></span></code></pre>
<p>But, turns out, that on it's own is not enough to get everything working. This is the part that made me lose several hours. Even though the <em>zfs.ko</em> module is present on the file-system, it isn't loaded in the kernel. In order to make that happen, one needs to tweak the node configuration once more and use the <a href="https://www.talos.dev/v1.8/reference/configuration/v1alpha1/config/#Config.machine.kernel" class="external">machine.kernel.modules</a> list to explicitly include zfs.</p>
<pre class="giallo" style="color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);"><code data-lang="yaml"><span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">m</span><span style="color: light-dark(#22863A, #85E89D);">achine</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> i</span><span style="color: light-dark(#22863A, #85E89D);">nstall</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> i</span><span style="color: light-dark(#22863A, #85E89D);">mage</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> h</span><span style="color: light-dark(#032F62, #9ECBFF);">ttps://factory.talos.dev/image/4dd8e3a8b6203d3c14f049da8db4d3bb0d6d3e70c5e89dfcc1e709e81914f63c/v1.8.3/metal-amd64.iso</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> k</span><span style="color: light-dark(#22863A, #85E89D);">ernel</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> m</span><span style="color: light-dark(#22863A, #85E89D);">odules</span><span>:</span></span>
<span class="giallo-l"><span> -</span><span style="color: light-dark(#032F62, #9ECBFF);"> z</span><span style="color: light-dark(#032F62, #9ECBFF);">fs</span></span></code></pre>
<p>After the above configuration has been <a href="https://www.talos.dev/v1.8/reference/cli/#talosctl-apply-config" class="external">applied</a> to the node, the regular <a href="https://www.talos.dev/v1.8/reference/cli/#talosctl-upgrade" class="external">upgrade</a> procedure will make everything ready for use.</p>
<h2 id="accessing-the-zfs-utilities">Accessing the ZFS utilities</h2>
<p>Even though the siderolabs/zfs extension includes the zfs tools (zfs, zpool, ...) on the node, using them is more involved as Talos doesn't support executing adhoc commands on the hosts directly. You need a root shell container on the node and execute the tools in the correct kernel namespace.</p>
<p>First, we need create a root shell that we can exec into. Depending on your needs it may be better to use DaemonSet to get the shell on to multiple nodes, but in case of a single node, we can just launch a simple Pod, like such:</p>
<pre class="giallo" style="color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);"><code data-lang="yaml"><span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">a</span><span style="color: light-dark(#22863A, #85E89D);">piVersion</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> v</span><span style="color: light-dark(#032F62, #9ECBFF);">1</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">k</span><span style="color: light-dark(#22863A, #85E89D);">ind</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> P</span><span style="color: light-dark(#032F62, #9ECBFF);">od</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">m</span><span style="color: light-dark(#22863A, #85E89D);">etadata</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> n</span><span style="color: light-dark(#22863A, #85E89D);">ame</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> z</span><span style="color: light-dark(#032F62, #9ECBFF);">fs-shell</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">s</span><span style="color: light-dark(#22863A, #85E89D);">pec</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> n</span><span style="color: light-dark(#22863A, #85E89D);">odeName</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> T</span><span style="color: light-dark(#032F62, #9ECBFF);">ARGET_NODE_NAME</span><span style="color: light-dark(#6A737D, #6A737D);"> #</span><span style="color: light-dark(#6A737D, #6A737D);"> TODO</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> h</span><span style="color: light-dark(#22863A, #85E89D);">ostIPC</span><span>:</span><span style="color: light-dark(#005CC5, #79B8FF);"> true</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> h</span><span style="color: light-dark(#22863A, #85E89D);">ostNetwork</span><span>:</span><span style="color: light-dark(#005CC5, #79B8FF);"> true</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> h</span><span style="color: light-dark(#22863A, #85E89D);">ostPID</span><span>:</span><span style="color: light-dark(#005CC5, #79B8FF);"> true</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> c</span><span style="color: light-dark(#22863A, #85E89D);">ontainers</span><span>:</span></span>
<span class="giallo-l"><span> -</span><span style="color: light-dark(#22863A, #85E89D);"> c</span><span style="color: light-dark(#22863A, #85E89D);">ommand</span><span>:</span><span> [</span><span style="color: light-dark(#032F62, #9ECBFF);">"</span><span style="color: light-dark(#032F62, #9ECBFF);">sleep</span><span style="color: light-dark(#032F62, #9ECBFF);">"</span><span>,</span><span style="color: light-dark(#032F62, #9ECBFF);"> "</span><span style="color: light-dark(#032F62, #9ECBFF);">infinity</span><span style="color: light-dark(#032F62, #9ECBFF);">"</span><span>]</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> i</span><span style="color: light-dark(#22863A, #85E89D);">mage</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> d</span><span style="color: light-dark(#032F62, #9ECBFF);">ebian</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> n</span><span style="color: light-dark(#22863A, #85E89D);">ame</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> s</span><span style="color: light-dark(#032F62, #9ECBFF);">hell</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> s</span><span style="color: light-dark(#22863A, #85E89D);">ecurityContext</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> p</span><span style="color: light-dark(#22863A, #85E89D);">rivileged</span><span>:</span><span style="color: light-dark(#005CC5, #79B8FF);"> true</span></span></code></pre>
<p>Afterwards we can run zfs tools via the nsenter command like so:</p>
<pre class="giallo" style="color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);"><code data-lang="shellscript"><span class="giallo-l"><span style="color: light-dark(#6F42C1, #B392F0);">kubectl</span><span style="color: light-dark(#032F62, #9ECBFF);"> exec</span><span style="color: light-dark(#032F62, #9ECBFF);"> pod/zfs-shell</span><span style="color: light-dark(#005CC5, #79B8FF);"> -</span><span style="color: light-dark(#005CC5, #79B8FF);">-</span><span style="color: light-dark(#005CC5, #79B8FF);"> \</span></span>
<span class="giallo-l"><span style="color: light-dark(#032F62, #9ECBFF);"> nsenter</span><span style="color: light-dark(#005CC5, #79B8FF);"> -</span><span style="color: light-dark(#005CC5, #79B8FF);">-mount=/proc/1/ns/mnt</span><span style="color: light-dark(#005CC5, #79B8FF);"> -</span><span style="color: light-dark(#005CC5, #79B8FF);">-</span><span style="color: light-dark(#005CC5, #79B8FF);"> \</span></span>
<span class="giallo-l"><span style="color: light-dark(#032F62, #9ECBFF);"> zpool</span><span style="color: light-dark(#032F62, #9ECBFF);"> status</span></span></code></pre><h2 id="zfs-backed-persistent-volumes">ZFS backed Persistent Volumes</h2>
<p>Now we have a ZFS capable node in our cluster with a bunch of disks attached to it. To make this storage available in Kubernetes, we can <a href="https://openebs.io/docs/user-guides/local-storage-user-guide/local-pv-zfs/zfs-installation" class="external">install OpenEBS</a> and make use of its Local PV ZFS storage engine.</p>
<p>First, we need to create the ZFS pool that will be used by OpenEBS. In my case, I had 6 disks and wanted to create a RAIDz2 pool on them.</p>
<pre class="giallo" style="color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);"><code data-lang="shellscript"><span class="giallo-l"><span style="color: light-dark(#6F42C1, #B392F0);">kubectl</span><span style="color: light-dark(#032F62, #9ECBFF);"> exec</span><span style="color: light-dark(#032F62, #9ECBFF);"> pod/zfs-shell</span><span style="color: light-dark(#005CC5, #79B8FF);"> -</span><span style="color: light-dark(#005CC5, #79B8FF);">-</span><span style="color: light-dark(#032F62, #9ECBFF);"> nsenter</span><span style="color: light-dark(#005CC5, #79B8FF);"> -</span><span style="color: light-dark(#005CC5, #79B8FF);">-mount=/proc/1/ns/mnt</span><span style="color: light-dark(#005CC5, #79B8FF);"> -</span><span style="color: light-dark(#005CC5, #79B8FF);">-</span><span style="color: light-dark(#005CC5, #79B8FF);"> \</span></span>
<span class="giallo-l"><span style="color: light-dark(#032F62, #9ECBFF);"> zpool</span><span style="color: light-dark(#032F62, #9ECBFF);"> create</span><span style="color: light-dark(#005CC5, #79B8FF);"> -</span><span style="color: light-dark(#005CC5, #79B8FF);">m</span><span style="color: light-dark(#032F62, #9ECBFF);"> legacy</span><span style="color: light-dark(#005CC5, #79B8FF);"> -</span><span style="color: light-dark(#005CC5, #79B8FF);">f</span><span style="color: light-dark(#032F62, #9ECBFF);"> zfspv-pool</span><span style="color: light-dark(#032F62, #9ECBFF);"> raidz2</span><span style="color: light-dark(#005CC5, #79B8FF);"> \</span></span>
<span class="giallo-l"><span style="color: light-dark(#032F62, #9ECBFF);"> /dev/disk/by-id/{DISK1,DISK2,DISK3,DISK4,DISK5,DISK6}</span></span></code></pre>
<p>By default ZFS wants to mount the main pool filesystem under some directory in the host. Instead, we can use the <code>-m legacy</code> parameter to tell ZFS to leave the mounting to us. When OpenEBS is creating new filesystems in the pool, it is also using the legacy option, meaning there's no requirement for the main pool to be mounted on the host side either.</p>
<p>Once the pool is created, the OpenEBS can be easily <a href="https://openebs.io/docs/quickstart-guide/installation#installation-via-helm" class="external">installed via their Helm chart</a>. After which, some ZFS storage clases need to be defined that are suitable for your workloads. Below are two basic storage classes I use.</p>
<pre class="giallo" style="color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);"><code data-lang="yaml"><span class="giallo-l"><span style="color: light-dark(#6A737D, #6A737D);">#</span><span style="color: light-dark(#6A737D, #6A737D);"> Storage class for random application files</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">a</span><span style="color: light-dark(#22863A, #85E89D);">piVersion</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> s</span><span style="color: light-dark(#032F62, #9ECBFF);">torage.k8s.io/v1</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">k</span><span style="color: light-dark(#22863A, #85E89D);">ind</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> S</span><span style="color: light-dark(#032F62, #9ECBFF);">torageClass</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">m</span><span style="color: light-dark(#22863A, #85E89D);">etadata</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> n</span><span style="color: light-dark(#22863A, #85E89D);">ame</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> h</span><span style="color: light-dark(#032F62, #9ECBFF);">ost-zfs-standard</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">p</span><span style="color: light-dark(#22863A, #85E89D);">rovisioner</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> z</span><span style="color: light-dark(#032F62, #9ECBFF);">fs.csi.openebs.io</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">a</span><span style="color: light-dark(#22863A, #85E89D);">llowVolumeExpansion</span><span>:</span><span style="color: light-dark(#005CC5, #79B8FF);"> true</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">p</span><span style="color: light-dark(#22863A, #85E89D);">arameters</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> r</span><span style="color: light-dark(#22863A, #85E89D);">ecordsize</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> "</span><span style="color: light-dark(#032F62, #9ECBFF);">128k</span><span style="color: light-dark(#032F62, #9ECBFF);">"</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> c</span><span style="color: light-dark(#22863A, #85E89D);">ompression</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> "</span><span style="color: light-dark(#032F62, #9ECBFF);">lz4</span><span style="color: light-dark(#032F62, #9ECBFF);">"</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> d</span><span style="color: light-dark(#22863A, #85E89D);">edup</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> "</span><span style="color: light-dark(#032F62, #9ECBFF);">off</span><span style="color: light-dark(#032F62, #9ECBFF);">"</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> f</span><span style="color: light-dark(#22863A, #85E89D);">stype</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> "</span><span style="color: light-dark(#032F62, #9ECBFF);">zfs</span><span style="color: light-dark(#032F62, #9ECBFF);">"</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> p</span><span style="color: light-dark(#22863A, #85E89D);">oolname</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> "</span><span style="color: light-dark(#032F62, #9ECBFF);">zfspv-pool</span><span style="color: light-dark(#032F62, #9ECBFF);">"</span></span>
<span class="giallo-l"><span style="color: light-dark(#6A737D, #6A737D);">#</span><span style="color: light-dark(#6A737D, #6A737D);"> Use allowedTopologies: in case ZFS is only available</span></span>
<span class="giallo-l"><span style="color: light-dark(#6A737D, #6A737D);">#</span><span style="color: light-dark(#6A737D, #6A737D);"> on some of the nodes in the cluster.</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: light-dark(#6F42C1, #B392F0);">---</span></span>
<span class="giallo-l"><span style="color: light-dark(#6A737D, #6A737D);">#</span><span style="color: light-dark(#6A737D, #6A737D);"> Storage class for file storage (documents, photos, videos, etc)</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">a</span><span style="color: light-dark(#22863A, #85E89D);">piVersion</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> s</span><span style="color: light-dark(#032F62, #9ECBFF);">torage.k8s.io/v1</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">k</span><span style="color: light-dark(#22863A, #85E89D);">ind</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> S</span><span style="color: light-dark(#032F62, #9ECBFF);">torageClass</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">m</span><span style="color: light-dark(#22863A, #85E89D);">etadata</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> n</span><span style="color: light-dark(#22863A, #85E89D);">ame</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> h</span><span style="color: light-dark(#032F62, #9ECBFF);">ost-zfs-files</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">p</span><span style="color: light-dark(#22863A, #85E89D);">rovisioner</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> z</span><span style="color: light-dark(#032F62, #9ECBFF);">fs.csi.openebs.io</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">a</span><span style="color: light-dark(#22863A, #85E89D);">llowVolumeExpansion</span><span>:</span><span style="color: light-dark(#005CC5, #79B8FF);"> true</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">r</span><span style="color: light-dark(#22863A, #85E89D);">eclaimPolicy</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> R</span><span style="color: light-dark(#032F62, #9ECBFF);">etain</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">p</span><span style="color: light-dark(#22863A, #85E89D);">arameters</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> r</span><span style="color: light-dark(#22863A, #85E89D);">ecordsize</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> "</span><span style="color: light-dark(#032F62, #9ECBFF);">1M</span><span style="color: light-dark(#032F62, #9ECBFF);">"</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> c</span><span style="color: light-dark(#22863A, #85E89D);">ompression</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> "</span><span style="color: light-dark(#032F62, #9ECBFF);">lz4</span><span style="color: light-dark(#032F62, #9ECBFF);">"</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> d</span><span style="color: light-dark(#22863A, #85E89D);">edup</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> "</span><span style="color: light-dark(#032F62, #9ECBFF);">off</span><span style="color: light-dark(#032F62, #9ECBFF);">"</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> f</span><span style="color: light-dark(#22863A, #85E89D);">stype</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> "</span><span style="color: light-dark(#032F62, #9ECBFF);">zfs</span><span style="color: light-dark(#032F62, #9ECBFF);">"</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> s</span><span style="color: light-dark(#22863A, #85E89D);">hared</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> "</span><span style="color: light-dark(#032F62, #9ECBFF);">yes</span><span style="color: light-dark(#032F62, #9ECBFF);">"</span><span style="color: light-dark(#6A737D, #6A737D);"> #</span><span style="color: light-dark(#6A737D, #6A737D);"> to enable ReadWriteMany access mode</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> p</span><span style="color: light-dark(#22863A, #85E89D);">oolname</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> "</span><span style="color: light-dark(#032F62, #9ECBFF);">zfspv-pool</span><span style="color: light-dark(#032F62, #9ECBFF);">"</span></span></code></pre>
<p>To enable taking snapshots of the ZFS backed PVs, the a volume snapshot class also needs to be defined.</p>
<pre class="giallo" style="color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);"><code data-lang="yaml"><span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">k</span><span style="color: light-dark(#22863A, #85E89D);">ind</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> V</span><span style="color: light-dark(#032F62, #9ECBFF);">olumeSnapshotClass</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">a</span><span style="color: light-dark(#22863A, #85E89D);">piVersion</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> s</span><span style="color: light-dark(#032F62, #9ECBFF);">napshot.storage.k8s.io/v1</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">m</span><span style="color: light-dark(#22863A, #85E89D);">etadata</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> n</span><span style="color: light-dark(#22863A, #85E89D);">ame</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> h</span><span style="color: light-dark(#032F62, #9ECBFF);">ost-zfs-snapshot</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">d</span><span style="color: light-dark(#22863A, #85E89D);">river</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> z</span><span style="color: light-dark(#032F62, #9ECBFF);">fs.csi.openebs.io</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">d</span><span style="color: light-dark(#22863A, #85E89D);">eletionPolicy</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> D</span><span style="color: light-dark(#032F62, #9ECBFF);">elete</span></span></code></pre><h2 id="monitoring-with-netdata">Monitoring with Netdata</h2>
<p>The ZFS backed PVs should be usable now, but we're effectively flying blind. When a disk fails, we wouldn't know about it unless we manually checked the <code>zpool status</code> every so often. And if enough disks fail, it's <em>sayonara</em> to our data.</p>
<p>There are some Prometheus metrics exporters for ZFS, but I didn't explore that avenue, as <a href="https://www.netdata.cloud/" class="external">Netdata</a> has built-in support for ZFS data. But it doesn't work out of the box on Talos, due to the zpool command being difficult to access from the container.</p>
<p>I ended up creating a small utility (<a href="https://github.com/roosmaa/zfs-http-query)" class="external">zfs-http-query</a> that runs as a DaemonSet on the ZFS nodes and exposes zpool data via an unix socket. This allows pods that have access to that socket to query zfs data from an unprivileged container.</p>
<p>With zfs-http-query in place, the Netdata Helm values.yaml can be updated to make the built in ZFS support work:</p>
<pre class="giallo" style="color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);"><code data-lang="yaml"><span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);">c</span><span style="color: light-dark(#22863A, #85E89D);">hild</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> c</span><span style="color: light-dark(#22863A, #85E89D);">onfigs</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> z</span><span style="color: light-dark(#22863A, #85E89D);">fspool</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> e</span><span style="color: light-dark(#22863A, #85E89D);">nabled</span><span>:</span><span style="color: light-dark(#005CC5, #79B8FF);"> true</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> p</span><span style="color: light-dark(#22863A, #85E89D);">ath</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> /</span><span style="color: light-dark(#032F62, #9ECBFF);">etc/netdata/go.d/zfspool.conf</span></span>
<span class="giallo-l"><span style="color: light-dark(#6A737D, #6A737D);"> #</span><span style="color: light-dark(#6A737D, #6A737D);"> tell netdata zfspool integration to use the zpool shim from zfs-http-query</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> d</span><span style="color: light-dark(#22863A, #85E89D);">ata</span><span>:</span><span style="color: light-dark(#D73A49, #F97583);"> |</span></span>
<span class="giallo-l"><span style="color: light-dark(#032F62, #9ECBFF);"> jobs:</span></span>
<span class="giallo-l"><span style="color: light-dark(#032F62, #9ECBFF);"> - name: zfspool</span></span>
<span class="giallo-l"><span style="color: light-dark(#032F62, #9ECBFF);"> binary_path: /opt/zfs-http-query/bin/zpool</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> e</span><span style="color: light-dark(#22863A, #85E89D);">xtraVolumeMounts</span><span>:</span></span>
<span class="giallo-l"><span> -</span><span style="color: light-dark(#22863A, #85E89D);"> n</span><span style="color: light-dark(#22863A, #85E89D);">ame</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> o</span><span style="color: light-dark(#032F62, #9ECBFF);">pt-zfs-http-query</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> m</span><span style="color: light-dark(#22863A, #85E89D);">ountPath</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> /</span><span style="color: light-dark(#032F62, #9ECBFF);">opt/zfs-http-query</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> r</span><span style="color: light-dark(#22863A, #85E89D);">eadOnly</span><span>:</span><span style="color: light-dark(#005CC5, #79B8FF);"> true</span></span>
<span class="giallo-l"><span> -</span><span style="color: light-dark(#22863A, #85E89D);"> n</span><span style="color: light-dark(#22863A, #85E89D);">ame</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> r</span><span style="color: light-dark(#032F62, #9ECBFF);">un-zfs-http-query</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> m</span><span style="color: light-dark(#22863A, #85E89D);">ountPath</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> /</span><span style="color: light-dark(#032F62, #9ECBFF);">var/run/zfs-http-query</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> r</span><span style="color: light-dark(#22863A, #85E89D);">eadOnly</span><span>:</span><span style="color: light-dark(#005CC5, #79B8FF);"> true</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> e</span><span style="color: light-dark(#22863A, #85E89D);">xtraVolumes</span><span>:</span></span>
<span class="giallo-l"><span> -</span><span style="color: light-dark(#22863A, #85E89D);"> n</span><span style="color: light-dark(#22863A, #85E89D);">ame</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> o</span><span style="color: light-dark(#032F62, #9ECBFF);">pt-zfs-http-query</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> h</span><span style="color: light-dark(#22863A, #85E89D);">ostPath</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> t</span><span style="color: light-dark(#22863A, #85E89D);">ype</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> D</span><span style="color: light-dark(#032F62, #9ECBFF);">irectoryOrCreate</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> p</span><span style="color: light-dark(#22863A, #85E89D);">ath</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> /</span><span style="color: light-dark(#032F62, #9ECBFF);">opt/zfs-http-query</span></span>
<span class="giallo-l"><span style="color: light-dark(#6A737D, #6A737D);"> #</span><span style="color: light-dark(#6A737D, #6A737D);"> contains bin/zpool and bin/zfs shims</span></span>
<span class="giallo-l"><span> -</span><span style="color: light-dark(#22863A, #85E89D);"> n</span><span style="color: light-dark(#22863A, #85E89D);">ame</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> r</span><span style="color: light-dark(#032F62, #9ECBFF);">un-zfs-http-query</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> h</span><span style="color: light-dark(#22863A, #85E89D);">ostPath</span><span>:</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> t</span><span style="color: light-dark(#22863A, #85E89D);">ype</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> D</span><span style="color: light-dark(#032F62, #9ECBFF);">irectoryOrCreate</span></span>
<span class="giallo-l"><span style="color: light-dark(#22863A, #85E89D);"> p</span><span style="color: light-dark(#22863A, #85E89D);">ath</span><span>:</span><span style="color: light-dark(#032F62, #9ECBFF);"> /</span><span style="color: light-dark(#032F62, #9ECBFF);">var/run/zfs-http-query</span></span>
<span class="giallo-l"><span style="color: light-dark(#6A737D, #6A737D);"> #</span><span style="color: light-dark(#6A737D, #6A737D);"> contains the unix socket used by the shims</span></span></code></pre>
<p>With this in place, Netdata will have access to the ZFS pool data and can show graphs about the pool health. And send notifications (if properly configured) when the pool state becomes degraded (i.e. a disk failure).</p>
<h2 id="sharing-storage-across-nodes">Sharing storage across nodes</h2>
<p>Unlike Ceph or other distributed storage solutions, the OpenEBS based ZFS is tied to the node. When a PV is created on a certain node, it can only be accessed on that specifc node and it cannot be migrated to another node.</p>
<p>I found <a href="https://github.com/kubernetes-csi/csi-driver-smb" class="external">csi-driver-smb</a> project that allows mounting of samba shares to containers. This allows working around the above limitation by hosting a samba server on the ZFS enabled node, and using the csi-driver-smb to access it on any other node in the cluster.</p>
<p><img src="https://www.roosmaa.net/blog/2024/setting-up-zfs-on-talos/storage-diagram.svg#no-hover" alt="Network storage diagram" /></p>