Mart Roosmaa

Ollama, LM Studio vs llama.cpp

2025-12-24T00:00:00+00:00

When getting started with local LLMs, one of the very first decisions you have to make is what kind of inference engine [1] to use. There are quite a few out there, so which one is best? I'll take a look at the three main options: LM Studio, Ollama and llama.cpp.

[1] Inference engine: a program that gives life to the huge data files on HuggingFace.

LM Studio</h2>
LM Studio</a> is primarily a GUI application. It has a user-friendly interface for downloading the models and getting them running. It includes a chat interface where you can quickly converse with the loaded models.
The models that LM Studio uses come from HuggingFace. In HuggingFace there's LM Studio Community</a> which provides a set of curated versions for LM Studio.
Behind the scenes, it uses llama.cpp to do the actual model inference - CPU, CUDA, ROCm, Vulkan & Metal are all supported. In addition to all the llama.cpp-based engines, there's also an Apple's MLX-based engine.
LM Studio provides its own APIs in addition to an OpenAI-compatible API for interacting with the models programmatically, or from agents and IDEs that support OpenAI-compatible providers.
It also has `lms</code> CLI utility for people who prefer to interact with it via the terminal.`

`Ollama</h2>`Ollama</a> was trying to be the Docker of LLMs (before Docker started supporting running models itself). It is a CLI based application that gives users a very convenient way to download and run models. It focused on developers first, and successfully got itself integrated in various applications that end up being the more user-friendly frontends for it. The main innovation that Ollama introduced, in my opinion, is automatic model unloading based on VRAM usage. Ollama can keep as many models loaded as memory permits and will unload some of them only when needed to make room for the newly requested model. There are a few other ideas that they've adopted which differ from the status-quo. For example, Ollama introduced a Docker-like registry for models where files are stored and downloaded based on their SHA-256 hashes. HuggingFace has only recently implemented their "registry" support, so now it's possible to pull GGUFs directly from HuggingFace. Then there is the Modelfile concept, which is meant to be a Dockerfile-inspired solution to defining models. To me, this seems like an unnecessary - or even wrong - abstraction. If you want to tweak parameters (e.g., num_ctx</code>, num_gpu</code>, use_mmap</code>) to optimise a model for your hardware, you must create a new version of each model you use. What puzzles me the most is Ollama's decision to replace the the de-facto standard Jinja templates (for chat messages) with Go templates. This means that if one wants to run a newer model on Ollama, you also have to port the Jinja template to Go template syntax. Choosing the quantization level for a model is also somewhat involved. Ollama defaults to Q4_K_M quantization level for all its published models. If you want to change that, you must first obtain the Modelfile of ollama's model (for the chat template and parameters), then find a GGUF with the desired quantization on HuggingFace and finally combine the two. Ollama has been using llama.cpp behind the scenes as its inference engine, but it is slowly moving towards its own version. In the short term, I think, it won't be great for users as the new engine has to play catch-up with llama.cpp; in the long term, however, it could be interesting. Although Ollama relies on llama.cpp, it lags behind in some areas. For example, Vulkan support is still experimental, and Ollama does not support partitioned GGUFs, which are becoming the norm on HuggingFace. Llama.cpp</h2> Llama.cpp</a> is a C++ library that powers the two offerings above. In addition to being a library that others can build on, llama.cpp ships with llama-server</a> - a tool that lets you run models via OpenAI-compatible APIs, just like Ollama and LM Studio. There is also LlamaBarn</a>, a light-weight GUI for macOS. Llama.cpp is very actively developed, support for new models and optimizations for existing ones are added almost daily. The project does not have scheduled releases - what gets implemented (merged) is shipped automatically. This means that you always get the latest changes, but you may also encounter bugs and/or regressions. Llama-server exposes a lot of knobs to tweak and supports various backends (the main ones being CUDA, ROCm, Vulkan, and CPU). This allows you to experiment with a lot of them to find the best configuration for your hardware. Most of these knobs are also exposed in Ollama & LM Studio, but not all. The easiest way to run llama-server is via their official Docker images. It is also available through winget, homebrew or nix package managers. Pre-built binaries can be downloaded from the Github releases. In general, llama-server is best if you want to run the latest models & get the most out of your hardware. Side note: Reusing models</h3> It is perfectly reasonable to use more than one of the tools above. One question that may arise is whether you can download the model once and run in different tools, instead of downloading the data multiple times. Between LM Studio and llama-server reusing the downloaded models is relatively simple. Some LM studio specific models may use Jinja template functions that don't exist in llama.cpp, but in most cases the models are interchangeable. Ollama makes this more difficult. Even though it uses GGUFs behind the scenes, it stores everything by hash, so an extra step is required to translate to/from the hash when trying to reuse data.



ExternalDNS webhook for Technitium DNS
2025-02-20T00:00:00+00:00
Outside, a massive storm is raging. I'm about to settle on the sofa for a late-night movie. The storm has knocked out the internet, making Netflix inaccessible. Fortunately, I have some movies saved on my local Jellyfin server that I've been meaning to watch.</p>
But... Android TV is unable to connect to it?! What's going on? Oh noes! It's the DNS. All the friendly domain names are defined on the public servers, and thus inaccessible in the current situation.</p>

Even though above is part fiction, the underlying issue was real. I had set up my lab to register all of its domains on the Cloudflare DNS. It was the most convenient way of doing it. ExternalDNS</a>, the project picking up all the domain names from the cluster and registering them, had Cloudflare DNS support built in.</p>
For my internal ad-blocking DNS server, I was running Pi-hole</a>. ExternalDNS also has built in support for Pi-hole, so I attempted to setup another instance of ExternalDNS to sync the internal domains to my Pi-hole. I found out, that Pi-hole doesn't support wildcard domains, which I make use of in my lab. Sadly the Pi-hole provider didn't support domain exclusions either, so there was no easy way for me to hack around this limitation.</p>
This led me to switch from Pi-hole to the Technitium DNS</a> server. I had already been eyeing Technitium for a while, so this gave me the push I needed. Technitium is a full-blown DNS server, with all the features that Pi-hole has and so much more (including support for wildcard domains). While ExternalDNS doesn't natively support Technitium, it does offer webhooks as an extension point. After reviewing the APIs required for the webhook, I decided I could throw together Technitium support for it down the line.</p>
After some months, I finally got around to implementing that webhook to integrate Technitium DNS with ExternalDNS. I present to you external-dns-technitium-webhook</a>.</p>
Here's how I deploy it on my kubernetes cluster:</p>
apiVersion</span>: </span>apps/v1
</span>kind</span>: </span>Deployment
</span>metadata</span>:
</span>  </span>name</span>: </span>external-dns-technitium-dns
</span>  </span>namespace</span>: </span>external-dns
</span>  </span>labels</span>:
</span>    </span>app.kubernetes.io/name</span>: </span>external-dns
</span>spec</span>:
</span>  </span>strategy</span>:
</span>    </span>type</span>: </span>Recreate
</span>  </span>selector</span>:
</span>    </span>matchLabels</span>:
</span>      </span>app.kubernetes.io/name</span>: </span>external-dns
</span>  </span>template</span>:
</span>    </span>metadata</span>:
</span>      </span>labels</span>:
</span>        </span>app.kubernetes.io/name</span>: </span>external-dns
</span>    </span>spec</span>:
</span>      </span>serviceAccountName</span>: </span>external-dns
</span>      </span>containers</span>:
</span>        - </span>name</span>: </span>external-dns
</span>          </span>image</span>: </span>registry.k8s.io/external-dns/external-dns
</span>          </span>args</span>:
</span>            - </span>--source=service
</span>            - </span>--source=ingress
</span>            - </span>--registry=noop
</span>            - </span>--provider=webhook
</span>            - </span>--webhook-provider-url=http://localhost:5580
</span>        - </span>name</span>: </span>webhook
</span>          </span>image</span>: </span>ghcr.io/roosmaa/external-dns-technitium-webhook
</span>          </span>env</span>:
</span>            - </span>name</span>: </span>RUST_LOG
</span>              </span>value</span>: "</span>external_dns_technitium_webhook=info</span>"
</span>            - </span>name</span>: </span>LISTEN_PORT
</span>              </span>value</span>: "</span>5580</span>"
</span>            - </span>name</span>: </span>TECHNITIUM_URL
</span>              </span>value</span>: "</span>http://technitium-dns-dashboard.dns.svc.cluster.local:5380</span>"
</span>            - </span>name</span>: </span>ZONE
</span>              </span>value</span>: "</span>roosmaa.net</span>"
</span>          </span>envFrom</span>:
</span>            - </span>secretRef</span>:
</span>                </span>name</span>: </span>technitium-dns
</span>          </span>resources</span>:
</span>            </span>requests</span>:
</span>              </span>cpu</span>: </span>1m
</span>              </span>memory</span>: </span>10Mi
</span>          </span>readinessProbe</span>:
</span>            </span>httpGet</span>:
</span>              </span>port</span>: </span>5580
</span>              </span>path</span>: </span>/health
</span>            </span>failureThreshold</span>: </span>1
</span>
</span>---
</span>kind</span>: </span>Secret
</span>type</span>: </span>Opaque
</span>apiVersion</span>: </span>v1
</span>stringData</span>:
</span>  </span>TECHNITIUM_USERNAME</span>: </span>admin
</span>  </span>TECHNITIUM_PASSWORD</span>: </span>admin
</span>metadata</span>:
</span>  </span>name</span>: </span>technitium-dns
</span>  </span>namespace</span>: </span>external-dns
</span>
</span></code></pre>
The external-dns-technitium-webhook takes the following environment variables:</p>

TECHNITIUM_URL</code> - the URL for the Technitium API; in my case it's the cluster internal service hostname, with the default port.</li>
TECHNITIUM_USERNAME</code> - username for authenticating with Technitium API.</li>
TECHNITIUM_PASSWORD</code> - password for authenticating with Technitium API.</li>
ZONE</code> - the DNS zone that the connector is expected to use, it will automatically create a zone of Forward type, if it doesn't exist in the server already.</li>
DOMAIN_FILTERS</code> - a list of domain suffixes to support, ;</code> separated (e.g., .foo.roosmaa.net;.bar.roosmaa.net</code>).</li>
</ul>
Be sure to check out the GitHub repository</a> for up-to-date usage information on the project. And if there's something missing, don't hesitate to open a PR.</p>


Upcycled 8-bay disk station
2024-11-15T00:00:00+00:00
My janky Ceph setup</a> with cheapest consumer grade SSDs started to get way slower and noticable than I had hoped.</p>
In terms of simplicity, I started thinking of moving all the disks to a single machine and sharing them out from there, i.e. a more traditional NAS box. This single machine would become the single-point of failure, but I think I can accept that.</p>
I was already looking at parts I would have to source to build this new machine that would allow me to hook up all the 6 SSDs to it, when on a whim I checked my desktop PC motherboard. To my surprise, the MSI H270 Tomahawk Artic</a> gaming motherboard had 6 SATA3 ports. And it even supported hot-swap.</p>
Without any further ado, I decided to not spend more money on new hardware and instead repurpose my desktop as the new homelab node for storage. After spending some time on figuring out how ZSF on Talos</a> works, I soon got everything working.</p>
</p>
I knew that I had to prepare for the inevitable future, when one of those cheap SSDs fails. And with all the drives mounted inside the case, replacing the faulty one would be an undertaking.</p>
Upcycle a backplane?</h2>
I got the wild idea that maybe I can find some cheap server backplane on eBay and repurpose it for my needs. Most used server parts are a bit harder to find (and more expensive) on EU eBay, so I didn't know what to expect.</p>
After some searching I managed to find a HP DL380 G9 backplane PCB for a very reasonable price of 10€. From the pictures, it looked like something that would fit the bill - it had two standard SFF-8087 ports, and a 6-pin molex connector for power (presumably).</p>


</p>
</p>
</p>


</div>
While I was waiting for the backplane to arrive, I did some preliminary research about it on the internet. Luckily someone already had mostly mapped out</a> the molex connector pins - 2x 12V, 2x GND and 2 unknown pins. According to another source</a>, one or the other of the 2 unknown pins are unconnected depending on where the backplane was in the original server.</p>
If these 2 unknown pins were some form of data pins, which were required to make this backplane work, I would be out of luck. It would be difficult to reverse engineer these without access to the original servers from where these backplanes were from. However, I was hoping they were optional or something simple, like pulling the voltage up/down to constant levels.</p>
Another big question mark for me was the caddy connectors below the SATA ports. In theory, based on what I could find on the internet, they are used to blink the LEDs on the front of the caddies on the front of the server. But the question lingering in my mind was, if they are also used as some sort of switch to indicate that the disk has been properly inserted. If that were the case, I would have to find a way to bypass that.</p>
As the main input voltage for this board was 12V, the easiest option would be to power it from the 12V PCIe connector that usually powers GPUs. Of course the connector on the backplane was not directly compatible with the PCIe connector. I would have to make my own cable. I ordered some sacrificial PCIe cables from AliExpress that I could splice together with the backplane cable that came with the backplane.</p>
The disk station is born</h2>
In the mean time, I could get started with designing an enclosure for the PCB. I wanted to mount it on top of my PC case, and make the disks easily swappable. As there were holes in the PCB for airflow, I added a grille to the enclosure design as well. Mostly because it looked better, but also just in case if I ever needed to quickly introduce some active cooling there.</p>
</p>
By the time the cables arrived from China, I had already finished 3d printing the enclosure and mounting the PCB in it.</p>
I spliced together the power cable, leaving the unknown pins floating. I verified with a multimeter that all the pins were connected to correct ones and was ready to test it out.</p>
I disconnected all the drives from the computer and connected the SATA ports from the motherboard to the SFF-8087 ports on the backplane, and connected the PCIe power connector from the PSU to the backplane. I powered up the computer. And... nothing blew up. The computer booted just fine.</p>
But the disks in the backplane weren't being recognized. I took out the multimeter again, and started probing. The backplane was getting 12V input voltage, and the it was bumping it down to 5V and 3.3V for the SATA connectors.</p>
Since the power seemed to be fine, it must've been something related to data. Either the 2 unknown pins or the caddy connectors, I thought. I disconnected the backplane from the computer and put the disks back as they were for the time being.</p>
I needed to do more research. I was looking up the datasheets for the chips on the backplane to get any hints for the unknown pins in the molex connector. Nothing suggested that these pins would cause the drives not to work. The caddy connectors seemed to be unlikely as well, as from the photos I could find online, the flex PCB inside the caddies seemed fairly simple, so I was guessing there's no chips on there and the caddy connectors drive the LEDs directly.</p>
Amidst my web searching, I came across a post on one forum, where I learned that SFF-8087 breakout cables exists in two flavours - forwards and backwards. Since it was my first time working with SFF-8087 format, I hadn't even considered this. It seemed like the most likely cause for my issues. I ordered the backwards cable that I needed, and waited.</p>
Once the new cable arrived, I hooked everything up again and this time, the drives were being recognized by the motherboard and everything worked. I'm quite happy how it turned out.</p>


</p>
</p>


</div>


Jetpack Halloween costume for my daughter
2024-10-31T00:00:00+00:00
Past month and a half my 4-year-old has been fixated with rockets and flying. At some point, I did the mistake of telling her to pretend our car was a rocket when we were going over a bridge. Now every time we're going up a hill or over speed bumps she demands I drive faster to get "lift-off".</p>
Since rockets and flying was all she could talk about, I got inspired to make her a jetpack for Halloween. Like the one that Skye has in her favourite cartoon (Paw Patrol).</p>
About a week later, I had finished the modelling in Freecad.</p>


</p>
</p>
</p>


</div>
The plan was to add straps to it and wear it like a backpack. And if time allowed, add some LEDs into the thrusters for some light effects.</p>
Let the printing begin!</h2>
I had some old white PLA around, which I wanted to use up. That meant I had to budget time for the post-processing, I had about 2 weeks until Halloween. Seemed doable. Even when considering I hadn't really done any post-processing of 3D prints before. I planned for 2 coats of primer, and ideally 2 coats of paint.</p>
The circular LEDs from AliExpress arrived suprisingly fast. I had only managed to print out about half of the pieces by that time. So, I dug out some Rasperry Pi Picos I had bought in bulk a few years before and started figuring out the light show.</p>
</video>
Once the lights were sorted, printing of all the pieces had also finished. I could get started with the post-procesing.</p>
Sand, prime, repeat...</h2>
I started sanding, but quickly decided to switch to wet sanding, as there was too much plastic particles that were just getting gumbled up with the heat generated. This decision meant, that I had to also put aside time for the pieces to dry completely. I was already getting concious of the time that I had left. I thought, I'd speed up the drying process a bit, by chucking the sanded pieces into my filament dryer for a bit. Only later I would come to learn of my hubris.</em></p>
Everything seemed to go well, but without prior experience doing this, I had no idea if I was right or if it was just wishful thinking. I did know that I would get a definitive answers once the first paint layer went on.</p>
</p>
I applied the first layer of primer, and waited it to dry.</p>
For some reason I had picked up transparent primer for this. I guess, I thought it wouldn't matter much, as the layer lines would be filled in regardless of what color it was. As soon as I started sanding again, I realised it would've been immensly helpful to see how much of the primer was being sanded off.</p>
Having finished the 2nd sanding pass, I again threw the pieces into my filament dryer for a bit. When I took them out, I noticed something.</p>
Oh, ****!</h2>
Upon closer inspection, I realised what I was seeing. I had screwed up... bad! The pieces were deformed. All of the previous days of work, down the drain.</p>
I knew there was no way I could use these pieces for the final costume anymore, but I still wanted to see how well I had done with my sanding. Instead of applying the 2nd layer of primer, as I had planned, I painted them instead.</p>
Once the paint had dried, I saw that my sanding hadn't been that great either. Even if they hadn't gotten bent in the dryier, I most likely wouldn't have been happy with the outcome.</p>
</p>
Plan B?!</h2>
I still had 4 days until Halloween. I had to figure something out. My daughter was already well aware what I was working on, so not delivering wasn't an option. I didn't have time or motivation to do post-processing again. Which left me with one option - getting new filament in correct colors.</p>
I ordered black and pink filament off of Amazon, and luckily they managed to deliver it the next day. I started printing everything again. After 48 hours of non-stop printing, it was done.</p>
Halloween of 2024 was saved.</p>
</p>


Setting up ZFS on Talos
2024-09-21T00:00:00+00:00
Talos, being an immutable distro, is amazing, but it does come with a caveat. When it's time to deviate from the defaults, it involves some additional steps. Talos has extension support for those cases. There's also a bunch of official/community extensions ready to go. Rolling your own is also possible, but involves a bit of a learning curve.</p>
I wanted to convert one of my Kubernetes nodes into a ZFS-based storage box. Luckily for me, Talos has a community-maintained ZFS extension</a> to get the ZFS kernel module and userland tools installed on the node. Unluckily, I wasn't able to locate sufficient documentation about it to get it working on first try. After a bunch of trial and error (and head banging), I was able to get things figured out.</p>
Configuring the Talos node</h2>
To customize the extensions included in the Talos node, one needs use a purpose built install images. There's two options - using Talos Factory</a> (the easy way), or using imager</a> to do it locally.</p>
Using Talos Factory web interface is very straightforward, just need to select the siderolabs/zfs</code> under the system extensions. Doing it in Yaml and using the HTTP API is as easy:</p>
customization</span>:
</span>  </span>systemExtensions</span>:
</span>    </span>officialExtensions</span>:
</span>      - </span>siderolabs/zfs
</span></code></pre>
Once you have the installation image, you can slot it into your Talos node configuration, like so:</p>
machine</span>:
</span>  </span>install</span>:
</span>    </span>image</span>: </span>https://factory.talos.dev/image/4dd8e3a8b6203d3c14f049da8db4d3bb0d6d3e70c5e89dfcc1e709e81914f63c/v1.8.3/metal-amd64.iso
</span></code></pre>
But, turns out, that on it's own is not enough to get everything working. This is the part that made me lose several hours. Even though the zfs.ko</em> module is present on the file-system, it isn't loaded in the kernel. In order to make that happen, one needs to tweak the node configuration once more and use the machine.kernel.modules</a> list to explicitly include zfs.</p>
machine</span>:
</span>  </span>install</span>:
</span>    </span>image</span>: </span>https://factory.talos.dev/image/4dd8e3a8b6203d3c14f049da8db4d3bb0d6d3e70c5e89dfcc1e709e81914f63c/v1.8.3/metal-amd64.iso
</span>  </span>kernel</span>:
</span>    </span>modules</span>:
</span>      - </span>zfs
</span></code></pre>
After the above configuration has been applied</a> to the node, the regular upgrade</a> procedure will make everything ready for use.</p>
Accessing the ZFS utilities</h2>
Even though the siderolabs/zfs extension includes the zfs tools (zfs, zpool, ...) on the node, using them is more involved as Talos doesn't support executing adhoc commands on the hosts directly. You need a root shell container on the node and execute the tools in the correct kernel namespace.</p>
First, we need create a root shell that we can exec into. Depending on your needs it may be better to use DaemonSet to get the shell on to multiple nodes, but in case of a single node, we can just launch a simple Pod, like such:</p>
apiVersion</span>: </span>v1
</span>kind</span>: </span>Pod
</span>metadata</span>:
</span>  </span>name</span>: </span>zfs-shell
</span>spec</span>:
</span>  </span>nodeName</span>: </span>TARGET_NODE_NAME </span># TODO
</span>  </span>hostIPC</span>: </span>true
</span>  </span>hostNetwork</span>: </span>true
</span>  </span>hostPID</span>: </span>true
</span>  </span>containers</span>:
</span>    - </span>command</span>: ["</span>sleep</span>", "</span>infinity</span>"]
</span>      </span>image</span>: </span>debian
</span>      </span>name</span>: </span>shell
</span>      </span>securityContext</span>:
</span>        </span>privileged</span>: </span>true
</span></code></pre>
Afterwards we can run zfs tools via the nsenter command like so:</p>
kubectl</span> exec pod/zfs-shell -- \
</span>  nsenter --mount=/proc/1/ns/mnt -- \
</span>  zpool status
</span></code></pre>
ZFS backed Persistent Volumes</h2>
Now we have a ZFS capable node in our cluster with a bunch of disks attached to it. To make this storage available in Kubernetes, we can install OpenEBS</a> and make use of its Local PV ZFS storage engine.</p>
First, we need to create the ZFS pool that will be used by OpenEBS. In my case, I had 6 disks and wanted to create a RAIDz2 pool on them.</p>
kubectl</span> exec pod/zfs-shell -- nsenter --mount=/proc/1/ns/mnt -- \
</span>  zpool create -m legacy -f zfspv-pool raidz2 \
</span>  /dev/disk/by-id/{DISK1,DISK2,DISK3,DISK4,DISK5,DISK6}
</span></code></pre>
By default ZFS wants to mount the main pool filesystem under some directory in the host. Instead, we can use the -m legacy</code> parameter to tell ZFS to leave the mounting to us. When OpenEBS is creating new filesystems in the pool, it is also using the legacy option, meaning there's no requirement for the main pool to be mounted on the host side either.</p>
Once the pool is created, the OpenEBS can be easily installed via their Helm chart</a>. After which, some ZFS storage clases need to be defined that are suitable for your workloads. Below are two basic storage classes I use.</p>
# Storage class for random application files
</span>apiVersion</span>: </span>storage.k8s.io/v1
</span>kind</span>: </span>StorageClass
</span>metadata</span>:
</span>  </span>name</span>: </span>host-zfs-standard
</span>provisioner</span>: </span>zfs.csi.openebs.io
</span>allowVolumeExpansion</span>: </span>true
</span>parameters</span>:
</span>  </span>recordsize</span>: "</span>128k</span>"
</span>  </span>compression</span>: "</span>lz4</span>"
</span>  </span>dedup</span>: "</span>off</span>"
</span>  </span>fstype</span>: "</span>zfs</span>"
</span>  </span>poolname</span>: "</span>zfspv-pool</span>"
</span># Use allowedTopologies: in case ZFS is only available
</span># on some of the nodes in the cluster.
</span>
</span>---
</span># Storage class for file storage (documents, photos, videos, etc)
</span>apiVersion</span>: </span>storage.k8s.io/v1
</span>kind</span>: </span>StorageClass
</span>metadata</span>:
</span>  </span>name</span>: </span>host-zfs-files
</span>provisioner</span>: </span>zfs.csi.openebs.io
</span>allowVolumeExpansion</span>: </span>true
</span>reclaimPolicy</span>: </span>Retain
</span>parameters</span>:
</span>  </span>recordsize</span>: "</span>1M</span>"
</span>  </span>compression</span>: "</span>lz4</span>"
</span>  </span>dedup</span>: "</span>off</span>"
</span>  </span>fstype</span>: "</span>zfs</span>"
</span>  </span>shared</span>: "</span>yes</span>" </span># to enable ReadWriteMany access mode
</span>  </span>poolname</span>: "</span>zfspv-pool</span>"
</span></code></pre>
To enable taking snapshots of the ZFS backed PVs, the a volume snapshot class also needs to be defined.</p>
kind</span>: </span>VolumeSnapshotClass
</span>apiVersion</span>: </span>snapshot.storage.k8s.io/v1
</span>metadata</span>:
</span>  </span>name</span>: </span>host-zfs-snapshot
</span>driver</span>: </span>zfs.csi.openebs.io
</span>deletionPolicy</span>: </span>Delete
</span></code></pre>
Monitoring with Netdata</h2>
The ZFS backed PVs should be usable now, but we're effectively flying blind. When a disk fails, we wouldn't know about it unless we manually checked the zpool status</code> every so often. And if enough disks fail, it's sayonara</em> to our data.</p>
There are some Prometheus metrics exporters for ZFS, but I didn't explore that avenue, as Netdata</a> has built-in support for ZFS data. But it doesn't work out of the box on Talos, due to the zpool command being difficult to access from the container.</p>
I ended up creating a small utility (zfs-http-query</a> that runs as a DaemonSet on the ZFS nodes and exposes zpool data via an unix socket. This allows pods that have access to that socket to query zfs data from an unprivileged container.</p>
With zfs-http-query in place, the Netdata Helm values.yaml can be updated to make the built in ZFS support work:</p>
child</span>:
</span>  </span>configs</span>:
</span>    </span>zfspool</span>:
</span>      </span>enabled</span>: </span>true
</span>      </span>path</span>: </span>/etc/netdata/go.d/zfspool.conf
</span>      </span># tell netdata zfspool integration to use the zpool shim from zfs-http-query
</span>      </span>data</span>: </span>|
</span>        jobs:
</span>          - name: zfspool
</span>            binary_path: /opt/zfs-http-query/bin/zpool
</span>  </span>extraVolumeMounts</span>:
</span>    - </span>name</span>: </span>opt-zfs-http-query
</span>      </span>mountPath</span>: </span>/opt/zfs-http-query
</span>      </span>readOnly</span>: </span>true
</span>    - </span>name</span>: </span>run-zfs-http-query
</span>      </span>mountPath</span>: </span>/var/run/zfs-http-query
</span>      </span>readOnly</span>: </span>true
</span>  </span>extraVolumes</span>:
</span>    - </span>name</span>: </span>opt-zfs-http-query
</span>      </span>hostPath</span>:
</span>        </span>type</span>: </span>DirectoryOrCreate
</span>        </span>path</span>: </span>/opt/zfs-http-query
</span>        </span># contains bin/zpool and bin/zfs shims
</span>    - </span>name</span>: </span>run-zfs-http-query
</span>      </span>hostPath</span>:
</span>        </span>type</span>: </span>DirectoryOrCreate
</span>        </span>path</span>: </span>/var/run/zfs-http-query
</span>        </span># contains the unix socket used by the shims
</span></code></pre>
With this in place, Netdata will have access to the ZFS pool data and can show graphs about the pool health. And send notifications (if properly configured) when the pool state becomes degraded (i.e. a disk failure).</p>
Sharing storage across nodes</h2>
Unlike Ceph or other distributed storage solutions, the OpenEBS based ZFS is tied to the node. When a PV is created on a certain node, it can only be accessed on that specifc node and it cannot be migrated to another node.</p>
I found csi-driver-smb</a> project that allows mounting of samba shares to containers. This allows working around the above limitation by hosting a samba server on the ZFS enabled node, and using the csi-driver-smb to access it on any other node in the cluster.</p>
</p>


Unexpiring Tailscale auth keys
2024-09-09T00:00:00+00:00
Ideally when running Tailscale in kubernetes one should use ephemeral configuration for the keys. But regular auth keys are limited to 90 days of validity, which means every 3 months someone would have to rotate them.</p>
In a small homelab, where keeping the software up to date already is a chore, I wouldn't want to add another manual action to the todo list. A workaround I was using, was to use non-ephemeral auth keys combined with persistent volumes. This allowed me to log the workloads into Tailscale with a valid key, and as long as the state was persisted on Ceph, I wouldn't have to worry.</p>
The problem was that my janky Ceph setup was turning out to be unbearably slow for "critical" components. Various issues with Tailscale pods starting up due to slow Ceph, or when I was abusing Ceph by restarting nodes randomly, the Tailscale containers would temporarily become unresponsive due to the underlying storage being unavailable.</p>
I wanted to move away from Ceph dependency in the Tailscale container, so I started looking for ways to remove the manual chore part from the key rotation.</p>
On various reddit posts and github issues Tailscale employees were referring people to the OAuth clients. The client secrets don't expire and can be used to generate device auth keys on-demand.</p>
Initially I was thinking of creating a cronjob to update the Tailscale kubernetes Secret every hour or so. Or perhaps an easier version to setup init-container to do the auth key provisioning.</p>
But then I saw a small mention (on reddit) to a "well hidden" documentation page - registering new nodes using OAuth credentials</a>. Turns out, the OAuth secrets can be used directly as device auth-keys. Meaning, the Tailscale app automatically provisions the auth keys without any extra effort on our side.</p>
Creating the OAuth client</h2>
Unlike with auth keys, when using OAuth client to authenticate, Tailscale requires the usage of device tags. Let's make sure that we have the approprate tag created in our Tailscale ACL</a>. I've named my tag tag:subnet-router</code>.</p>
{
</span>    "</span>tagOwners</span>": {
</span>        "</span>tag:subnet-routes</span>": ["</span>autogroup:admin</span>"]
</span>    }
</span>}
</span></code></pre>
Then, under Settings → Tailnet Settings → OAuth clients</a>, we can create the new OAuth client.</p>
</p>
Pick a suitable description for the client, then choose the appropriate scopes. Since we're only using the OAuth client to generate auth keys, we only need the auth_keys</code> scope (read & write access for auth keys). Select the tag you want the device to get after it has authenticated.</p>

	
		</i>Note</p>
	
Tailscale updated their OAuth client scopes to be more granular on 14/10/2024. Before that, it used to be the devices</code> scope that was needed for this.</p>

</blockquote>
</p>
</p>
Configuring the Tailscale docker container</h2>
If you're using the Tailscale docker container, you need to pass in the OAuth client secret using the TS_AUTHKEY</code> environment variable.</p>
There's no dedicated environment variable for advertise tags, but the TS_EXTRA_ARGS</code> can be used for that, by passing in the full command line flag, as such --advertise-tags=tag:subnet-router</code>.</p>

Example of a kubernetes Deployment</summary>
This is how I deploy my subnet-router Tailscale containers on kubernetes. It retrieves the client secret from the tailscale-subnet-router</code> secret.</p>
apiVersion</span>: </span>apps/v1
</span>kind</span>: </span>Deployment
</span>metadata</span>:
</span>  </span>name</span>: </span>subnet-router
</span>  </span>labels</span>:
</span>    </span>app</span>: </span>tailscale-subnet-router
</span>spec</span>:
</span>  </span>replicas</span>: </span>2
</span>  </span>selector</span>:
</span>    </span>matchLabels</span>:
</span>      </span>app</span>: </span>tailscale-subnet-router
</span>  </span>template</span>:
</span>    </span>metadata</span>:
</span>      </span>name</span>: </span>subnet-router
</span>      </span>labels</span>:
</span>        </span>app</span>: </span>tailscale-subnet-router
</span>    </span>spec</span>:
</span>      </span>containers</span>:
</span>        - </span>name</span>: </span>tailscale
</span>          </span>image</span>: </span>ghcr.io/tailscale/tailscale:latest
</span>          </span>env</span>:
</span>            - </span>name</span>: </span>TS_AUTHKEY
</span>              </span>valueFrom</span>:
</span>                </span>secretKeyRef</span>:
</span>                  </span>name</span>: </span>tailscale-subnet-router
</span>                  </span>key</span>: </span>authKey
</span>            - </span>name</span>: </span>TS_KUBE_SECRET
</span>              </span>value</span>: ""
</span>            - </span>name</span>: </span>TS_USERSPACE
</span>              </span>value</span>: "</span>true</span>"
</span>            - </span>name</span>: </span>TS_ROUTES
</span>              </span>value</span>: </span>192.168.1.0/24
</span>            - </span>name</span>: </span>TS_EXTRA_ARGS
</span>              </span>value</span>: </span>>-
</span>                --advertise-tags=tag:subnet-router
</span>                --accept-dns=false
</span>                --stateful-filtering=true
</span>          </span>resources</span>:
</span>            </span>requests</span>:
</span>              </span>cpu</span>: </span>100m
</span>              </span>memory</span>: </span>50Mi
</span>          </span>securityContext</span>:
</span>            </span>runAsUser</span>: </span>1000
</span>            </span>runAsGroup</span>: </span>1000
</span>      </span>securityContext</span>:
</span>        </span>fsGroup</span>: </span>1000
</span></code></pre>
</details>


Routing Talos cluster traffic over specific NIC
2024-09-02T00:00:00+00:00
When your Talos nodes have multiple NICs attached to them and you'd like to route in-cluster traffic over a specific NIC. How would you go about doing that?</p>
There can be various reasons why you'd want to do that. For example, the NICs available could offer differ speeds, and the with the given workloads it could make sense to route in-cluster traffic through the faster one and the egress over the slower one.</p>
The following assumes that Cilium CNI is being used and it has been configured to use native routing</a>. Let's also assume that the nodes are configured in the following fashion:</p>
</p>
First, we need to assign both of the network interfaces IPs in different subnets. For example, all the NIC-1's would get 10.1.0.0/24 and all the NIC-2's would get 10.2.0.0/24. The Talos patch for the node-1 would be the following:</p>
machine</span>:
</span>  </span>network</span>:
</span>    </span>interfaces</span>:
</span>      - </span>interface</span>: </span>enp0s31f6
</span>        </span>addresses</span>:
</span>          - </span>10.1.0.1/24
</span>        </span>routes</span>:
</span>          - </span>network</span>: </span>0.0.0.0/0
</span>            </span>gateway</span>: </span>10.0.0.254 </span># for example
</span>      - </span>interface</span>: </span>enp2s0
</span>        </span>addresses</span>:
</span>          - </span>10.2.0.1/24
</span></code></pre>
At this point, the nodes are reachable to each other via either of the links. The traffic may or may not flow through your desired NIC. To make it explicit, we need to tell kubelet which of the subnets it is meant to use. We can do that with the following patch file:</p>
machine</span>:
</span>  </span>kubelet</span>:
</span>    </span>nodeIP</span>:
</span>      </span>validSubnets</span>:
</span>        - </span>10.2.0.0/24
</span></code></pre>
And that's it. After applying the configuration to nodes, the in-cluster traffic will now use NIC-2 and all the egress traffic will get routed via NIC-1. It can be verified with talosctl pcap</code> command.</p>


Hello, Homelab!
2024-03-10T00:00:00+00:00
About a year ago, my homelab journey began. It was a combination of the Self-Hosted Show</a> and the Homelab Show</a> podcasts and ServeTheHome</a> YouTube channel that gave me the push needed to take the plunge.</p>
I knew I wanted my lab to be Kubernetes based. I also wanted my lab to help me move some of my data off the cloud. I had noticed that Google Photos was corrupting some of the older pictures/videos, so I wanted to take ownership of keeping my data safe.</p>
As I wanted to learn more about running a Kubernetes cluster in HA configuration, I needed a minimum of 3 machines. I set up a saved search on eBay to find some old 1L PCs that would be decent for my needs. Most of the good deals were based in US or UK, with ridiculous shipping prices, though.</p>
After a while, I got a notification. A seller in France was selling a lot of 3x ThinkCentre m710q. The specs weren't to my liking, though. However, after some research online, I found that they could be upgraded to something decent. The most performant CPU for that system was the i7-7700T, which could be found in abundance on AliExpress. The M.2 slot originally meant for the Wi-Fi card could be used for a 2.5G NIC.</p>
The eBay listing for the ThinkCentre lot was a reasonable 230€. However, after upgrading the CPUs, maxing out the RAM, adding the 2.5G NICs and getting new NVMe drives it totalled up to around 1400€. Later in the year I would also get a bunch of additional 3.5" SSDs for storage, which pushed the total slightly over 2200€.</p>
Proxmox and Ansible</h2>
Proxmox was all the rage on all the podcasts I was listening to, so I wanted to take it for a spin as well. Proxmox</a> is a Debian based OS with a web UI for managing VMs easily. It was quick to set-up and soon I had some Debian VMs running k3s</a>. As I was playing around with Proxmox (PVE), I learned about Ceph</a> - a distributed storage solution. I set it up as well, in a completely un-advised way possible - using a partition on my NVMe drive alongside the OS partition. Tested out the high availability</a> feature in PVE and all the other major features as well.</p>
So far everything had been set-up manually. I knew, I wanted something codified, so that I could recreate everything quickly without having to remember all the little details that went into setting up the machines. I picked up Ansible</a> for that.</p>
Over the next several weeks, I created Ansible playbooks for setting up everything I had done manually up until now. For both Proxmox and the VMs running k3s. There were several things with Ansible that didn't jive with me too much, one of which being the fairly slow playbook execution, even after having spent a bunch of time reworking everything to be fast, in theory. But I was willing to live with these grievances, I wasn't going to redo everything in some other system (Salt or Chef)...</p>
I planned to set up Semaphore</a> to execute the playbooks from Git in an automated way. That would allow me to update the lab from any of my computers. There was a challenge there. In certain conditions, my playbooks would automatically restart the Proxmox hosts. However, if the Semaphore is running in a VM on one of the hosts, the playbook would get interrupted. I kind of solved it by running Semaphore in a separate VM that was setup in PVE to do live-migration</a>. That way, the Semaphore VM would survive the restarts of the underlying hosts. In the end, I wasn't happy with Semaphore features (at the time), and decided not to use it going forward.</p>
Going bare metal</h2>
Out of all the things I wanted to do with my homelab, nothing had a hard-requirement on running in a VM. I decided to eliminate Proxmox from the equation all together and to double down on just Kubernetes. Less things to maintain seemed like a good idea.</p>
After reworking my Ansible playbooks a bit, I was ready to deploy bare metal Debian with k3s. I started to focus more on setting up the Kubernetes side of things. While doing that, I came across Talos and it piqued my interest.</p>
Talos</a> is a minimal and immutable Linux distribution meant for one thing and one thing only - running Kubernetes. By this time I had already found several ways to shoot myself in the foot managing Debian and k3s via Ansible and Talos seemed to not have these pain-points. After few experiments in VMs on my main computer, I was convinced it was time to redo my 3 ThinkCentre nodes yet again.</p>
Since Talos is immutable, I had no further use for Ansible and I stopped using it all together. Everything was now configured on the Kubernetes side with YAML.</p>
After some 8 months of running Talos I couldn't be happier. Whenever there's a new version, I just let Talos do a fresh installs on all the nodes and everything just works.</p>
Longhorn? or Ceph?</h2>
One of the first things I had to figure out in the fresh Kubernetes cluster was storage. Applications need a way to store user (my) data. Longhorn</a> was the most recommended option in various subreddits when I got started. It was supposed to be easy to setup and low maintenance.</p>
I did get it running fairly quickly. But since I had learned a bit about Ceph (while experimenting with PVE), I knew I wouldn't be happy with what Longhorn had to offer. For one, CephFS was something I wanted to make use of for shared storage between different services. And the sentiment online about Longhorn had slowly started to change. People were telling stories about how they had lost some of their data.</p>
I ended up going with Rook</a>, a Kubernetes operator for managing Ceph. It was a bit of a learning curve getting Rook up and running, but everything has worked fairly well since then. Some random troubleshooting and bug hunting, but none of which has ever brought down storage in my cluster. I'm amazed at the resilience.</p>
Having said that, it is not the fastest storage setup. And it's completely my own doing. I'm using the cheapest consumer SSDs I could get my hands on. And most of them are connected to the machines via USB, since I have no way to add additional SATA ports to my 1L boxes. Over USB? What is he thinking!? He mad?</q> Hear me out, my primary use for Ceph storage is to store my personal data. It's more important for me to have the data safe rather than it being fast. I don't plan on running a database on Ceph backed block storage, so chasing speed is not something I want to do here.</p>
Each 1L box, has one internal SATA port available that I'm taking advantage of, and I'm also connecting 2x 3.5" drives over USB3. That's 3 drives per node and 9 drives in total. This allows me to store valuable data with 3x replication and less valuable data (e.g. DVD backups) with 4+2 EC (erasure coding). The EC gives me slightly more usable storage out of my disks.</p>
</p>
Over the past 6 months, I've simulated various failure conditions (kill power to a single host, pull the drive from the host, kill the data on the some of the drives) and Ceph has not complained one bit and has successfully kept my data safe.</p>
One thing to keep in mind, is that currently I'm only using slightly above 1% of the available storage. I'm certain that as the cluster fills up more, it will have a few more surprises for me in store. But by that time, I may already be running it on better hardware.</p>
And the databases?</em></p>
I'm very much Postgres biased when it comes to databases, and I can run that off of local-path storage. Postgres itself takes care of replication in a HA setup (shout out to CloudNativePG</a>. Combined that with regular backups to Ceph backed S3, I feel fairly confident there's not going to be any data loss there either.</p>
That said, none of the databases so far contain critical personal data. So, if I am proven wrong, it won't be too painful of a lesson.</p>
All the other Kubernetes cohabitants</h2>
In addition to Ceph, I have a whole bunch of other software running in the cluster to make it "usable" for regular applications:</p>

Cilium</a> - for networking</li>
ArgoCD</a> - for deploying software into the cluster</li>
sealed-secrets</a> - for storing encrypted secrets in Git</li>
OpenEBS</a> - for local (host-path storage)</li>
cert-manager</a> - for generating LetsEncrypt SSL certs for services</li>
external-dns</a> - for automatically setting up DNS for services</li>
node-feature-discovery</a> & intel-device-plugins</a> - for passing integrated GPUs into pods</li>
traefik</a> - as the main ingress controller</li>
tailscale</a> - secure access to the cluster from anywhere</li>
</ul>
And then there's the user-facing applications, of which there aren't many currently:</p>

Pi-hole</a> - provides DNS services for the home LAN and also on my phone via tailscale</li>
Traccar</a> - keeps track of my car with notifications for unexpected events (crashes, towing, etc)</li>
Jellyfin</a> - for enjoying backed up DVDs in a more modern way</li>
Syncthing</a> - for syncing notes across devices & keeping a copy of all the photos from the phones on the cluster</li>
</ul>
Tunnelling into the cluster</h2>
Even though most of my devices can access the services through tailscale, there are some trackers that need to connect to traccar via the public web. Ideally, I would've wanted to use Cloudflare Tunnels or similar services, but these only support HTTP traffic. The trackers, however, use various different binary protocols.</p>
The option I settled on, was to get the cheapest VPS I could find. Then installing just tailscale package and some firewall forwarding rules on it. By enabling automatic updates, it means this setup is effectively configure once and forget.</p>
Keeping everything up to date</h2>
I knew that I had to develop the habit of keeping most of the software as recent as possible. Otherwise, upgrading severely out of date software would become such a headache I would never want to do that.</p>
I started off keeping track of all the software in the cluster using NewReleases</a>. I would get a weekly digest of all the updates. After which, I have to find an evening where I would review the changelogs for potential dangers to look out for and update everything.</p>
A lot of the projects mentioned above, are very actively developed. This meant, that every Monday, the weekly email from NewReleases, was fairly feature-packed. It didn't take long, for the update ritual to become a chore I didn't look forward to. It was also one of the reasons why I effectively stopped deploying new services on the cluster, as I didn't want to increase the burden on myself.</p>
At some point, I decided to rethink how I approached updates. I restructured my repository containing the kubernetes manifests, and introduced Renovate</a>. Previously, I had to update all the dependencies at once to be able to delete the digest email from NewReleases. Otherwise, I would've lost track of some updates. With Renovate, each of the updates gets its own pull-request. And depending on the service & the changelog I can decide to postpone certain updates for evenings where I have time to take care of them. The other - low risk - updates I can just merge and forget about.</p>
Is it all worth it?</h2>
It has been a somewhat bumpy road, so far. I've enjoyed learning about new things and tinkering with both the hardware and software side of running a homelab. What I didn't anticipate was the responsibilities of maintenance that come with running a "production" cluster - Why is the internet down? Oh, the pi-hole pods are in a crashloop</q>. This has paced me from moving more of my life onto self-hosted services.</p>
But there's still plenty I want to explore, so I'm not throwing in my towel just yet. I'll take my time with it, and won't rush myself. And, in doing so, I hope I'll eventually reach the goal of having all my data on my own hardware.</p>
If you managed to get this far, this topic is probably something you're interested in. Drop me a DM on any of the social platforms I'm on, if you want to chat.</em></p>

Mart Roosmaa

Ollama, LM Studio vs llama.cpp

ExternalDNS webhook for Technitium DNS

Upcycled 8-bay disk station

Jetpack Halloween costume for my daughter

Setting up ZFS on Talos

Monitoring with Netdata</h2> The ZFS backed PVs should be usable now, but we're effectively flying blind. When a disk fails, we wouldn't know about it unless we manually checked the zpool status</code> every so often. And if enough disks fail, it's sayonara</em> to our data.</p>

Sharing storage across nodes</h2> Unlike Ceph or other distributed storage solutions, the OpenEBS based ZFS is tied to the node. When a PV is created on a certain node, it can only be accessed on that specifc node and it cannot be migrated to another node.</p>

Unexpiring Tailscale auth keys

Routing Talos cluster traffic over specific NIC

Hello, Homelab!

Keeping everything up to date</h2> I knew that I had to develop the habit of keeping most of the software as recent as possible. Otherwise, upgrading severely out of date software would become such a headache I would never want to do that.</p>