One of the issues I faced with K3s is that there is no way for me to replicate the data everywhere, I mean the local storage is bound to the PI it created on.

One of the options out there that you can use is Longhorn, which K3s says it does not support ARM 32 based devices (at the time of writing), while I think starting from v1.6 it supports ARM64 based devices like the Raspberry PI.

But, I'll not be using Longhorn, at least not now 😅 instead I'll be using my NAS which comes with NFS support out of the box, so I'll not be explaining how to create your NFS server nor how to configure it, I'll assume you already have one ready.

Also, I'll assume that you have already setup your PI Cluster and that you don't need any help from me to do so, in case you want to remember how to do it, check my old post here.

As you know Raspberry PI 5 has been released recently, so if you want to upgrade your Cluster, like what I did, its the time to do so 😁.

K3s with NFS

First make sure to install the required package which is nfs-kernel-server since I am on piOS, installing it can be as easy as running the following command

sudo apt install nfs-kernel-server

Once you got it installed and ready on all your nods, you need to create a new manifests for K3s to automatically pick it up.

You can create a file called nfs-controller.yml inside /var/lib/rancher/k3s/server/manifests/ and then add the following code to it:

apiVersion: v1
kind: Namespace
metadata:
  name: default
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: nfs
  namespace: default
spec:
  chart: nfs-subdir-external-provisioner
  repo: https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner
  targetNamespace: default
  set:
    storageClass.name: nfs
  valuesContent: |-
    nfs:
        server: 192.168.68.118
        path: /i-data/f01e5fea/nfs/k3s
        mountOptions:
        - nfsvers=3

Remember to replace the server address and the path to match your server configuration, otherwise it wont work.

Since my NAS does not work with NFS v4, or at least I couldn't connect to it via NFS v4 address, I need to set the mount option to be nfsvers=3 and the set object does not get any thing other than strings, so using <span data-token-index="5">valuesContent</span> is the best to deal with it.

Using valuesContent you can override the values of the provisioner as you like, and since mountOptions accepts only array elements, this is the best way to do it.

Once you finished, wait a few second, then run the command  kubectl get sc and the result should be something like

NAME                   PROVISIONER                                         RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)   rancher.io/local-path                               Delete          WaitForFirstConsumer   false                  22h
nfs                    cluster.local/nfs-nfs-subdir-external-provisioner   Delete          Immediate              true                   22m

Resources: