One of the issues I faced with K3s is that there is no way for me to replicate the data everywhere, I mean the local storage is bound to the PI it created on.
One of the options out there that you can use is Longhorn, which K3s still think it does not support ARM based devices (at the time of writing), while I think v1.5 supports ARM based devices like the Raspberry PI (check their docker hub repo here).
But, I'll not be using Longhorn, at least not now 😅 instead I'll be using my NAS which comes with NFS support out of the box, so I'll not be explaining how to create your NFS server nor how to configure it, I'll assume you already have one ready.
Also, I'll assume that you have already setup your PI Cluster and that you don't need any help from me to do so, in case you want to remember how to do it, check my old post here.
As you know Raspberry PI 5 has been released recently, so if you want to upgrade your Cluster, like what I did, its the time to do so 😁.
K3s with NFS
First make sure to install the required package which is
nfs-kernel-server since I am on piOS, installing it can be as easy as running the following command
sudo apt install nfs-kernel-server
Once you got it installed and ready on all your nods, you need to create a new manifests for K3s to automatically pick it up.
You can create a file called
/var/lib/rancher/k3s/server/manifests/ and then add the following code to it:
apiVersion: v1 kind: Namespace metadata: name: default --- apiVersion: helm.cattle.io/v1 kind: HelmChart metadata: name: nfs namespace: default spec: chart: nfs-subdir-external-provisioner repo: https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner targetNamespace: default set: storageClass.name: nfs valuesContent: |- nfs: server: 192.168.68.118 path: /i-data/f01e5fea/nfs/k3s mountOptions: - nfsvers=3
Remember to replace the server address and the path to match your server configuration, otherwise it wont work.
Since my NAS does not work with NFS v4, or at least I couldn't connect to it via NFS v4 address, I need to set the mount option to be
nfsvers=3 and the set object does not get any thing other than strings, so using
<span data-token-index="5">valuesContent</span> is the best to deal with it.
valuesContent you can override the values of the provisioner as you like, and since
mountOptions accepts only array elements, this is the best way to do it.
Once you finished, wait a few second, then run the command
kubectl get sc and the result should be something like
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 22h nfs cluster.local/nfs-nfs-subdir-external-provisioner Delete Immediate true 22m