Mastodon on K3s:

  1. Part 1, the Hardware.
  2. Part 2, the Software.
  3. Part 3, the Installation.

Preparations:

Before we start we need to make sure that you have the following steps done, and I am not going to explain it as if I did, 3 parts won't be enough, so I'll assume that you have:

  1. You own a domain name.
  2. Your domain is setup in Cloudflare and that you are using their DNS/Zero Trust service.
  3. You have setup for an email service provider, like Mailgun or Postmark (I use postmark), any provider which provides SMTP is good.
  4. You have a paid account with iDrive, and you have created a public bucket with a CDN (check this post here, but you don't need to use AWS CLI to setup the permissions as iDrive team has added it as an option from the UI)

Assuming that you have followed along, and you have all your nodes connected to each other, let's start doing our magic

PS: I'll create a github repo with all the code for reference.

Redis:

This one is the easiest and simplest, we can create it using the following yml which will run redis on the 3rd node (worker-2):

---
apiVersion: v1
kind: Namespace
metadata:
  name: mastodon-server

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-pvc
  namespace: mastodon-server
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 5Gi

---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: mastodon-server
spec:
  selector:
    app: redis
  ports:
    - name: redis-port
      protocol: TCP
      port: 6379

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: mastodon-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
        name: redis
    spec:
      nodeSelector:
        kubernetes.io/hostname: worker-2
      containers:
        - name: redis
          image: redis:latest
          imagePullPolicy: Always
          args: ["--appendonly", "yes"]
          ports:
            - name: redis
              containerPort: 6379
          volumeMounts:
            - name: redis-storage
              mountPath: /data
          env:
              - name: ALLOW_EMPTY_PASSWORD
                value: "yes"
      volumes:
        - name: redis-storage
          persistentVolumeClaim:
            claimName: redis-pvc

PostgreSQL:

For the database, we will use ubuntu postgresql docker image, and we will run the instance on one of the instances preferably not the master.

As I mentioned in my post here, we will need we will need to define a storage, a port and a deployment, I don't think we needs the custom configuration since everything will be connected locally.

I'll also define a new thing, which is secrets to hide the sensitive data we are going to use.

---
apiVersion: v1
kind: Namespace
metadata:
  name: mastodon-server

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc
  namespace: mastodon-server
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 5Gi

---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: mastodon-server
spec:
  selector:
    app: postgres
  ports:
    - name: postgres-port
      protocol: TCP
      port: 5432

---
apiVersion: v1
kind: Secret
metadata:
  namespace: mastodon-server
  name: postgres-secret
type: Opaque
stringData:
  POSTGRES_USER: postgres
  POSTGRES_PASSWORD: <CHANGE-ME>
  POSTGRES_HOST_AUTH_METHOD: trust
  POSTGRES_DB: mastodon
  POSTGRES_NON_ROOT_USER: <CHANGE-ME>
  POSTGRES_NON_ROOT_PASSWORD: <CHANGE-ME>

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  namespace: mastodon-server
data:
  init-data.sh: |
    #!/bin/bash
    set -e;
    if [ -n "${POSTGRES_NON_ROOT_USER:-}" ] && [ -n "${POSTGRES_NON_ROOT_PASSWORD:-}" ]; then
      psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
        CREATE USER ${POSTGRES_NON_ROOT_USER} WITH PASSWORD '${POSTGRES_NON_ROOT_PASSWORD}';
        GRANT ALL PRIVILEGES ON DATABASE ${POSTGRES_DB} TO ${POSTGRES_NON_ROOT_USER};
      EOSQL
    fi

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: mastodon-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
        name: postgres
    spec:
      nodeSelector:
        kubernetes.io/hostname: worker-1
      containers:
        - name: postgres
          image: ubuntu/postgres:latest
          imagePullPolicy: Always
          ports:
            - name: postgres
              containerPort: 5432
          volumeMounts:
            - name: postgres-storage
              mountPath: /var/lib/postgresql/data        
          env:
              - name: POSTGRES_USER
                valueFrom:
                  secretKeyRef:
                    name: secrets
                    key: POSTGRES_USER
              - name: POSTGRES_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: secrets
                    key: POSTGRES_PASSWORD
              - name: POSTGRES_HOST_AUTH_METHOD
                valueFrom:
                  secretKeyRef:
                    name: secrets
                    key: POSTGRES_HOST_AUTH_METHOD
      volumes:
        - name: secrets
          secret:
            secretName: postgres-secret
        - name: postgres-storage
          persistentVolumeClaim:
            claimName: postgres-pvc

Make sure to go over the file and change the value of each <CHANGE-ME> with something you use, like the password, the user .. etc.

Mastodon:

Before we install Mastodon, you need to go over to Mastodon documentation and get yourself familiar with the environment variables and what they means, you can find them here.

It is important that you do so, as you will need to modify the environment variables to meet your needs.

You can generate the SECRET_KEY_BASE and OTP_SECRET using the following commands

docker run --rm -it -w /app/www --entrypoint rake lscr.io/linuxserver/mastodon secret

docker run --rm -it -w /app/www --entrypoint rake lscr.io/linuxserver/mastodon mastodon:webpush:generate_vapid_key

Once you do, you can you can change them in this big yml file:

---
apiVersion: v1
kind: Namespace
metadata:
  name: mastodon-server

---
apiVersion: v1
kind: Service
metadata:
  name: mastodon
  namespace: mastodon-server
spec:
  selector:
    app: mastodon
  ports:   
    - name: mastodon-plain
      protocol: TCP
      port: 80
    - name: mastodon-secure
      protocol: TCP
      port: 443

---
apiVersion: v1
kind: Secret
metadata:
  namespace: mastodon-server
  name: mastodon-secret
type: Opaque
stringData:
  SECRET_KEY_BASE: <CHANGE-ME>
  OTP_SECRET: <CHANGE-ME>
  VAPID_PRIVATE_KEY: <CHANGE-ME>
  VAPID_PUBLIC_KEY: <CHANGE-ME>    
  SMTP_LOGIN: <CHANGE-ME>
  SMTP_PASSWORD: <CHANGE-ME>
  SMTP_FROM_ADDRESS: <CHANGE-ME>
  AWS_ACCESS_KEY_ID: <CHANGE-ME>
  AWS_SECRET_ACCESS_KEY: <CHANGE-ME>
  S3_ENDPOINT: <CHANGE-ME>
  SMTP_FROM_ADDRESS: <CHANGE-ME>

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mastodon
  namespace: mastodon-server
  labels:
    app: mastodon 
spec:
  replicas: 3
  selector:
    matchLabels:
      app: mastodon
  template:
    metadata:
      labels:
        app: mastodon
        name: mastodon
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: mastodon
      containers:
      - name: mastodon
        image: lscr.io/linuxserver/mastodon:latest
        imagePullPolicy: Always
        ports:
          - name: plain
            containerPort: 80
          - name: secure
            containerPort: 443                        
        env:
          - name: PUID
            value: "1000"
          - name: GUID
            value: "1000"
          - name: TZ
            value: Europe/Istanbul
          - name: LOCAL_DOMAIN
            value: <CHANGE-ME>   # domain.com
          - name: WEB_DOMAIN
            value: <CHANGE-ME>   # domain.com         
          - name: REDIS_HOST
            value: redis.redis-server
          - name: REDIS_PORT
            value: "6379"
          - name: DB_NAME
            value: mastodon
          - name: DB_HOST
            value: postgres.mastodon-server
          - name: DB_USER
            valueFrom:
              secretKeyRef:
                name: db_secrets
                key: POSTGRES_NON_ROOT_USER
          - name: DB_PASS
            valueFrom:
              secretKeyRef:
                name: db_secrets
                key: POSTGRES_NON_ROOT_PASSWORD
          - name: DB_PORT
            value: "5432"
          - name: ES_ENABLED
            value: "false"
          - name: SECRET_KEY_BASE
            valueFrom:
              secretKeyRef:
                name: secrets
                key: SECRET_KEY_BASE
          - name: OTP_SECRET
            valueFrom:
              secretKeyRef:
                name: secrets
                key: OTP_SECRET
          - name: VAPID_PRIVATE_KEY
            valueFrom:
              secretKeyRef:
                name: secrets
                key: VAPID_PRIVATE_KEY
          - name: VAPID_PUBLIC_KEY
            valueFrom:
              secretKeyRef:
                name: secrets
                key: VAPID_PUBLIC_KEY
          - name: RAILS_SERVE_STATIC_FILES
            value: "false"
          - name: SMTP_SERVER
            valueFrom:
              secretKeyRef:
                name: secrets
                key: SMTP_SERVER
          - name: SMTP_PORT
            value: "587" # change this one accordenly
          - name: SMTP_LOGIN
            valueFrom:
              secretKeyRef:
                name: secrets
                key: SMTP_LOGIN
          - name: SMTP_PASSWORD
            valueFrom:
              secretKeyRef:
                name: secrets
                key: SMTP_PASSWORD
          - name: SMTP_FROM_ADDRESS
            valueFrom:
              secretKeyRef:
                name: secrets
                key: SMTP_FROM_ADDRESS
          - name: S3_ENABLED
            value: "true"
          - name: CDN_HOST
            value: <CHANGE-ME>
          - name: S3_BUCKET
            value: "mastodon"
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: secrets
                key: AWS_ACCESS_KEY_ID
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: secrets
                key: AWS_SECRET_ACCESS_KEY
          - name: S3_REGION
            value: auto  
          - name: S3_ENDPOINT
            valueFrom:
              secretKeyRef:
                name: secrets
                key: S3_ENDPOINT
          - name: S3_HOSTNAME
            value: <CHANGE-ME> # assets.domain.com
          - name: S3_FORCE_SINGLE_REQUEST
            value: "true"
          - name: S3_READ_TIMEOUT
            value: "15"
          - name: S3_OPEN_TIMEOUT
            value: "15"
        resources:
          limits:
            cpu: "1.0"
            memory: "2560Mi"
          requests:
            cpu: "0.5"
            memory: "2560Mi"      
      volumes:
        - name: db_secrets
          secret:
            secretName: postgres-secret
        - name: secrets
          secret:
            secretName: mastodon-secret

I am not going to go over each and everything as most of them are self explained, the name of each variable explain exactly what it is, all you have to do is to find the one I marked as  and change it, and/or change the one you don't like.

Cloudflare Tunnels:

It is the same thing as you read here so I won't explain it, I'll paste the yml content for reference

---
apiVersion: v1
kind: Namespace
metadata:
  name: cloudflare-server

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudflare
  namespace: cloudflare-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cloudflare
  template:
    metadata:
      labels:
        app: cloudflare
        name: cloudflare
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: cloudflare
      containers:
        - name: cloudflare
          image: cloudflare/cloudflared:latest
          imagePullPolicy: Always
          args:
            - "tunnel"
            - "--no-autoupdate"
            - "run"
            - "--token"
            - <ADD_YOUR_TOKEN_HERE>

Accessing Your Instance

Now that you have your instance up and running, we need to configure Cloudflare tunnel to let you access it, but you need to be careful about one simple point, we used LinuxServer mastodon image, which generate a selfcertificate which is not 100% secure and will cause some problems when talking with Cloudflare tunnels, so we will need to ask Cloudflare to not validate it, this is a simple thing

Our service URL should be <app>.<namespace> and since our app is called mastodon and the namespace is mastodon-server the url is mastodon.mastodon-server

Accessing the Terminal

There is still something you need to do, after registering and creating your account on mastodon, you need to mark it as the owner of the instance, you can do that using the following command:

app/www/bin/tootctl accounts modify <CHANGE-ME> --role Owner

but how can you access the terminal? since you have lens/openlens installed you can click on any running pod named mastodon-<random> and in the top right you will choose the Pod Shell icon, which will give you access to the docker image shell, this way now you can run the command without any problem.

Remember to change <CHANGE-ME> to your username.

Daily Database Backup

We have everything in place, and we only need to make sure we don't loose our data, so we will create a cronjob that will backup our databases and put it on a different S3 bucket that we created on iDrive.

---
apiVersion: v1
kind: Namespace
metadata:
  name: mastodon-server

---
apiVersion: v1
kind: Secret
metadata:
  namespace: mastodon-server
  name: backup-secret
type: Opaque
stringData:
  S3_ACCESS_KEY_ID: <CHANGE-ME>
  S3_SECRET_ACCESS_KEY: <CHANGE-ME>
  S3_BUCKET: <CHANGE-ME>
  S3_ENDPOINT: <CHANGE-ME>

---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgresql-backup-cron-job
  namespace: mastodon-server
spec: 
  schedule: "0 */12 * * *"
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1  
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: postgresql-backup-job
            image: ghcr.io/zaherg/postgres_backup:latest
            imagePullPolicy: Always
            env:
            - name: POSTGRES_DATABASE
              value: "all"
            - name: POSTGRES_HOST
              value: "postgres.mastodon-server"
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db_secrets
                  key: POSTGRES_USER              
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: db_secrets
                  key: POSTGRES_USER
            - name: S3_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: backup_secrets
                  key: S3_ACCESS_KEY_ID   
            - name: S3_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: db_sbackup_secretsecrets
                  key: S3_SECRET_ACCESS_KEY   
            - name: S3_BUCKET
              valueFrom:
                secretKeyRef:
                  name: backup_secrets
                  key: S3_BUCKET   
            - name: S3_ENDPOINT
              valueFrom:
                secretKeyRef:
                  name: backup_secrets
                  key: S3_ENDPOINT   
          restartPolicy: OnFailure
          volumes:
            - name: db_secrets
              secret:
                secretName: postgres-secret
            - name: backup_secrets
              secret:
                secretName: backup-secret
      backoffLimit: 3

Remember to change <CHANGE-ME> to its accurate value.