Build a PI Cluster for Local Development - Part 3
Published 5 months and 6 days ago, Categorized under: development, docker, docker-swarm, coding, pi, raspberrypi

In the last post we installed Docker and Gluster . In this article, which is the last one, we will talk see how to run MariaDB, PostgreSQL, Redis and Minio on our cluster.

Docker Compose File:

Form Docker Swarm documentation:

When running Docker Engine in swarm mode, you can use docker stack deploy to deploy a complete application stack to the swarm. The deploy command accepts a stack description in the form of a Compose file. The docker stack deploy command supports any Compose file of version “3.0” or above. If you have an older version, see the upgrade guide.

So we need to have a docker-compose file to describe our services and see which one we need to run. The differences between the normal docker-compose file and the one which we will use is simple, the one which we will use will have a section called deploy which is mainly used in Docker Swarm.

I am not going to go into details and talk about the deploy section, it is advisable that you check the documentations.

The Final Compose File:

To manage the swarm without login to your PIs you might want to install and configure Docker Machine, you can check this article to learn more about managing your Swarm using Docker Machine.

# docker-compose.yml

version: "3.7"

services:
  reverse-proxy:
    image: traefik:v2.2
    command:
      - "--api.dashboard=true"
      - "--providers.docker"
      - "--providers.docker.swarmMode=true"
      - "--providers.docker.swarmModeRefreshSeconds=60s"      
      - "--providers.docker.endpoint=unix:///var/run/docker.sock"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entryPoints.mariadb.address=:3306"
      - "--entryPoints.redis.address=:6379"
      - "--entryPoints.pgsql.address=:5432"
    ports:
      - "80:80"
      - "3306:3306"
      - "5432:5432"
      - "6379:6379"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      labels:
          - "traefik.http.routers.proxy.service=proxy"
          - "traefik.http.routers.proxy.rule=Host(`docker.pi`)"
          - "traefik.http.routers.proxy.middlewares=proxy-compress"
          - "traefik.http.middlewares.proxy-compress.compress=true"
          - "traefik.http.services.proxy.loadbalancer.server.port=8080"          
      mode: global
      placement:
        constraints:
          - node.role == manager
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure

  redis:
    image: arm64v8/redis:6-alpine
    volumes:
      - type: bind
        source: /mnt/data/redis
        target: /data
    stop_grace_period: 1m30s
    deploy:
      labels:
          - "traefik.enable=true"
          - "traefik.tcp.routers.redis.service=redis"
          - "traefik.tcp.routers.redis.entrypoints=redis"
          - "traefik.tcp.routers.redis.rule=HostSNI(`*`)"
          - "traefik.tcp.services.redis.loadbalancer.server.port=6379"
      replicas: 1
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 3
        window: 120s

  pgsql:
    image: postgres:12-alpine
    volumes:
      - type: bind
        source: /mnt/data/pgsql
        target: /var/lib/postgresql/data
    environment:
      - PGDATA=/var/lib/postgresql/data/pgdata
      - POSTGRES_PASSWORD=example
      - POSTGRES_HOST_AUTH_METHOD=trust
    stop_grace_period: 1m30s
    deploy:
      labels:
          - "traefik.enable=true"
          - "traefik.tcp.routers.pgsql.service=pgsql"
          - "traefik.tcp.routers.pgsql.entrypoints=pgsql"
          - "traefik.tcp.routers.pgsql.rule=HostSNI(`*`)"
          - "traefik.tcp.services.pgsql.loadbalancer.server.port=5432"          
      replicas: 1
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 3
        window: 120s
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

  mariadb:
    image: mariadb:latest
    volumes:
      - type: bind
        source: /mnt/data/maria
        target: /var/lib/mysql
    environment:
      - MYSQL_ALLOW_EMPTY_PASSWORD=true
    stop_grace_period: 1m30s
    deploy:
      labels:
          - "traefik.enable=true"
          - "traefik.tcp.routers.mariadb.service=mariadb"
          - "traefik.tcp.routers.mariadb.entrypoints=mariadb"
          - "traefik.tcp.routers.mariadb.rule=HostSNI(`*`)"
          - "traefik.tcp.services.mariadb.loadbalancer.server.port=3306"          
      replicas: 1
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 3
        window: 120s
    healthcheck:
      test: "mysqladmin ping"
      interval: 10s
      timeout: 5s
      retries: 5

  minio:
    image: jessestuart/minio:latest
    command: server /data
    volumes:
      - type: bind
        source: /mnt/data/minio
        target: /data
    environment:
      - MINIO_ACCESS_KEY=P8DvPRjulE04Zc3dVLbK1G6mrmGWxBhCAIt
      - MINIO_SECRET_KEY=MqakT6oWeVsVex5BBwPWui0hMy8riEFwpPKw8
    stop_grace_period: 1m30s
    deploy:
      labels:
          - "traefik.enable=true"
          - "traefik.http.middlewares.minio-compress.compress=true"   
          - "traefik.http.routers.minio.entrypoints=web"
          - "traefik.http.routers.minio.rule=Host(`minio.docker.pi`)"
          - "traefik.http.services.minio.loadbalancer.server.port=9000"
          - "traefik.http.routers.minio.middlewares=minio-compress"          
      replicas: 2
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 3
        window: 120s

Now that you have saw the final compose file, let's talk about the content of it. The file contains a few services that we run a reverse proxy (Traefik), 3 databases and an object storage.

Traefik:

The main usage for Traefik in our case is to act as a reverse proxy to our internal services, as this will make it easier for us to access the services directly without worrying about managing the internal network of the cluster.

  1. We start the Traefik services with a few commands to enable the API and the dashboard, plus telling Traefik that we are using docker with swarm mode enabled, and the ports (entrypoints) we want to be handled.
    command:
      - "--api.dashboard=true"
      - "--providers.docker"
      - "--providers.docker.swarmMode=true"
      - "--providers.docker.endpoint=unix:///var/run/docker.sock"
      - "--entrypoints.web.address=:80"
      - "--entryPoints.mariadb.address=:3306"
      - "--entryPoints.redis.address=:6379"
      - "--entryPoints.pgsql.address=:5432"
    ports:
      - "80:80"
      - "3306:3306"
      - "5432:5432"
      - "6379:6379"
      - "8080:8080"

Most of Traefik configuration can be done using a configuration file, but I opt-out to use the inline commands as we are using a few options, more about the configuration can be read here.

  1. It is important to share the docker socket with Traefik, so it can listen to docker events.
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  1. We define the deploy section for the service. The labels in this section define the name of the service, the rule , the port and the middleware. We are specifying the port to tell Traefik to route requests to that port.
    deploy:
      labels:
          - "traefik.enable=true"
          - "traefik.http.routers.proxy.service=proxy"
          - "traefik.http.routers.proxy.rule=Host(`docker.pi`)"
          - "traefik.http.services.proxy.loadbalancer.server.port=8080"
      mode: global
      placement:
        constraints:
          - node.role == manager
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure

We need to understand that traefik is using the labels section in your docker-compose.yml file to discover the services when using Docker as a provider, more about discovering the services can be found here.

Since we enabled the swarm mode, these labels should be part of the deploy section.

Redis

Defining redis as a service is not different than anything you have done before with docker compose, we just need to add the deploy section and tells Traefik to manage it.

  1. We define the service, the image and the shared volume, notice that we are using the Gluster mounted drive:
    image: arm64v8/redis:6-alpine
    volumes:
      - type: bind
        source: /mnt/data/redis
        target: /data
    stop_grace_period: 1m30s
  1. We define the deploy section and add the labels needed for Traefik to manage it:
    deploy:
      labels:
          - "traefik.enable=true"
          - "traefik.tcp.routers.redis.service=redis"
          - "traefik.tcp.routers.redis.entrypoints=redis"
          - "traefik.tcp.routers.redis.rule=HostSNI(`*`)"
          - "traefik.tcp.services.redis.loadbalancer.server.port=6379"
      replicas: 1
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 3
        window: 120s

If you look closely you will notice that the labels is somehow similar to what we have defined in Traefik, but be careful we are using the tcp protocol not http, this is due to the fact that the databases listen to a TCP port not HTTP. This rule traefik.tcp.routers.redis.rule=HostSNI(``)* instruct Traefik to listen to a Server Name Indication.

It is important to note that the Server Name Indication is an extension of the TLS protocol. Hence, only TLS routers will be able to specify a domain name with that rule. However, non-TLS routers will have to explicitly use that rule with * (every domain) to state that every non-TLS request will be handled by the router.

We also specified the redis enterypoint which listen to the port 6379.

PostgreSQL:

PostgreSQL configuration is somehow the same as Redis, except the changes in the enterypoint, the port and the shared folder, so I am going to list it here without explaining it:

  pgsql:
    image: postgres:12-alpine
    volumes:
      - type: bind
        source: /mnt/data/pgsql
        target: /var/lib/postgresql/data
    environment:
      - PGDATA=/var/lib/postgresql/data/pgdata
      - POSTGRES_PASSWORD=example
      - POSTGRES_HOST_AUTH_METHOD=trust
    stop_grace_period: 1m30s
    deploy:
      labels:
          - "traefik.enable=true"
          - "traefik.tcp.routers.pgsql.service=pgsql"
          - "traefik.tcp.routers.pgsql.entrypoints=pgsql"
          - "traefik.tcp.routers.pgsql.rule=HostSNI(`*`)"
          - "traefik.tcp.services.pgsql.loadbalancer.server.port=5432"          
      replicas: 1
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 3
        window: 120s
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

MariaDB:

MariaDB configuration is somehow the same as Redis, except the changes in the enterypoint, the port and the shared folder, so I am going to list it here without explaining it:

  mariadb:
    image: mariadb:latest
    volumes:
      - type: bind
        source: /mnt/data/maria
        target: /var/lib/mysql
    environment:
      - MYSQL_ALLOW_EMPTY_PASSWORD=true
    stop_grace_period: 1m30s
    deploy:
      labels:
          - "traefik.enable=true"
          - "traefik.tcp.routers.mariadb.service=mariadb"
          - "traefik.tcp.routers.mariadb.entrypoints=mariadb"
          - "traefik.tcp.routers.mariadb.rule=HostSNI(`*`)"
          - "traefik.tcp.services.mariadb.loadbalancer.server.port=3306"          
      replicas: 1
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 3
        window: 120s
    healthcheck:
      test: "mysqladmin ping"
      interval: 10s
      timeout: 5s
      retries: 5

I have added a small health check for both MardiaDB and PostgreSQL, but you are free to add it to Redis, you can check this Github repository to get more information about the health check.

Please note that increasing the replica in the databases will case problems, as each database has its own way to implement replications between main and secondary (or read-only) servers, please check the documentation for each database server.

Minio:

For those who don't know what is Minio, from Minio website:

MinIO's high-performance, software-defined object storage suite enables customers to build cloud-native data infrastructure for machine learning, analytics and application data workloads.

To be honest, now that you become familiar with how to add a new service to Traefik, adding Minio won't be a problem. Sadly Minio does not provide an official docker image for ARM devices, they do have the binary ready, they promised to publish an official one soon.

Luckily, there is a multi-arch docker image that we can use:

  1. This section is simple, we defined the image, we run the command (check the docker image documentation), the shared volume and finally we defined two important environment variables MINIO_ACCESS_KEY and MINIO_SECRET_KEY
    image: jessestuart/minio:latest
    command: server /data
    volumes:
      - type: bind
        source: /mnt/data/minio
        target: /data
    environment:
      - MINIO_ACCESS_KEY=my-access-key
      - MINIO_SECRET_KEY=my-secret-key
    stop_grace_period: 1m30s
  1. We define the deploy section and lets focus on the labels inside of it:
    deploy:
      labels:
          - "traefik.enable=true"
          - "traefik.http.routers.minio.entrypoints=web"
          - "traefik.http.routers.minio.rule=Host(`minio.docker.pi`)"
          - "traefik.http.services.minio.loadbalancer.server.port=9000"
      replicas: 2
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 3
        window: 120s

As you can see it is simple, we defined 4 important things, we told Traefik to handle this service, we specify the entrypoints, we set the routers role so now Traefik will listen to the domain minio.docker.pi and send it to the Minio service, and lastly we specified the port.

Final Steps:

  1. Before you deploy the stack make sure to create the following directories within /mnt/data by running the following command:
$ mkdir -p /mnt/data/{maria,redis,pgsql,minio}
  1. Edit your hosts file and add docker.pi and mini.docker.pi to it, and associate it the PIs IP.

Once you created the directories and added the domains, run the following command to create our stack:

# Deploy the stack
$ docker stack deploy --prune raspi --compose-file docker-compose.yml

# Check the services status
$ docker service ls

# you should get something like:

ID                  NAME                  MODE                REPLICAS            IMAGE                      PORTS
cbq8a4wkcqjm        raspi_mariadb         replicated          1/1                 mariadb:latest
y1d17h9xgxqf        raspi_minio           replicated          2/2                 jessestuart/minio:latest
d8e45v506s28        raspi_pgsql           replicated          1/1                 postgres:13-alpine
ywbj2u4siau8        raspi_redis           replicated          1/1                 arm64v8/redis:6-alpine
5fp0d2d0b5rj        raspi_reverse-proxy   global              2/2                 traefik:v2.2               *:80->80/tcp, *:443->443/tcp, *:3306->3306/tcp, *:5432->5432/tcp, *:6379->6379/tcp, *:8080->8080/tcp

Now opening your browser to http://docker.pi:8080/dashboard should present you with Traefik dashboard, something like:

My Local RaspberryPI 4 Cluster Dashboard

Login to Minio:

Minio is now running under the url http://mini.docker.pi once you open this url you will be presented with the UI for it:

Minio Login Page

Using the access key and the secret you will be login to your Minio Instance:

Minio Dashboard

Connect to your Databases:

Via GUI:

Using an application like TablePlus you can connect to our databases instance using the following configuration:

  1. MariaDB

MariaDB configuration

  1. PostgreSQL

PostgreSQL configuration

  1. Redis

Redis configuration

Via Code (example Laravel):

Remember to connect to your Databases via any GUI application to create the databases before you edit your .env file.

For laravel edit your .env file and change the database section based on the database server you are using:

  1. MariaDB:
DB_CONNECTION=mysql
DB_HOST=docker.pi
DB_PORT=3306
DB_DATABASE=laravel
DB_USERNAME=root
DB_PASSWORD=
  1. PostgreSQL:
DB_CONNECTION=pgsql
DB_HOST=docker.pi
DB_PORT=5432
DB_DATABASE=laravel
DB_USERNAME=postgres
DB_PASSWORD=example
  1. Redis:
REDIS_HOST=docker.pi
REDIS_PASSWORD=null
REDIS_PORT=6379
REDIS_DB=3
REDIS_CACHE_DB=3

This is the end of our series, if you find any problem let me know.

Take a look at this docker hub account where you can find many docker images which was built specifically for arm 64bit devices.



Build a PI Cluster for Local Development:

  1. Part one: Preparing your PIs.
  2. Part two: Installing Docker, Docker swarm and Gluster.
  3. Part three: Create your Stack (MariaDB, PostgreSQL, Redis and Minio)
Share It via: