# K3s cluster

| Name                    | Usage                                | Accessibility    | Host                    | DB type    | Additional data      | Backup configuration | Loki integration | Prometheus integration | Secret management | Status                            |
|-------------------------|--------------------------------------|------------------|-------------------------|------------|----------------------|----------------------|------------------|------------------------|-------------------|-----------------------------------|
| Traefik                 | Reverse proxy and load balancer      | Public & Private | Socrates & Pythagoras-b | -          | -                    | -                    | Configured       | Configured             | -                 | Completed<sup>5</sup>             |
| Vaultwarden             | Password manager                     | Public           | Pythagoras-b            | MariaDB    | -                    | 4AM K8s CronJob      | Configured       | Not available          | -                 | Completed                         |
| Gitlab                  | Version control system               | Public           | Pythagoras-b            | PostgreSQL | User created content | 5AM internal CronJob | Configured       | Configured             | Configured        | Completed<sup>4</sup>             |
| Radarr                  | Movie collection manager             | Private          | Plato                   | PostgreSQL | -                    | -                    | Configured       | Configured             | Configured        | Completed                         |
| Flaresolverr            | Cloudflare proxy                     | Private          | Plato                   | -          | -                    | -                    | -                | -                      | -                 | Completed                         |
| Prometheus              | Metrics aggregator                   | Private          | Pythagoras-b            | TBD        | -                    | Not configured       | Configured       | Configured             | -                 | Partial                           |
| Loki                    | Log aggregator                       | Private          | Pythagoras-b            | TBD        | -                    | Not configured       | Configured       | Configured             | -                 | Partial                           |
| Grafana                 | Graph visualizer                     | Public           | Pythagoras-b            | -          | -                    | Not configured       | Configured       | Configured             | Configured        | Partial                           |
| Sonarr                  | TV shows collection manager          | Private          | Plato                   | SQLite     | -                    | Not configured       | Configured       | Configured             | Configured        | Partial                           |
| Prowlarr                | Torrent indexer                      | Private          | Plato                   | PostgreSQL | -                    | Not configured       | Configured       | Not available          | Configured        | Partial                           |
| Jellyfin                | Media streaming                      | Public           | Archimedes              | SQLite**   | -                    | -                    | Configured       | Not configured         | Not configured    | Awaiting configuration            |
| Jellyseerr              | Media requesting WebUI               | Public           | Pythagoras-b            | -          | -                    | -                    | Not configured   | Not configured         | Not configured    | Awaiting configuration            |
| Adguard                 | DNS ad blocker and custom DNS server | Private          | Socrates                | -          | -                    | -                    | Not configured   | Not configured         | Not configured    | Pending configuration<sup>1</sup> |
| Owncloud Infinity Scale | File hosting webUI                   | Public           | Plato                   | ?          | Drive files          | Not configured       | Configured       | Not available          | Not configured    | Pending configuration<sup>2</sup> |
| Synapse                 | Matrix server - Message centralizer  | Public           | Pythagoras-b            | PostgreSQL | User medias          | 4AM K8s CronJob      | Configured       | Configured             | Not configured    | Pending configuration<sup>3</sup> |
| therbron.com            | Personal website                     | Public           | Socrates                | -          | -                    | -                    | Not configured   | Not configured         | -                 | Awaiting configuration            |
| Home assistant          | Home automation and monitoring       | Private          | Pythagoras-a            | MariaDB    | -                    | Not configured       | Not configured   | Not configured         | Not configured    | Awaiting configuration            |
| Vikunja                 | To-do and Kanban boards              | Public           | Pythagoras-b            | -          | -                    | -                    | Not configured   | Not configured         | -                 | Migrate to Gitlab                 |
| Wiki                    | Documentation manager                | Public           | Pythagoras-b            | -          | -                    | -                    | Not configured   | Not configured         | -                 | Migrate to VuePress and Gitlab    |
| PaperlessNG             | PDF viewer and organiser             | Public           | Pythagoras-b            | PostgreSQL | -                    | -                    | Not configured   | Not configured         | -                 | Research migration into OCIS      |
| Deluge                  | Torrent client                       | Private          | Plato                   | -          | <center> ? </center> | -                    | Not configured   | Not configured         | Not configured    | Awaiting configuration            |
| Minecraft               | Vanilla minecraft server for friends | Public           | Archimedes              | -          | Game map             | Not configured       | Not configured   | Not configured         | -                 | Awaiting configuration            |
| Satisfactory            | Satisfactory server for friends      | Public           | Archimedes              | -          | Game map             | Not configured       | Not configured   | Not configured         | -                 | Not needed for v1                 |
| Space engineers         | Space engineers server for friends   | Public           | Archimedes              | -          | Game map             | Not configured       | Not configured   | Not configured         | -                 | Not needed for v1                 |
| Raspsnir                | Bachelor memorial website            | Public           | Pythagoras-b            | PostgreSQL | -                    | Not configured       | Not configured   | Not configured         | -                 | Not needed for v1                 |

\* Configuration panel only available internally<br>
** Current implementation only support SQLite, making manual backups a necessity<br>
<sup>1</sup> Missing automated configuration pipeline for environment variable injection<br>
<sup>2</sup> Missing configuration for NAS volume mounting (over network)<br>
<sup>3</sup> Missing Longhorn scheduling for saving media_store and secret management<br>
<sup>4</sup> Backup management is not handled by k3s but by an internal cronjob rule (Change image name when putting to production)<br>
<sup>5</sup> Missing dashboard configuration<br>

## Backup management

### Databases

All services needing a database to function come with a sidecar pod running a crontab to automate individual database backups.
These backups are saved into a longhorn volume, to benefit from general snapshots later one.
Each sidecar pod can only mount the backup folder it has been linked with, and cannot see other services' backups.

### Additional data

All additional data needing to be backed up is mounted to a longhorn volume, to also benefit from scheduled backups.

Example :
```
longhorn
└───backups
    └───vaultwarden
    │   └───<backup_date>.sql
    │   │   ...
    └───gitlab
        └───<backup_date>.sql
        │   ...
```
## TODO
- Migrate Vaultwarden & Homeassistant to PostgreSQL instead of MariaDB
- Deploy PostgresQL cluster using operator for database HA and easy maintenance
- Change host/deployment specific variables to use environment variables (using Kustomize)
- ~~Write CI/CD pipeline to create environment loaded files~~ Done with Kustomize migration
- Write CI/CD pipeline to deploy cluster
- ~~Setup internal traefik with nodeport as reverse proxy for internal only services~~ Done through double ingress class and LB
- ~~Setup DB container sidecars for automated backups to Longhorn volume~~
- Setup secrets configuration through CI/CD variable injection (using Kustomzie)
- Explore permission issues when issuing OVH API keys (not working for wildcard and `beta.halia.dev` subdomain)
- Setup default users for deployments
- ~~Setup log and metric monitoring~~
- ~~Define namespaces through yaml files~~
- ~~Look into CockroachDB for redundant database~~ Judged too complicated, moving to a 1 to 1 relationship between services and databases
- ~~Configure IP range accessibility through Traefik (Internal vs external services)~~ Impossible because of flannel ip-masq
- Schedule longhorn S3 backups
- Move secrets to separate, private Git repository ?
- ~~Configure NFS connection for media library~~
- ~~Research IPv6 configuration for outsider node~~ Impossible in Denmark while using YouSee as an ISP for now (no IPv6 support)

## Notes
### Cluster base setup
Setup the cluster's backbone
```
kubectl apply -k environment/dev
```
NOTE: It might be required to update the metallb IP range as well as traefik LoadBalancerIPs

### Convert helm chart to k3s manifest
`helm template chart stable/chart --output-dir ./chart`

### Gitlab backup process
Because gitlab does not offer the possibility to backup a container's data from an external container, a cronjob has been implemented in the custom image used for deployment.

### VPN configuration for Deluge
Instead of adding an extra networking layer to the whole cluster, it seems like a better idea to just integrate a wireguard connection inside of the deluge image, and self-build everything within Gitlab registry.
This image could utilize kubernetes secrets, including a "torrent-vpn" secret produces by the initial wireguard configuration done via Ansible.
This ansible script could create one (or more) additional client(s) depending on the inventory configuration, and keep the "torrent-vpn" configuration file within a k3s formated file, inside of the auto-applied directory on CP.<br>
Cf : https://docs.k3s.io/advanced#auto-deploying-manifests

### Development domains
To access a service publicly when developing, the domain name should be *.beta.halia.dev
To only expose a service internally, the domain name should be *.beta.entos

### Ingresses
To split between external and internal services, two traefik ingresses are implemented through the `ingressclass` annotation.
`traefik-external` will only allow external access to a given service, while `traefik-internal` restrict to an internal only access.