K3s-cluster/README.md

# K3s cluster

| Name                    | Usage                                | Accessibility    | Host                    | DB type    | Additional data      | Backup configuration | Loki integration | Prometheus integration | Secret management | Status                            |
|-------------------------|--------------------------------------|------------------|-------------------------|------------|----------------------|----------------------|------------------|------------------------|-------------------|-----------------------------------|
| Traefik                 | Reverse proxy and load balancer      | Public & Private | Socrates & Pythagoras-b | -          | -                    | -                    | Configured       | Configured             | -                 | Completed<sup>5</sup>             |
| Vaultwarden             | Password manager                     | Public           | Pythagoras-b            | MariaDB    | -                    | 4AM K8s CronJob      | Configured       | Not available          | -                 | Completed                         |
| Gitlab                  | Version control system               | Public           | Pythagoras-b            | PostgreSQL | User created content | 5AM internal CronJob | Configured       | Configured             | Configured        | Completed<sup>4</sup>             |
| Radarr                  | Movie collection manager             | Private          | Plato                   | PostgreSQL | -                    | -                    | Configured       | Configured             | Configured        | Completed                         |
| Flaresolverr            | Cloudflare proxy                     | Private          | Plato                   | -          | -                    | -                    | -                | -                      | -                 | Completed                         |
| Prometheus              | Metrics aggregator                   | Private          | Pythagoras-b            | TBD        | -                    | Not configured       | Configured       | Configured             | -                 | Partial                           |
| Loki                    | Log aggregator                       | Private          | Pythagoras-b            | TBD        | -                    | Not configured       | Configured       | Configured             | -                 | Partial                           |
| Grafana                 | Graph visualizer                     | Public           | Pythagoras-b            | -          | -                    | Not configured       | Configured       | Configured             | Configured        | Partial                           |
| Sonarr                  | TV shows collection manager          | Private          | Plato                   | SQLite     | -                    | Not configured       | Configured       | Configured             | Configured        | Partial                           |
| Prowlarr                | Torrent indexer                      | Private          | Plato                   | PostgreSQL | -                    | Not configured       | Configured       | Not available          | Configured        | Partial                           |
| Jellyfin                | Media streaming                      | Public           | Archimedes              | SQLite**   | -                    | -                    | Configured       | Not configured         | Not configured    | Awaiting configuration            |
| Jellyseerr              | Media requesting WebUI               | Public           | Pythagoras-b            | -          | -                    | -                    | Not configured   | Not configured         | Not configured    | Awaiting configuration            |
| Adguard                 | DNS ad blocker and custom DNS server | Private          | Socrates                | -          | -                    | -                    | Not configured   | Not configured         | Not configured    | Pending configuration<sup>1</sup> |
| Owncloud Infinity Scale | File hosting webUI                   | Public           | Plato                   | ?          | Drive files          | Not configured       | Configured       | Not available          | Not configured    | Pending configuration<sup>2</sup> |
| Synapse                 | Matrix server - Message centralizer  | Public           | Pythagoras-b            | PostgreSQL | User medias          | 4AM K8s CronJob      | Configured       | Configured             | Not configured    | Pending configuration<sup>3</sup> |
| therbron.com            | Personal website                     | Public           | Socrates                | -          | -                    | -                    | Not configured   | Not configured         | -                 | Awaiting configuration            |
| Home assistant          | Home automation and monitoring       | Private          | Pythagoras-a            | MariaDB    | -                    | Not configured       | Not configured   | Not configured         | Not configured    | Awaiting configuration            |
| Vikunja                 | To-do and Kanban boards              | Public           | Pythagoras-b            | -          | -                    | -                    | Not configured   | Not configured         | -                 | Migrate to Gitlab                 |
| Wiki                    | Documentation manager                | Public           | Pythagoras-b            | -          | -                    | -                    | Not configured   | Not configured         | -                 | Migrate to VuePress and Gitlab    |
| PaperlessNG             | PDF viewer and organiser             | Public           | Pythagoras-b            | PostgreSQL | -                    | -                    | Not configured   | Not configured         | -                 | Research migration into OCIS      |
| Deluge                  | Torrent client                       | Private          | Plato                   | -          | <center> ? </center> | -                    | Not configured   | Not configured         | Not configured    | Awaiting configuration            |
| Minecraft               | Vanilla minecraft server for friends | Public           | Archimedes              | -          | Game map             | Not configured       | Not configured   | Not configured         | -                 | Awaiting configuration            |
| Satisfactory            | Satisfactory server for friends      | Public           | Archimedes              | -          | Game map             | Not configured       | Not configured   | Not configured         | -                 | Not needed for v1                 |
| Space engineers         | Space engineers server for friends   | Public           | Archimedes              | -          | Game map             | Not configured       | Not configured   | Not configured         | -                 | Not needed for v1                 |
| Raspsnir                | Bachelor memorial website            | Public           | Pythagoras-b            | PostgreSQL | -                    | Not configured       | Not configured   | Not configured         | -                 | Not needed for v1                 |

\* Configuration panel only available internally<br>
** Current implementation only support SQLite, making manual backups a necessity<br>
<sup>1</sup> Missing automated configuration pipeline for environment variable injection<br>
<sup>2</sup> Missing configuration for NAS volume mounting (over network)<br>
<sup>3</sup> Missing Longhorn scheduling for saving media_store and secret management<br>
<sup>4</sup> Backup management is not handled by k3s but by an internal cronjob rule (Change image name when putting to production)<br>
<sup>5</sup> Missing dashboard configuration<br>

## Backup management

### Databases

All services needing a database to function come with a sidecar pod running a crontab to automate individual database backups.
These backups are saved into a longhorn volume, to benefit from general snapshots later one.
Each sidecar pod can only mount the backup folder it has been linked with, and cannot see other services' backups.

### Additional data

All additional data needing to be backed up is mounted to a longhorn volume, to also benefit from scheduled backups.

Example :
```
longhorn
└───backups
    └───vaultwarden
    │   └───<backup_date>.sql
    │   │   ...
    └───gitlab
        └───<backup_date>.sql
        │   ...
```
## TODO
- Migrate Vaultwarden & Homeassistant to PostgreSQL instead of MariaDB
- Deploy PostgresQL cluster using operator for database HA and easy maintenance
- Change host/deployment specific variables to use environment variables (using Kustomize)
- ~~Write CI/CD pipeline to create environment loaded files~~ Done with Kustomize migration
- Write CI/CD pipeline to deploy cluster
- ~~Setup internal traefik with nodeport as reverse proxy for internal only services~~ Done through double ingress class and LB
- ~~Setup DB container sidecars for automated backups to Longhorn volume~~
- Setup secrets configuration through CI/CD variable injection (using Kustomzie)
- Explore permission issues when issuing OVH API keys (not working for wildcard and `beta.halia.dev` subdomain)
- Setup default users for deployments
- ~~Setup log and metric monitoring~~
- ~~Define namespaces through yaml files~~
- ~~Look into CockroachDB for redundant database~~ Judged too complicated, moving to a 1 to 1 relationship between services and databases
- ~~Configure IP range accessibility through Traefik (Internal vs external services)~~ Impossible because of flannel ip-masq
- Schedule longhorn S3 backups
- Move secrets to separate, private Git repository ?
- ~~Configure NFS connection for media library~~
- ~~Research IPv6 configuration for outsider node~~ Impossible in Denmark while using YouSee as an ISP for now (no IPv6 support)

## Notes
### Cluster base setup
Setup the cluster's backbone
```
kubectl apply -k environment/dev
```
NOTE: It might be required to update the metallb IP range as well as traefik LoadBalancerIPs

### Convert helm chart to k3s manifest
`helm template chart stable/chart --output-dir ./chart`

### Gitlab backup process
Because gitlab does not offer the possibility to backup a container's data from an external container, a cronjob has been implemented in the custom image used for deployment.

### VPN configuration for Deluge
Instead of adding an extra networking layer to the whole cluster, it seems like a better idea to just integrate a wireguard connection inside of the deluge image, and self-build everything within Gitlab registry.
This image could utilize kubernetes secrets, including a "torrent-vpn" secret produces by the initial wireguard configuration done via Ansible.
This ansible script could create one (or more) additional client(s) depending on the inventory configuration, and keep the "torrent-vpn" configuration file within a k3s formated file, inside of the auto-applied directory on CP.<br>
Cf : https://docs.k3s.io/advanced#auto-deploying-manifests

### Development domains
To access a service publicly when developing, the domain name should be *.beta.halia.dev
To only expose a service internally, the domain name should be *.beta.entos

### Ingresses
To split between external and internal services, two traefik ingresses are implemented through the `ingressclass` annotation.
`traefik-external` will only allow external access to a given service, while `traefik-internal` restrict to an internal only access.
Initial commit 2022-02-16 15:26:46 +00:00			`# K3s cluster`

docs(formating): Update README formating and service list 2022-12-30 23:28:24 +00:00			`\| Name \| Usage \| Accessibility \| Host \| DB type \| Additional data \| Backup configuration \| Loki integration \| Prometheus integration \| Secret management \| Status \|`
			`\|-------------------------\|--------------------------------------\|------------------\|-------------------------\|------------\|----------------------\|----------------------\|------------------\|------------------------\|-------------------\|-----------------------------------\|`
			`\| Traefik \| Reverse proxy and load balancer \| Public & Private \| Socrates & Pythagoras-b \| - \| - \| - \| Configured \| Configured \| - \| Completed<sup>5</sup> \|`
			`\| Vaultwarden \| Password manager \| Public \| Pythagoras-b \| MariaDB \| - \| 4AM K8s CronJob \| Configured \| Not available \| - \| Completed \|`
			`\| Gitlab \| Version control system \| Public \| Pythagoras-b \| PostgreSQL \| User created content \| 5AM internal CronJob \| Configured \| Configured \| Configured \| Completed<sup>4</sup> \|`
			`\| Radarr \| Movie collection manager \| Private \| Plato \| PostgreSQL \| - \| - \| Configured \| Configured \| Configured \| Completed \|`
docs(readme): Update setup documentation and notes 2023-02-07 08:47:24 +00:00			`\| Flaresolverr \| Cloudflare proxy \| Private \| Plato \| - \| - \| - \| - \| - \| - \| Completed \|`
docs(formating): Update README formating and service list 2022-12-30 23:28:24 +00:00			`\| Prometheus \| Metrics aggregator \| Private \| Pythagoras-b \| TBD \| - \| Not configured \| Configured \| Configured \| - \| Partial \|`
			`\| Loki \| Log aggregator \| Private \| Pythagoras-b \| TBD \| - \| Not configured \| Configured \| Configured \| - \| Partial \|`
			`\| Grafana \| Graph visualizer \| Public \| Pythagoras-b \| - \| - \| Not configured \| Configured \| Configured \| Configured \| Partial \|`
			`\| Sonarr \| TV shows collection manager \| Private \| Plato \| SQLite \| - \| Not configured \| Configured \| Configured \| Configured \| Partial \|`
			`\| Prowlarr \| Torrent indexer \| Private \| Plato \| PostgreSQL \| - \| Not configured \| Configured \| Not available \| Configured \| Partial \|`
			`\| Jellyfin \| Media streaming \| Public \| Archimedes \| SQLite** \| - \| - \| Configured \| Not configured \| Not configured \| Awaiting configuration \|`
			`\| Jellyseerr \| Media requesting WebUI \| Public \| Pythagoras-b \| - \| - \| - \| Not configured \| Not configured \| Not configured \| Awaiting configuration \|`
			`\| Adguard \| DNS ad blocker and custom DNS server \| Private \| Socrates \| - \| - \| - \| Not configured \| Not configured \| Not configured \| Pending configuration<sup>1</sup> \|`
			`\| Owncloud Infinity Scale \| File hosting webUI \| Public \| Plato \| ? \| Drive files \| Not configured \| Configured \| Not available \| Not configured \| Pending configuration<sup>2</sup> \|`
			`\| Synapse \| Matrix server - Message centralizer \| Public \| Pythagoras-b \| PostgreSQL \| User medias \| 4AM K8s CronJob \| Configured \| Configured \| Not configured \| Pending configuration<sup>3</sup> \|`
			`\| therbron.com \| Personal website \| Public \| Socrates \| - \| - \| - \| Not configured \| Not configured \| - \| Awaiting configuration \|`
			`\| Home assistant \| Home automation and monitoring \| Private \| Pythagoras-a \| MariaDB \| - \| Not configured \| Not configured \| Not configured \| Not configured \| Awaiting configuration \|`
			`\| Vikunja \| To-do and Kanban boards \| Public \| Pythagoras-b \| - \| - \| - \| Not configured \| Not configured \| - \| Migrate to Gitlab \|`
			`\| Wiki \| Documentation manager \| Public \| Pythagoras-b \| - \| - \| - \| Not configured \| Not configured \| - \| Migrate to VuePress and Gitlab \|`
			`\| PaperlessNG \| PDF viewer and organiser \| Public \| Pythagoras-b \| PostgreSQL \| - \| - \| Not configured \| Not configured \| - \| Research migration into OCIS \|`
			`\| Deluge \| Torrent client \| Private \| Plato \| - \| <center> ? </center> \| - \| Not configured \| Not configured \| Not configured \| Awaiting configuration \|`
			`\| Minecraft \| Vanilla minecraft server for friends \| Public \| Archimedes \| - \| Game map \| Not configured \| Not configured \| Not configured \| - \| Awaiting configuration \|`
			`\| Satisfactory \| Satisfactory server for friends \| Public \| Archimedes \| - \| Game map \| Not configured \| Not configured \| Not configured \| - \| Not needed for v1 \|`
			`\| Space engineers \| Space engineers server for friends \| Public \| Archimedes \| - \| Game map \| Not configured \| Not configured \| Not configured \| - \| Not needed for v1 \|`
			`\| Raspsnir \| Bachelor memorial website \| Public \| Pythagoras-b \| PostgreSQL \| - \| Not configured \| Not configured \| Not configured \| - \| Not needed for v1 \|`
Add DN status to README 2022-06-06 22:31:49 +00:00
Add missing line break 2022-06-07 22:07:40 +00:00			`\* Configuration panel only available internally<br>`
docs(README): Fix table legend format 2022-09-23 23:08:05 +00:00			`** Current implementation only support SQLite, making manual backups a necessity<br>`
			`<sup>1</sup> Missing automated configuration pipeline for environment variable injection<br>`
			`<sup>2</sup> Missing configuration for NAS volume mounting (over network)<br>`
			`<sup>3</sup> Missing Longhorn scheduling for saving media_store and secret management<br>`
feat(gitlab): Finish gitlab configuration Add backup documentation for internal backup management 2022-10-22 18:43:28 +00:00			`<sup>4</sup> Backup management is not handled by k3s but by an internal cronjob rule (Change image name when putting to production)<br>`
feat(ingress): Add additional traefik for internal only access A new traefik deployment has been added, and two ingressclasses have been set for the new instance, as well as the previous one. This allows the network to be split in two, one for external access and another for internal access. Each traefik deployment is connected to a loadbalancer requesting the IP necessary for each type of access. 2022-11-20 02:38:22 +00:00			`<sup>5</sup> Missing dashboard configuration<br>`
Add some notes + update service table & status 2022-06-06 16:59:28 +00:00
Add details to cluster organisation and backup management 2022-07-12 21:15:45 +00:00			`## Backup management`
docs(formating): Update README formating and service list 2022-12-30 23:28:24 +00:00
Add details to cluster organisation and backup management 2022-07-12 21:15:45 +00:00			`### Databases`
docs(formating): Update README formating and service list 2022-12-30 23:28:24 +00:00
Add details to cluster organisation and backup management 2022-07-12 21:15:45 +00:00			`All services needing a database to function come with a sidecar pod running a crontab to automate individual database backups.`
			`These backups are saved into a longhorn volume, to benefit from general snapshots later one.`
			`Each sidecar pod can only mount the backup folder it has been linked with, and cannot see other services' backups.`

			`### Additional data`
docs(formating): Update README formating and service list 2022-12-30 23:28:24 +00:00
Add details to cluster organisation and backup management 2022-07-12 21:15:45 +00:00			`All additional data needing to be backed up is mounted to a longhorn volume, to also benefit from scheduled backups.`

			`Example :`
			```
			`longhorn`
			`└───backups`
			`└───vaultwarden`
			`│ └───<backup_date>.sql`
			`│ │ ...`
			`└───gitlab`
			`└───<backup_date>.sql`
			`│ ...`
			```
Add some notes + update service table & status 2022-06-06 16:59:28 +00:00			`## TODO`
docs(formating): Update README formating and service list 2022-12-30 23:28:24 +00:00			`- Migrate Vaultwarden & Homeassistant to PostgreSQL instead of MariaDB`
docs(readme): Update setup documentation and notes 2023-02-07 08:47:24 +00:00			`- Deploy PostgresQL cluster using operator for database HA and easy maintenance`
			`- Change host/deployment specific variables to use environment variables (using Kustomize)`
			`- ~~Write CI/CD pipeline to create environment loaded files~~ Done with Kustomize migration`
Add some notes + update service table & status 2022-06-06 16:59:28 +00:00			`- Write CI/CD pipeline to deploy cluster`
feat(ingress): Add additional traefik for internal only access A new traefik deployment has been added, and two ingressclasses have been set for the new instance, as well as the previous one. This allows the network to be split in two, one for external access and another for internal access. Each traefik deployment is connected to a loadbalancer requesting the IP necessary for each type of access. 2022-11-20 02:38:22 +00:00			`- ~~Setup internal traefik with nodeport as reverse proxy for internal only services~~ Done through double ingress class and LB`
docs(README): Update documentation Add logging, metrics and visualization setup, custom namespaces and update documentation for new helm-less configuration 2022-09-14 22:16:12 +00:00			`- ~~Setup DB container sidecars for automated backups to Longhorn volume~~`
docs(readme): Update setup documentation and notes 2023-02-07 08:47:24 +00:00			`- Setup secrets configuration through CI/CD variable injection (using Kustomzie)`
docs(README): Update documentation Add logging, metrics and visualization setup, custom namespaces and update documentation for new helm-less configuration 2022-09-14 22:16:12 +00:00			- Explore permission issues when issuing OVH API keys (not working for wildcard and `beta.halia.dev` subdomain)
docs(Network&Norm): Add networking documentation for ingress and naming norms 2022-11-20 02:46:46 +00:00			`- Setup default users for deployments`
docs(README): Update documentation Add logging, metrics and visualization setup, custom namespaces and update documentation for new helm-less configuration 2022-09-14 22:16:12 +00:00			`- ~~Setup log and metric monitoring~~`
			`- ~~Define namespaces through yaml files~~`
Update README with missing services and new progress 2022-06-20 23:30:00 +00:00			`- ~~Look into CockroachDB for redundant database~~ Judged too complicated, moving to a 1 to 1 relationship between services and databases`
Update TODO list in README.md 2022-06-20 15:34:37 +00:00			`- ~~Configure IP range accessibility through Traefik (Internal vs external services)~~ Impossible because of flannel ip-masq`
feat(kustomization): Traefik can now be deployed using Kustomize Add kustomization files to traefik, for both ingresses. 2022-12-30 23:16:17 +00:00			`- Schedule longhorn S3 backups`
			`- Move secrets to separate, private Git repository ?`
docs(readme): Update setup documentation and notes 2023-02-07 08:47:24 +00:00			`- ~~Configure NFS connection for media library~~`
			`- ~~Research IPv6 configuration for outsider node~~ Impossible in Denmark while using YouSee as an ISP for now (no IPv6 support)`
Add configuration folder for nginx sample, adguard and kubernetes dashboard 2022-03-30 17:11:15 +00:00
			`## Notes`
docs(README): Update documentation Add logging, metrics and visualization setup, custom namespaces and update documentation for new helm-less configuration 2022-09-14 22:16:12 +00:00			`### Cluster base setup`
feat(kustomization): Add kustomization for deployment When deploying anything on the cluster, a `kustomization.yaml` is to be used to allow for on-the-fly patches, and autonomous cluster configuration through native k8s file configuration. This commit adds the first iteration for such implementation, essentially fixing the issue of setting up the cluster from a clean base. Now everything is consolidated into one single file, which later applies desired resources. This also fixes having to copy and paste URLs into the console, as Kustomize allows one to put those URLs into the kustomization file. 2022-12-30 22:48:09 +00:00			`Setup the cluster's backbone`
Add some notes + update service table & status 2022-06-06 16:59:28 +00:00			```
docs(readme): Update setup documentation and notes 2023-02-07 08:47:24 +00:00			`kubectl apply -k environment/dev`
Add some notes + update service table & status 2022-06-06 16:59:28 +00:00			```
docs(readme): Update setup documentation and notes 2023-02-07 08:47:24 +00:00			`NOTE: It might be required to update the metallb IP range as well as traefik LoadBalancerIPs`
Add dev cluster setup documentation 2022-09-10 01:44:07 +00:00
docs(README): Update documentation Add logging, metrics and visualization setup, custom namespaces and update documentation for new helm-less configuration 2022-09-14 22:16:12 +00:00			`### Convert helm chart to k3s manifest`
			`helm template chart stable/chart --output-dir ./chart`
docs(README): Update to reflect latest update 2022-10-17 22:52:14 +00:00
feat(gitlab): Finish gitlab configuration Add backup documentation for internal backup management 2022-10-22 18:43:28 +00:00			`### Gitlab backup process`
			`Because gitlab does not offer the possibility to backup a container's data from an external container, a cronjob has been implemented in the custom image used for deployment.`
feat(LB): Add MetalLB and Traefik managed LBs Add custom IP range for LBs to split internal and external access 2022-11-16 22:52:06 +00:00
			`### VPN configuration for Deluge`
			`Instead of adding an extra networking layer to the whole cluster, it seems like a better idea to just integrate a wireguard connection inside of the deluge image, and self-build everything within Gitlab registry.`
			`This image could utilize kubernetes secrets, including a "torrent-vpn" secret produces by the initial wireguard configuration done via Ansible.`
docs(formating): Update README formating and service list 2022-12-30 23:28:24 +00:00			`This ansible script could create one (or more) additional client(s) depending on the inventory configuration, and keep the "torrent-vpn" configuration file within a k3s formated file, inside of the auto-applied directory on CP.<br>`
feat(LB): Add MetalLB and Traefik managed LBs Add custom IP range for LBs to split internal and external access 2022-11-16 22:52:06 +00:00			`Cf : https://docs.k3s.io/advanced#auto-deploying-manifests`

			`### Development domains`
feat(ingress): Add additional traefik for internal only access A new traefik deployment has been added, and two ingressclasses have been set for the new instance, as well as the previous one. This allows the network to be split in two, one for external access and another for internal access. Each traefik deployment is connected to a loadbalancer requesting the IP necessary for each type of access. 2022-11-20 02:38:22 +00:00			`To access a service publicly when developing, the domain name should be *.beta.halia.dev`
style(convention): Remove file naming convention The previously used file naming convention is no longer necessary as Kustomize already orders any request done to the fluster, without having to rely on the `--recursive` fly of the `apply` command. 2022-12-30 23:24:09 +00:00			`To only expose a service internally, the domain name should be *.beta.entos`
feat(ingress): Add additional traefik for internal only access A new traefik deployment has been added, and two ingressclasses have been set for the new instance, as well as the previous one. This allows the network to be split in two, one for external access and another for internal access. Each traefik deployment is connected to a loadbalancer requesting the IP necessary for each type of access. 2022-11-20 02:38:22 +00:00
			`### Ingresses`
			To split between external and internal services, two traefik ingresses are implemented through the `ingressclass` annotation.
			`traefik-external` will only allow external access to a given service, while `traefik-internal` restrict to an internal only access.