K3s-cluster/README.md

K3s cluster

/!\ CHECK PORT RULES FOR EXPOSING INTERNAL SERVICES

Name	Usage	Accessibility	Host	DB type	Additional data	Backup configuration	Loki integration	Prometheus integration	Status
Traefik	Reverse proxy and load balancer	Public*	Socrates	-	-	-	Configured	Configured	Completed5
Vaultwarden	Password manager	Public	Pythagoras-b	MariaDB	-	4AM K8s CronJob	Configured	Not available	Completed
Gitlab	Version control system	Public	Pythagoras-b	PostgreSQL	User created content	5AM internal CronJob	Configured	Configured	Completed4
Prometheus	Metrics aggregator	Private	Pythagoras-b	TBD	-	Not configured	Configured	Configured	Partial
Loki	Log aggregator	Private	Pythagoras-b	TBD	-	Not configured	Configured	Configured	Partial
Grafana	Graph visualizer	Public	Pythagoras-b	-	-	Not configured	Configured	Configured	Partial
Adguard	DNS ad blocker and custom DNS server	Private	Socrates	-	-	-	Not configured	Not configured	Pending configuration1
Owncloud Infinity Scale	File hosting webUI	Public	Plato	?	Drive files	Not configured	Configured	Not available	Pending configuration2
Synapse	Matrix server - Message centralizer	Public	Pythagoras-b	PostgreSQL	User medias	4AM K8s CronJob	Configured	Configured	Pending configuration3
therbron.com	Personal website	Public	Socrates	-	-	-	Not configured	Not configured	Awaiting configuration
Home assistant	Home automation and monitoring	Private	Pythagoras-a	MariaDB	-	Not configured	Not configured	Not configured	Awaiting configuration
Vikunja	To-do and Kanban boards	Public	Pythagoras-b	-	-	-	Not configured	Not configured	Migrate to Gitlab
Wiki	Documentation manager	Public	Pythagoras-b	-	-	-	Not configured	Not configured	Migrate to VuePress and Gitlab
PaperlessNG	PDF viewer and organiser	Public	Pythagoras-b	PostgreSQL	-	-	Not configured	Not configured	Research migration into OCIS
Jellyfin	Media streaming	Public	Archimedes	-	-	-	Not configured	Not configured	Awaiting configuration
Sonarr	TV shows collection manager	Private	Plato	SQLite**	Internal backups	Not configured	Not configured	Not configured	Awaiting configuration
Radarr	Movie collection manager	Private	Plato	SQLite**	Internal backups	Not configured	Not configured	Not configured	Awaiting configuration
Jackett	Torrent indexer	Private	Plato	-	?	Not configured	Not configured	Not configured	Awaiting configuration
Deluge	Torrent client	Private	Plato	-	?	-	Not configured	Not configured	Awaiting configuration
Minecraft	Vanilla minecraft server for friends	Public	Archimedes	-	Game map	Not configured	Not configured	Not configured	Awaiting configuration
Satisfactory	Satisfactory server for friends	Public	Archimedes	-	Game map	Not configured	Not configured	Not configured	Not needed for v1
Space engineers	Space engineers server for friends	Public	Archimedes	-	Game map	Not configured	Not configured	Not configured	Not needed for v1
Raspsnir	Bachelor memorial website	Public	Pythagoras-b	PostgreSQL	-	Not configured	Not configured	Not configured	Not needed for v1

* Configuration panel only available internally
** Current implementation only support SQLite, making manual backups a necessity
¹ Missing automated configuration pipeline for environment variable injection
² Missing configuration for NAS volume mounting (over network)
³ Missing Longhorn scheduling for saving media_store and secret management
⁴ Backup management is not handled by k3s but by an internal cronjob rule (Change image name when putting to production)
⁵ Missing dashboard configuration

Backup management

Databases

All services needing a database to function come with a sidecar pod running a crontab to automate individual database backups. These backups are saved into a longhorn volume, to benefit from general snapshots later one. Each sidecar pod can only mount the backup folder it has been linked with, and cannot see other services' backups.

Additional data

All additional data needing to be backed up is mounted to a longhorn volume, to also benefit from scheduled backups.

Example :

longhorn
└───backups
    └───vaultwarden
    │   └───<backup_date>.sql
    │   │   ...
    └───gitlab
        └───<backup_date>.sql
        │   ...

TODO

• Change host/deployment specific variables to use environment variables
• Write CI/CD pipeline to create environment loaded files
• Write CI/CD pipeline to deploy cluster
• Setup internal traefik with nodeport as reverse proxy for internal only services Done through double ingress class and LB
• Setup DB container sidecars for automated backups to Longhorn volume
• Setup secrets configuration through CI/CD variable injection
• Explore permission issues when issuing OVH API keys (not working for wildcard and beta.halia.dev subdomain)
• Setup default users for services
• Setup log and metric monitoring
• Define namespaces through yaml files
• Look into CockroachDB for redundant database Judged too complicated, moving to a 1 to 1 relationship between services and databases
• Configure IP range accessibility through Traefik (Internal vs external services) Impossible because of flannel ip-masq

Notes

Cluster base setup

Add node to the list of available load balancer kubectl label node <node-name> svccontroller.k3s.cattle.io/enablelb=true NOTE: For development, don't forget to also add the cp to the LB list, in order to access local only services

Setup OVH configuration kubectl apply -f ovh-config.yaml

Install traefik kubectl apply -f traefik

Install longhorn

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml

Remove local-path from default StorageClass kubectl patch storageclass standard -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

Add longhorn storage classes kubectl apply -f res

Convert helm chart to k3s manifest

helm template chart stable/chart --output-dir ./chart

Gitlab backup process

Because gitlab does not offer the possibility to backup a container's data from an external container, a cronjob has been implemented in the custom image used for deployment.

VPN configuration for Deluge

Instead of adding an extra networking layer to the whole cluster, it seems like a better idea to just integrate a wireguard connection inside of the deluge image, and self-build everything within Gitlab registry. This image could utilize kubernetes secrets, including a "torrent-vpn" secret produces by the initial wireguard configuration done via Ansible. This ansible script could create one (or more) additional client(s) depending on the inventory configuration, and keep the "torrent-vpn" configuration file within a k3s formated file, inside of the auto-applied directory on CP. Cf : https://docs.k3s.io/advanced#auto-deploying-manifests

Development domains

To access a service publicly when developing, the domain name should be *.beta.halia.dev To only expose a service internally, the domain name should be *.k3s.beta

Ingresses

To split between external and internal services, two traefik ingresses are implemented through the ingressclass annotation. traefik-external will only allow external access to a given service, while traefik-internal restrict to an internal only access.