Tanguy Herbron
0a41373688
A new traefik deployment has been added, and two ingressclasses have been set for the new instance, as well as the previous one. This allows the network to be split in two, one for external access and another for internal access. Each traefik deployment is connected to a loadbalancer requesting the IP necessary for each type of access.
117 lines
10 KiB
Markdown
117 lines
10 KiB
Markdown
# K3s cluster
|
|
|
|
/!\ CHECK PORT RULES FOR EXPOSING INTERNAL SERVICES
|
|
|
|
| Name | Usage | Accessibility | Host | DB type | Additional data | Backup configuration | Loki integration | Prometheus integration | Status |
|
|
|-------------------------|--------------------------------------|---------------|--------------|------------|----------------------|----------------------|------------------|------------------------|-----------------------------------|
|
|
| Traefik | Reverse proxy and load balancer | Public* | Socrates | - | - | - | Configured | Configured | Completed<sup>5</sup> |
|
|
| Vaultwarden | Password manager | Public | Pythagoras-b | MariaDB | - | 4AM K8s CronJob | Configured | Not available | Completed |
|
|
| Gitlab | Version control system | Public | Pythagoras-b | PostgreSQL | User created content | 5AM internal CronJob | Configured | Configured | Completed<sup>4</sup> |
|
|
| Prometheus | Metrics aggregator | Private | Pythagoras-b | TBD | - | Not configured | Configured | Configured | Partial |
|
|
| Loki | Log aggregator | Private | Pythagoras-b | TBD | - | Not configured | Configured | Configured | Partial |
|
|
| Grafana | Graph visualizer | Public | Pythagoras-b | - | - | Not configured | Configured | Configured | Partial |
|
|
| Adguard | DNS ad blocker and custom DNS server | Private | Socrates | - | - | - | Not configured | Not configured | Pending configuration<sup>1</sup> |
|
|
| Owncloud Infinity Scale | File hosting webUI | Public | Plato | ? | Drive files | Not configured | Configured | Not available | Pending configuration<sup>2</sup> |
|
|
| Synapse | Matrix server - Message centralizer | Public | Pythagoras-b | PostgreSQL | User medias | 4AM K8s CronJob | Configured | Configured | Pending configuration<sup>3</sup> |
|
|
| therbron.com | Personal website | Public | Socrates | - | - | - | Not configured | Not configured | Awaiting configuration |
|
|
| Home assistant | Home automation and monitoring | Private | Pythagoras-a | MariaDB | - | Not configured | Not configured | Not configured | Awaiting configuration |
|
|
| Vikunja | To-do and Kanban boards | Public | Pythagoras-b | - | - | - | Not configured | Not configured | Migrate to Gitlab |
|
|
| Wiki | Documentation manager | Public | Pythagoras-b | - | - | - | Not configured | Not configured | Migrate to VuePress and Gitlab |
|
|
| PaperlessNG | PDF viewer and organiser | Public | Pythagoras-b | PostgreSQL | - | - | Not configured | Not configured | Research migration into OCIS |
|
|
| Jellyfin | Media streaming | Public | Archimedes | - | - | - | Not configured | Not configured | Awaiting configuration |
|
|
| Sonarr | TV shows collection manager | Private | Plato | SQLite** | Internal backups | Not configured | Not configured | Not configured | Awaiting configuration |
|
|
| Radarr | Movie collection manager | Private | Plato | SQLite** | Internal backups | Not configured | Not configured | Not configured | Awaiting configuration |
|
|
| Jackett | Torrent indexer | Private | Plato | - | <center> ? </center> | Not configured | Not configured | Not configured | Awaiting configuration |
|
|
| Deluge | Torrent client | Private | Plato | - | <center> ? </center> | - | Not configured | Not configured | Awaiting configuration |
|
|
| Minecraft | Vanilla minecraft server for friends | Public | Archimedes | - | Game map | Not configured | Not configured | Not configured | Awaiting configuration |
|
|
| Satisfactory | Satisfactory server for friends | Public | Archimedes | - | Game map | Not configured | Not configured | Not configured | Not needed for v1 |
|
|
| Space engineers | Space engineers server for friends | Public | Archimedes | - | Game map | Not configured | Not configured | Not configured | Not needed for v1 |
|
|
| Raspsnir | Bachelor memorial website | Public | Pythagoras-b | PostgreSQL | - | Not configured | Not configured | Not configured | Not needed for v1 |
|
|
|
|
\* Configuration panel only available internally<br>
|
|
** Current implementation only support SQLite, making manual backups a necessity<br>
|
|
<sup>1</sup> Missing automated configuration pipeline for environment variable injection<br>
|
|
<sup>2</sup> Missing configuration for NAS volume mounting (over network)<br>
|
|
<sup>3</sup> Missing Longhorn scheduling for saving media_store and secret management<br>
|
|
<sup>4</sup> Backup management is not handled by k3s but by an internal cronjob rule (Change image name when putting to production)<br>
|
|
<sup>5</sup> Missing dashboard configuration<br>
|
|
|
|
## Backup management
|
|
### Databases
|
|
All services needing a database to function come with a sidecar pod running a crontab to automate individual database backups.
|
|
These backups are saved into a longhorn volume, to benefit from general snapshots later one.
|
|
Each sidecar pod can only mount the backup folder it has been linked with, and cannot see other services' backups.
|
|
|
|
### Additional data
|
|
All additional data needing to be backed up is mounted to a longhorn volume, to also benefit from scheduled backups.
|
|
|
|
Example :
|
|
```
|
|
longhorn
|
|
└───backups
|
|
└───vaultwarden
|
|
│ └───<backup_date>.sql
|
|
│ │ ...
|
|
└───gitlab
|
|
└───<backup_date>.sql
|
|
│ ...
|
|
```
|
|
## TODO
|
|
- Change host/deployment specific variables to use environment variables
|
|
- Write CI/CD pipeline to create environment loaded files
|
|
- Write CI/CD pipeline to deploy cluster
|
|
- ~~Setup internal traefik with nodeport as reverse proxy for internal only services~~ Done through double ingress class and LB
|
|
- ~~Setup DB container sidecars for automated backups to Longhorn volume~~
|
|
- Setup secrets configuration through CI/CD variable injection
|
|
- Explore permission issues when issuing OVH API keys (not working for wildcard and `beta.halia.dev` subdomain)
|
|
- Setup default users for services
|
|
- ~~Setup log and metric monitoring~~
|
|
- ~~Define namespaces through yaml files~~
|
|
- ~~Look into CockroachDB for redundant database~~ Judged too complicated, moving to a 1 to 1 relationship between services and databases
|
|
- ~~Configure IP range accessibility through Traefik (Internal vs external services)~~ Impossible because of flannel ip-masq
|
|
|
|
## Notes
|
|
### Cluster base setup
|
|
Add node to the list of available load balancer
|
|
`kubectl label node <node-name> svccontroller.k3s.cattle.io/enablelb=true`
|
|
NOTE: For development, don't forget to also add the `cp` to the LB list, in order to access local only services
|
|
|
|
|
|
Setup OVH configuration
|
|
`kubectl apply -f ovh-config.yaml`
|
|
|
|
Install traefik
|
|
`kubectl apply -f traefik`
|
|
|
|
Install longhorn
|
|
```
|
|
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml
|
|
```
|
|
Remove `local-path` from default StorageClass
|
|
`kubectl patch storageclass standard -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'`
|
|
|
|
Add longhorn storage classes
|
|
`kubectl apply -f res`
|
|
|
|
### Convert helm chart to k3s manifest
|
|
`helm template chart stable/chart --output-dir ./chart`
|
|
|
|
### Gitlab backup process
|
|
Because gitlab does not offer the possibility to backup a container's data from an external container, a cronjob has been implemented in the custom image used for deployment.
|
|
|
|
### VPN configuration for Deluge
|
|
Instead of adding an extra networking layer to the whole cluster, it seems like a better idea to just integrate a wireguard connection inside of the deluge image, and self-build everything within Gitlab registry.
|
|
This image could utilize kubernetes secrets, including a "torrent-vpn" secret produces by the initial wireguard configuration done via Ansible.
|
|
This ansible script could create one (or more) additional client(s) depending on the inventory configuration, and keep the "torrent-vpn" configuration file within a k3s formated file, inside of the auto-applied directory on CP.
|
|
Cf : https://docs.k3s.io/advanced#auto-deploying-manifests
|
|
|
|
### Development domains
|
|
To access a service publicly when developing, the domain name should be *.beta.halia.dev
|
|
To only expose a service internally, the domain name should be *.k3s.beta
|
|
|
|
### Ingresses
|
|
To split between external and internal services, two traefik ingresses are implemented through the `ingressclass` annotation.
|
|
`traefik-external` will only allow external access to a given service, while `traefik-internal` restrict to an internal only access.
|
|
|
|
|