Homelab: Kubernetes Home Cluster - Applications

This repository contains applications deployed on the home-cluster via Flux using GitOps.

---

Bootstrapping

A Kubernetes cluster needs to be bootstrapped with the Cilium CNI and Flux pointing to this repository.

For ksops and ArgoCD to decrypt the initial secrets for configuring the External Secrets Operator using Doppler, a Google Cloud Service Account with access to the correct KMS key needs to be set in the flux namespace.

Attention: some applications will be automatically deployed, others not (yet).

---

App-of-Apps

The repository follows the app-of-apps pattern.

The first Flux Kustomization being defined needs to reference app-of-apps/.

These are bootstrapping the main Flux applications, referring to the respective <PROJECT>/applications/ kosutomizations:

• infrastructure: core cluster infrastructure
• core: core applications
• applications: (user) applications running on the cluster/network
• home-assistant: Home Assistant related applications

Each of these applications follows the app-of-apps pattern again using sub-kustomizations defined in the respective application directories.

---

Hosted Services

Infrastructure

The following applications are defined in infrastructure/.

• Cilium - Provides the cluster CNI.
• External Secrets Operator - Synchronizes secrets from external stores to Kubernetes Secret objects.
  • External Secrets Stores - Deploys the required ClusterSecretStores and Vault credentials as Kubernetes Secrets.
• Kubelet Serving Cert Approver - Enables automatic certificate approval by the kubelet.
• MetalLB - Provides a Kubernetes network load balancer to expose Kubernetes Services.
• Metrics Server - Collects container resource metrics.
• NVIDIA Device Plugin - Makes the NVIDIA GPU accessible in the cluster.
• Rook Ceph - Manages persistent storage in the cluster.
• Traefik - Exposes Kubernetes Ingress resources to the "outside world".

Core Applications

The following applications are defined in core/.

• Cert Manager - Certificate management using ACME Let's Encrypt.
• CloudNativePG - PostgreSQL database operator.
• External DNS with Google Cloud DNS integration - Creates DNS records in Google Cloud DNS domains for publicly reachable services.
• Victoria Metrics Stack - Monitoring stack using Victoria Metrics.
• Loki Stack - Logging stack using Grafana Loki.
• Grafana - Visualization of metrics, and other data.
• Velero - Performs cluster backups.
  • Includes deployment of backup schedules.

(User) Applications

The following applications are defined in applications/.

• External DNS with Adguard integration - Creates DNS records in Adguard for internal, local only, reachable services. Uses the External DNS Adguard Webhook provider.
• External Services - Deploys Kubernetes Services and Ingresses to local endpoints, and existing services outside of the cluster.
• Gatus - Service status page.
• Immich - Photo management solution.
• InfluxDB - Time-Series database.
• LibreChat - Open Source AI platform integrating with multiple models.
• Ollama - Run LLM models locally.
• Samba - Exposes Samba shares for various applications.

Home Assistant

The following applications are defined in home-assistant/.

• ecowitt2mqtt - Forwards data received from ecowitt devices to the MQTT broker.
• EMQX - A MQTT broker.
• Home Assistant - The Home Assistant instance.
  • PostgreSQL instance as the Home Assistant recorder target and configured via the CloudNativePG operator.
• Node-RED - Automation based on flows and Home Assistant data.
• Ring MQTT - Amazon Ring devices to MQTT bridge.
• Telegraf - Forwards Home Assistant state changes to a local InfluxDB instance.
• Z-Wave JS - Full featured Z-Wave Control Panel and MQTT Gateway.

Notes: Backup and Restore

Home Assistant related backup and restore is handled via S3 backups.

The following services implement an initContainer as well as a nightly CronJob to backup data to an S3 bucket. If no data is found in the Persistent Volume yet, the data from will be retrieved and copied over which results in a full restore.

• Ring MQTT

The following services use API calls to determine whether a backup or restore is necessary.

• Node-RED
• Home Assistant
• Z-Wave JS UI

The following services also have Git repositories to store their configuration which gets pulled in upon start.

• Home Assistant
  • Home Assistant also defines it's own backup method via a trigger and a shell_command, and doesn't rely on a CronJob.
• Ring MQTT

---

Backup and Restore

The current backup and restore strategy consists of:

• CloudNativePG backups for persistent PostgreSQL data
• Home Assistant: see (#notes-backup-and-restore)
• Velero as a second layer disaster recovery for critical workloads

Timewise, the layers of backups follow the strategy:

1. 12:00am: in-application backups
2. 02:00am: Velero backups

---

Continuous Integration and Automations

• GitHub Actions are linting all YAML files.
• Renovate Bot is updating Helm releases and used container images in the values.yaml files, and GitHub Actions.