k8s-cluster hosts customer-facing web services. It runs
on two Talos Linux VMs under Incus on the server, with Cilium as CNI,
the Ceph RBD CSI driver for persistent volumes, and ArgoCD for GitOps
delivery. The host’s Traefik is the ingress controller — the cluster
runs neither an in-cluster Traefik nor cert-manager.
Design goals: robustness, horizontal scalability by adding nodes to the WireGuard mesh, and durability of customer data.
Talos is immutable and API-only: there is no SSH and no package
manager, so every node change goes through talosctl against
a machine config.
Everything speaks over the WireGuard mesh, so cluster traffic never leaves the VPN.
flowchart TD
net[Internet] --> tr["Host Traefik<br/>10.100.0.1:443"]
tr --> nixos["NixOS services<br/>git.yhkze.net, id.yhkze.net, …"]
tr -->|IngressRoute CRDs| pods
subgraph mesh["WireGuard mesh 10.100.0.0/24"]
srv["10.100.0.1 server<br/>Traefik · Forgejo · Ceph MON/OSD · RadosGW"]
master["10.100.0.10 k8s-master<br/>control plane · etcd · API :6443"]
worker["10.100.0.11 k8s-worker<br/>all workloads"]
end
pods[Pods on k8s-worker] --- worker
worker -->|rbd :6789 · S3 :7480| srv
Additional nodes join by adding a WireGuard peer and applying a Talos worker config — the host Traefik then discovers their pods through the Kubernetes API with no configuration change.
Both VMs live on the server in the Incus local btrfs
pool, so neither depends on shared storage to boot.
| VM | Role | vCPU | RAM | Disk | WireGuard IP |
|---|---|---|---|---|---|
k8s-master |
Talos control plane + etcd | 2 | 6 GB | 40 GB | 10.100.0.10 |
k8s-worker |
All workloads | 6 | 6 GB | 150 GB | 10.100.0.11 |
Platform versions in use:
| Item | Value |
|---|---|
| Talos | v1.13.2 |
| Kubernetes | v1.36.0 |
| Container runtime | containerd 2.2.3 |
| Pod CIDR | 10.244.0.0/16 |
| Service CIDR | 172.20.0.0/16 |
| KubePrism | localhost:7445 (Talos’ built-in API proxy, used by
Cilium) |
| Incus bridge | 10.200.3.0/24 (DHCP, bootstrap only) |
| Factory schematic | 613e1592…961245 (iscsi-tools,
util-linux-tools) |
The service CIDR is deliberately off the Kubernetes default of 10.96.0.0/12 so it cannot collide with the WireGuard mesh.
The schematic still carries iscsi-tools, which only
Longhorn needed. The Ceph RBD CSI driver uses the kernel rbd client
instead, so the extension is inert — dropping it would mean reimaging
both nodes, which is why it stays.
| Component | Namespace | Version | Purpose |
|---|---|---|---|
| Cilium | kube-system |
1.19.4 | CNI, eBPF networking, kube-proxy replacement |
| Ceph RBD CSI | ceph-csi-rbd |
3.18.1 | Block storage for PVCs, ceph-rbd StorageClass |
| ArgoCD | argocd |
3.4.2 | GitOps CD, watches the Forgejo gitops org |
| Metrics Server | kube-system |
0.8.0 | Resource metrics for HPA |
| Forgejo Runners | ci |
12.10.1 | CI runners with DinD sidecar, HPA 1-4 replicas |
| Telemetry collection | observability |
— | OpenTelemetry operator and collectors, OBI, kube-state-metrics, node-exporter — storage lives on the server, reference Observability |
| OpenBao | openbao |
2.4.4 | Secret management — reference OpenBao |
| CloudNativePG | cnpg-system |
1.30.1 | PostgreSQL for product apps, Barman Cloud backups to RGW — reference PostgreSQL |
Cilium, Metrics Server and ArgoCD itself are Helm releases created
during bootstrap; the Forgejo runner is a manifest in
talos-config/manifests/. The Ceph CSI driver, the telemetry
collectors and OpenBao are reconciled by ArgoCD from the charts
repo.
Deliberately not in-cluster: Traefik (runs on the host), and the PostgreSQL and Redis of platform services (native NixOS services). Product applications get their PostgreSQL from CloudNativePG.