Cluster Architecture

Overview

k8s-cluster hosts customer-facing web services. It runs on two Talos Linux VMs under Incus on the server, with Cilium as CNI, the Ceph RBD CSI driver for persistent volumes, and ArgoCD for GitOps delivery. The host’s Traefik is the ingress controller — the cluster runs neither an in-cluster Traefik nor cert-manager.

Design goals: robustness, horizontal scalability by adding nodes to the WireGuard mesh, and durability of customer data.

Talos is immutable and API-only: there is no SSH and no package manager, so every node change goes through talosctl against a machine config.

Topology

Everything speaks over the WireGuard mesh, so cluster traffic never leaves the VPN.

flowchart TD
    net[Internet] --> tr["Host Traefik<br/>10.100.0.1:443"]
    tr --> nixos["NixOS services<br/>git.yhkze.net, id.yhkze.net, …"]
    tr -->|IngressRoute CRDs| pods

    subgraph mesh["WireGuard mesh 10.100.0.0/24"]
        srv["10.100.0.1 server<br/>Traefik · Forgejo · Ceph MON/OSD · RadosGW"]
        master["10.100.0.10 k8s-master<br/>control plane · etcd · API :6443"]
        worker["10.100.0.11 k8s-worker<br/>all workloads"]
    end

    pods[Pods on k8s-worker] --- worker
    worker -->|rbd :6789 · S3 :7480| srv

Additional nodes join by adding a WireGuard peer and applying a Talos worker config — the host Traefik then discovers their pods through the Kubernetes API with no configuration change.

Nodes

Both VMs live on the server in the Incus local btrfs pool, so neither depends on shared storage to boot.

VM Role vCPU RAM Disk WireGuard IP
k8s-master Talos control plane + etcd 2 6 GB 40 GB 10.100.0.10
k8s-worker All workloads 6 6 GB 150 GB 10.100.0.11

Platform versions in use:

Item Value
Talos v1.13.2
Kubernetes v1.36.0
Container runtime containerd 2.2.3
Pod CIDR 10.244.0.0/16
Service CIDR 172.20.0.0/16
KubePrism localhost:7445 (Talos’ built-in API proxy, used by Cilium)
Incus bridge 10.200.3.0/24 (DHCP, bootstrap only)
Factory schematic 613e1592…961245 (iscsi-tools, util-linux-tools)

The service CIDR is deliberately off the Kubernetes default of 10.96.0.0/12 so it cannot collide with the WireGuard mesh.

The schematic still carries iscsi-tools, which only Longhorn needed. The Ceph RBD CSI driver uses the kernel rbd client instead, so the extension is inert — dropping it would mean reimaging both nodes, which is why it stays.

In-Cluster Components

Component Namespace Version Purpose
Cilium kube-system 1.19.4 CNI, eBPF networking, kube-proxy replacement
Ceph RBD CSI ceph-csi-rbd 3.18.1 Block storage for PVCs, ceph-rbd StorageClass
ArgoCD argocd 3.4.2 GitOps CD, watches the Forgejo gitops org
Metrics Server kube-system 0.8.0 Resource metrics for HPA
Forgejo Runners ci 12.10.1 CI runners with DinD sidecar, HPA 1-4 replicas
Telemetry collection observability — OpenTelemetry operator and collectors, OBI, kube-state-metrics, node-exporter — storage lives on the server, reference Observability
OpenBao openbao 2.4.4 Secret management — reference OpenBao
CloudNativePG cnpg-system 1.30.1 PostgreSQL for product apps, Barman Cloud backups to RGW — reference PostgreSQL

Cilium, Metrics Server and ArgoCD itself are Helm releases created during bootstrap; the Forgejo runner is a manifest in talos-config/manifests/. The Ceph CSI driver, the telemetry collectors and OpenBao are reconciled by ArgoCD from the charts repo.

Deliberately not in-cluster: Traefik (runs on the host), and the PostgreSQL and Redis of platform services (native NixOS services). Product applications get their PostgreSQL from CloudNativePG.