Reference for how k8s-cluster is built from nothing: the
two Talos VMs, the Helm releases that are not under GitOps, the Traefik
credential that lets the host serve cluster ingress, and the single
manual apply that hands the rest to ArgoCD. Follow it in order — each
stage depends on the one before.
For recovery decisions (what is lost in which failure mode) reference Operations.
Tooling on the workstation and the server:
talosctl v1.13.2 — the nixpkgs version may lag, in
which case take the binary from the Talos
releaseskubectl and helm — already in
hosts/server/nixos/packages.nix and
values/package-bundle.nixServer-side state that must already exist:
kubernetes pool, and
RadosGW running if S3 is wanted, reference Storagevalues/network-details.nixRun talosctl apply-config from the server. The
workstation has no route to the Incus bridge subnet 10.200.3.0/24, which
is the only address the nodes answer on before their WireGuard config
lands.
Everything Talos needs lives in talos-config/. Only the
patches are tracked; the machine configs are generated artefacts and the
rest hold private keys.
| Path | Tracked | Contents |
|---|---|---|
patches/ |
yes | The config patches listed below |
manifests/ |
yes | Raw manifests applied by hand during bootstrap |
secrets.yaml |
no | Cluster PKI, including the CA private key |
controlplane.yaml, worker.yaml |
no | Generated machine configs |
talosconfig |
no | talosctl client credentials |
patches/wireguard-{master,worker}.yaml |
no | Per-node WireGuard private key |
The patches each isolate one decision:
install-image.yaml — factory installer with
iscsi-tools and util-linux-toolscni-none.yaml — disables the bundled Flannel CNIcilium-no-kube-proxy.yaml — disables kube-proxy, which
Cilium replaceskubelet-wireguard-ip.yaml — binds the kubelet to the
WireGuard subnetkubelet-extra-args.yaml — enables kubelet certificate
rotationnetwork-subnets.yaml — pod and service CIDRsregistry-mirrors.yaml — pulls through the server’s
registry cacheswireguard-{master,worker}.yaml — the node’s mesh
address and peerBecause the two WireGuard patches are gitignored, a repo-wide search never sees them. Reference the port-coupling note in Networking before changing any mesh port.
Generate secrets only when building a genuinely new cluster. Reusing
an existing secrets.yaml preserves the cluster identity and
its PKI, which is what makes a rebuild a recovery rather than a
replacement:
cd /persist/home/user/nixos/talos-config
talosctl gen secrets -o secrets.yamlRender the machine configs, passing every patch. The WireGuard patches are role-specific, so they use the control-plane and worker variants of the flag:
talosctl gen config k8s-cluster https://10.100.0.10:6443 \
--with-secrets secrets.yaml \
--config-patch @patches/install-image.yaml \
--config-patch @patches/cilium-no-kube-proxy.yaml \
--config-patch @patches/cni-none.yaml \
--config-patch @patches/kubelet-wireguard-ip.yaml \
--config-patch @patches/kubelet-extra-args.yaml \
--config-patch @patches/network-subnets.yaml \
--config-patch @patches/registry-mirrors.yaml \
--config-patch-control-plane @patches/wireguard-master.yaml \
--config-patch-worker @patches/wireguard-worker.yaml \
--output-dir .Fetch the installer ISO for the pinned factory schematic, which bakes in the two system extensions the cluster relies on:
SCHEMATIC="613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245"
curl -Lo /tmp/talos-factory.iso \
"https://factory.talos.dev/image/${SCHEMATIC}/v1.13.2/metal-amd64.iso"Create both VMs on the server, attaching the ISO at a boot priority that wins over the empty root disk:
incus init --empty k8s-master --vm --profile local --target server \
-c limits.cpu=2 -c limits.memory=6GiB
incus config device override k8s-master root size=40GiB
incus config device add k8s-master install disk \
source=/tmp/talos-factory.iso boot.priority=10
incus start k8s-master
incus init --empty k8s-worker --vm --profile local --target server \
-c limits.cpu=6 -c limits.memory=12GiB
incus config device override k8s-worker root size=150GiB
incus config device add k8s-worker install disk \
source=/tmp/talos-factory.iso boot.priority=10
incus start k8s-workerApply the configs from the server, addressing each node by the bridge
IP that incus list reports — this is before WireGuard
exists, so the mesh addresses do not answer yet:
talosctl apply-config --insecure --nodes <master-bridge-ip> --file controlplane.yaml
talosctl apply-config --insecure --nodes <worker-bridge-ip> --file worker.yamlDetach the ISOs once both nodes have installed to disk, or they boot the installer again instead of the system:
incus config device remove k8s-master install
incus config device remove k8s-worker installBootstrap etcd on the control plane and pull the kubeconfig. The second command uses the mesh address, which is live by now:
export TALOSCONFIG=/persist/home/user/nixos/talos-config/talosconfig
talosctl bootstrap --nodes <master-bridge-ip> --endpoints <master-bridge-ip>
talosctl kubeconfig --nodes 10.100.0.10 --endpoints 10.100.0.10 ~/.kube/configApprove the kubelet CSRs. Until the nodes hold serving certificates,
kubectl logs, kubectl exec and metrics all
fail:
kubectl get csr -o name | xargs kubectl certificate approveNothing else can proceed before the CNI: without it no pod has
networking and both nodes stay NotReady. Talos needs
specific values — cgroup automount off because the host path is already
mounted, the KubePrism endpoint instead of a Service IP, and explicit
capabilities because Talos ships no permissive defaults.
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium \
--namespace kube-system \
--version 1.19.4 \
--set ipam.mode=kubernetes \
--set kubeProxyReplacement=true \
--set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
--set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
--set cgroup.autoMount.enabled=false \
--set cgroup.hostRoot=/sys/fs/cgroup \
--set k8sServiceHost=localhost \
--set k8sServicePort=7445 \
--set bpf.hostLegacyRouting=trueWait for the DaemonSet and confirm the nodes flip to
Ready:
kubectl -n kube-system rollout status daemonset/cilium
kubectl get nodesThe pod CIDR is an AllowedIP on the k8s-worker peer in
modules/features/wireguard.nix. If the server has not been
rebuilt since that was added, run rebuild 10.100.0.1.
ArgoCD is installed with Helm rather than by itself, since it is the
thing that installs everything else. server.insecure=true
disables its internal TLS because the host Traefik terminates TLS.
helm repo add argo https://argoproj.github.io/argo-helm
kubectl create namespace argocd
kubectl label namespace argocd pod-security.kubernetes.io/enforce=baseline
helm install argocd argo/argo-cd \
--namespace argocd \
--version 9.5.14 \
--set server.service.type=ClusterIP \
--set configs.params."server\.insecure"=trueRead the generated admin password and store it in the password manager:
kubectl -n argocd get secret argocd-initial-admin-secret \
-o jsonpath="{.data.password}" | base64 -dMetrics Server backs the HPA on the CI runners, and needs
--kubelet-insecure-tls because the kubelet serving
certificates are signed by the cluster CA rather than a publicly rooted
one:
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm install metrics-server metrics-server/metrics-server \
--namespace kube-system \
--version 3.13.0 \
--set args="{--kubelet-insecure-tls}"The host Traefik needs a kubeconfig to read IngressRoute
CRDs. Apply the RBAC first:
kubectl apply -f talos-config/manifests/traefik-rbac.yamlBuild a kubeconfig from the ServiceAccount token and the cluster CA:
TOKEN=$(kubectl -n traefik-external get secret traefik-token \
-o jsonpath='{.data.token}' | base64 -d)
CA=$(kubectl -n traefik-external get secret traefik-token \
-o jsonpath='{.data.ca\.crt}')
cat <<EOF
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority-data: ${CA}
server: https://10.100.0.10:6443
name: talos
contexts:
- context:
cluster: talos
user: traefik
name: traefik@talos
current-context: traefik@talos
users:
- name: traefik
user:
token: ${TOKEN}
EOFStore that output in SOPS under
kubernetes/traefik-kubeconfig, then deploy the server so
Traefik picks it up:
sops hosts/server/secrets/secrets.yaml
rebuild 10.100.0.1No chart in the cluster installs Traefik, so the
IngressRoute CRDs come straight from upstream, pinned to
the host Traefik’s version:
V=$(ssh 10.100.0.1 'systemctl show traefik -p ExecStart --value' \
| grep -oE 'traefik-[0-9.]+' | head -1 | cut -d- -f2)
kubectl apply --server-side -f \
"https://raw.githubusercontent.com/traefik/traefik/v${V}/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml"Expose ArgoCD itself, which is the last step that uses a raw manifest:
kubectl apply -f talos-config/manifests/argocd-ingress.yamlArgoCD is then reachable at https://argocd.yhkze.net.
Two secrets are deliberately outside git, and ArgoCD cannot bring their charts to a healthy state without them. Create both before the root Application syncs.
The Ceph CSI credential, scoped to the kubernetes pool —
the full recipe is in the header comment of
ceph-csi-rbd/values.yaml in the charts repo:
kubectl create namespace ceph-csi-rbd
kubectl -n ceph-csi-rbd create secret generic csi-rbd-secret \
--from-literal=userID=kubernetes \
--from-literal=userKey="$(ssh 10.100.0.1 "sudo ceph auth get-key client.kubernetes")"The Forgejo Actions runner registration token, taken from
Site Admin → Actions → Runners → Create New Runner. The
UUID that registration returns goes in the chart’s
values.yaml as runner.uuid:
kubectl create namespace ci
kubectl -n ci create secret generic runner-secret --from-literal=token='<TOKEN>'ArgoCD reads both gitops repos as the Forgejo machine
user argocd, whose team grants read-only access. A rebuilt
cluster cannot reuse the old key (its private half lived in the old
cluster), so create a new pair, add the public half to
argocd under
Site Administration → User Accounts → argocd → SSH Keys (or
POST /api/v1/admin/users/argocd/keys), and delete the stale
entry:
ssh-keygen -t ed25519 -C argocd-k8s-repo-server -f /tmp/argocd-gitops -N ""Pin Forgejo’s host key through configs.ssh.extraHosts so
it survives Helm upgrades, taking it from the unproxied record on port
30354. The repo server reads the known hosts only at start, so restart
it afterwards:
ssh-keyscan -p 30354 -t rsa ssh-git.yhkze.net
helm upgrade argocd argo-cd \
--repo https://argoproj.github.io/argo-helm \
--namespace argocd --reuse-values \
--set "configs.ssh.extraHosts=[ssh-git.yhkze.net]:30354 ssh-rsa <KEY>"
kubectl -n argocd rollout restart deploy/argocd-repo-serverCreate one repository credential per repo:
for repo in apps charts; do
kubectl -n argocd create secret generic "repo-gitops-${repo}" \
--from-literal=type=git \
--from-literal=url="ssh://[email protected]:30354/gitops/${repo}.git" \
--from-file=sshPrivateKey=/tmp/argocd-gitops \
--dry-run=client -o json | \
jq ".metadata.labels[\"argocd.argoproj.io/secret-type\"] = \"repository\"" | \
kubectl apply -f -
doneApply the root Application once. From here ArgoCD is self-managing:
it reconciles gitops/apps recursively, which creates the
platform Applications and the development-services
ApplicationSet. Reference GitOps and
CI/CD.
kubectl apply -n argocd -f \
<(curl -sL https://git.yhkze.net/gitops/apps/raw/branch/master/root-app.yaml)The argocd Application syncs manually by design; trigger
its first sync once the root Application has created it, which replaces
the Helm defaults with the declared configuration (Keycloak SSO, RBAC,
known hosts):
kubectl -n argocd patch application argocd --type merge \
-p '{"operation":{"sync":{"syncOptions":["ServerSideApply=true"]}}}'OpenBao runs on the server, outside the cluster, and validates
External Secrets’ ServiceAccount tokens through a reviewer token issued
by this cluster. Once the external-secrets Application has
created the openbao-token-reviewer Secret, write it and the
cluster CA into OpenBao’s Kubernetes auth as in OpenBao. The roles
survive a rebuild; only that config has to be rewritten. Until then
every SecretStore reports InvalidProviderConfig and logins
return 403.