Cluster Bootstrap

Scope

Reference for how k8s-cluster is built from nothing: the two Talos VMs, the Helm releases that are not under GitOps, the Traefik credential that lets the host serve cluster ingress, and the single manual apply that hands the rest to ArgoCD. Follow it in order — each stage depends on the one before.

For recovery decisions (what is lost in which failure mode) reference Operations.

Prerequisites

Tooling on the workstation and the server:

Server-side state that must already exist:

Run talosctl apply-config from the server. The workstation has no route to the Incus bridge subnet 10.200.3.0/24, which is the only address the nodes answer on before their WireGuard config lands.

Machine Config Layout

Everything Talos needs lives in talos-config/. Only the patches are tracked; the machine configs are generated artefacts and the rest hold private keys.

Path Tracked Contents
patches/ yes The config patches listed below
manifests/ yes Raw manifests applied by hand during bootstrap
secrets.yaml no Cluster PKI, including the CA private key
controlplane.yaml, worker.yaml no Generated machine configs
talosconfig no talosctl client credentials
patches/wireguard-{master,worker}.yaml no Per-node WireGuard private key

The patches each isolate one decision:

Because the two WireGuard patches are gitignored, a repo-wide search never sees them. Reference the port-coupling note in Networking before changing any mesh port.

Talos Nodes

Generate secrets only when building a genuinely new cluster. Reusing an existing secrets.yaml preserves the cluster identity and its PKI, which is what makes a rebuild a recovery rather than a replacement:

cd /persist/home/user/nixos/talos-config
talosctl gen secrets -o secrets.yaml

Render the machine configs, passing every patch. The WireGuard patches are role-specific, so they use the control-plane and worker variants of the flag:

talosctl gen config k8s-cluster https://10.100.0.10:6443 \
  --with-secrets secrets.yaml \
  --config-patch @patches/install-image.yaml \
  --config-patch @patches/cilium-no-kube-proxy.yaml \
  --config-patch @patches/cni-none.yaml \
  --config-patch @patches/kubelet-wireguard-ip.yaml \
  --config-patch @patches/kubelet-extra-args.yaml \
  --config-patch @patches/network-subnets.yaml \
  --config-patch @patches/registry-mirrors.yaml \
  --config-patch-control-plane @patches/wireguard-master.yaml \
  --config-patch-worker @patches/wireguard-worker.yaml \
  --output-dir .

Fetch the installer ISO for the pinned factory schematic, which bakes in the two system extensions the cluster relies on:

SCHEMATIC="613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245"
curl -Lo /tmp/talos-factory.iso \
  "https://factory.talos.dev/image/${SCHEMATIC}/v1.13.2/metal-amd64.iso"

Create both VMs on the server, attaching the ISO at a boot priority that wins over the empty root disk:

incus init --empty k8s-master --vm --profile local --target server \
  -c limits.cpu=2 -c limits.memory=6GiB
incus config device override k8s-master root size=40GiB
incus config device add k8s-master install disk \
  source=/tmp/talos-factory.iso boot.priority=10
incus start k8s-master

incus init --empty k8s-worker --vm --profile local --target server \
  -c limits.cpu=6 -c limits.memory=12GiB
incus config device override k8s-worker root size=150GiB
incus config device add k8s-worker install disk \
  source=/tmp/talos-factory.iso boot.priority=10
incus start k8s-worker

Apply the configs from the server, addressing each node by the bridge IP that incus list reports — this is before WireGuard exists, so the mesh addresses do not answer yet:

talosctl apply-config --insecure --nodes <master-bridge-ip> --file controlplane.yaml
talosctl apply-config --insecure --nodes <worker-bridge-ip> --file worker.yaml

Detach the ISOs once both nodes have installed to disk, or they boot the installer again instead of the system:

incus config device remove k8s-master install
incus config device remove k8s-worker install

Bootstrap etcd on the control plane and pull the kubeconfig. The second command uses the mesh address, which is live by now:

export TALOSCONFIG=/persist/home/user/nixos/talos-config/talosconfig
talosctl bootstrap --nodes <master-bridge-ip> --endpoints <master-bridge-ip>
talosctl kubeconfig --nodes 10.100.0.10 --endpoints 10.100.0.10 ~/.kube/config

Approve the kubelet CSRs. Until the nodes hold serving certificates, kubectl logs, kubectl exec and metrics all fail:

kubectl get csr -o name | xargs kubectl certificate approve

Cilium

Nothing else can proceed before the CNI: without it no pod has networking and both nodes stay NotReady. Talos needs specific values — cgroup automount off because the host path is already mounted, the KubePrism endpoint instead of a Service IP, and explicit capabilities because Talos ships no permissive defaults.

helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium \
  --namespace kube-system \
  --version 1.19.4 \
  --set ipam.mode=kubernetes \
  --set kubeProxyReplacement=true \
  --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
  --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
  --set cgroup.autoMount.enabled=false \
  --set cgroup.hostRoot=/sys/fs/cgroup \
  --set k8sServiceHost=localhost \
  --set k8sServicePort=7445 \
  --set bpf.hostLegacyRouting=true

Wait for the DaemonSet and confirm the nodes flip to Ready:

kubectl -n kube-system rollout status daemonset/cilium
kubectl get nodes

The pod CIDR is an AllowedIP on the k8s-worker peer in modules/features/wireguard.nix. If the server has not been rebuilt since that was added, run rebuild 10.100.0.1.

ArgoCD and Metrics Server

ArgoCD is installed with Helm rather than by itself, since it is the thing that installs everything else. server.insecure=true disables its internal TLS because the host Traefik terminates TLS.

helm repo add argo https://argoproj.github.io/argo-helm
kubectl create namespace argocd
kubectl label namespace argocd pod-security.kubernetes.io/enforce=baseline

helm install argocd argo/argo-cd \
  --namespace argocd \
  --version 9.5.14 \
  --set server.service.type=ClusterIP \
  --set configs.params."server\.insecure"=true

Read the generated admin password and store it in the password manager:

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d

Metrics Server backs the HPA on the CI runners, and needs --kubelet-insecure-tls because the kubelet serving certificates are signed by the cluster CA rather than a publicly rooted one:

helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm install metrics-server metrics-server/metrics-server \
  --namespace kube-system \
  --version 3.13.0 \
  --set args="{--kubelet-insecure-tls}"

Traefik Credential

The host Traefik needs a kubeconfig to read IngressRoute CRDs. Apply the RBAC first:

kubectl apply -f talos-config/manifests/traefik-rbac.yaml

Build a kubeconfig from the ServiceAccount token and the cluster CA:

TOKEN=$(kubectl -n traefik-external get secret traefik-token \
  -o jsonpath='{.data.token}' | base64 -d)
CA=$(kubectl -n traefik-external get secret traefik-token \
  -o jsonpath='{.data.ca\.crt}')

cat <<EOF
apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: ${CA}
    server: https://10.100.0.10:6443
  name: talos
contexts:
- context:
    cluster: talos
    user: traefik
  name: traefik@talos
current-context: traefik@talos
users:
- name: traefik
  user:
    token: ${TOKEN}
EOF

Store that output in SOPS under kubernetes/traefik-kubeconfig, then deploy the server so Traefik picks it up:

sops hosts/server/secrets/secrets.yaml
rebuild 10.100.0.1

No chart in the cluster installs Traefik, so the IngressRoute CRDs come straight from upstream, pinned to the host Traefik’s version:

V=$(ssh 10.100.0.1 'systemctl show traefik -p ExecStart --value' \
  | grep -oE 'traefik-[0-9.]+' | head -1 | cut -d- -f2)
kubectl apply --server-side -f \
  "https://raw.githubusercontent.com/traefik/traefik/v${V}/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml"

Expose ArgoCD itself, which is the last step that uses a raw manifest:

kubectl apply -f talos-config/manifests/argocd-ingress.yaml

ArgoCD is then reachable at https://argocd.yhkze.net.

Out-of-Band Secrets

Two secrets are deliberately outside git, and ArgoCD cannot bring their charts to a healthy state without them. Create both before the root Application syncs.

The Ceph CSI credential, scoped to the kubernetes pool — the full recipe is in the header comment of ceph-csi-rbd/values.yaml in the charts repo:

kubectl create namespace ceph-csi-rbd
kubectl -n ceph-csi-rbd create secret generic csi-rbd-secret \
  --from-literal=userID=kubernetes \
  --from-literal=userKey="$(ssh 10.100.0.1 "sudo ceph auth get-key client.kubernetes")"

The Forgejo Actions runner registration token, taken from Site Admin → Actions → Runners → Create New Runner. The UUID that registration returns goes in the chart’s values.yaml as runner.uuid:

kubectl create namespace ci
kubectl -n ci create secret generic runner-secret --from-literal=token='<TOKEN>'

GitOps Handover

ArgoCD reads both gitops repos as the Forgejo machine user argocd, whose team grants read-only access. A rebuilt cluster cannot reuse the old key (its private half lived in the old cluster), so create a new pair, add the public half to argocd under Site Administration → User Accounts → argocd → SSH Keys (or POST /api/v1/admin/users/argocd/keys), and delete the stale entry:

ssh-keygen -t ed25519 -C argocd-k8s-repo-server -f /tmp/argocd-gitops -N ""

Pin Forgejo’s host key through configs.ssh.extraHosts so it survives Helm upgrades, taking it from the unproxied record on port 30354. The repo server reads the known hosts only at start, so restart it afterwards:

ssh-keyscan -p 30354 -t rsa ssh-git.yhkze.net

helm upgrade argocd argo-cd \
  --repo https://argoproj.github.io/argo-helm \
  --namespace argocd --reuse-values \
  --set "configs.ssh.extraHosts=[ssh-git.yhkze.net]:30354 ssh-rsa <KEY>"
kubectl -n argocd rollout restart deploy/argocd-repo-server

Create one repository credential per repo:

for repo in apps charts; do
  kubectl -n argocd create secret generic "repo-gitops-${repo}" \
    --from-literal=type=git \
    --from-literal=url="ssh://[email protected]:30354/gitops/${repo}.git" \
    --from-file=sshPrivateKey=/tmp/argocd-gitops \
    --dry-run=client -o json | \
  jq ".metadata.labels[\"argocd.argoproj.io/secret-type\"] = \"repository\"" | \
  kubectl apply -f -
done

Apply the root Application once. From here ArgoCD is self-managing: it reconciles gitops/apps recursively, which creates the platform Applications and the development-services ApplicationSet. Reference GitOps and CI/CD.

kubectl apply -n argocd -f \
  <(curl -sL https://git.yhkze.net/gitops/apps/raw/branch/master/root-app.yaml)

The argocd Application syncs manually by design; trigger its first sync once the root Application has created it, which replaces the Helm defaults with the declared configuration (Keycloak SSO, RBAC, known hosts):

kubectl -n argocd patch application argocd --type merge \
  -p '{"operation":{"sync":{"syncOptions":["ServerSideApply=true"]}}}'

OpenBao Access

OpenBao runs on the server, outside the cluster, and validates External Secrets’ ServiceAccount tokens through a reviewer token issued by this cluster. Once the external-secrets Application has created the openbao-token-reviewer Secret, write it and the cluster CA into OpenBao’s Kubernetes auth as in OpenBao. The roles survive a rebuild; only that config has to be rewritten. Until then every SecretStore reports InvalidProviderConfig and logins return 403.