Operations and Recovery

Health Checks

Talos reports node-level health over its own API, which is the only way to see problems below the Kubernetes layer:

export TALOSCONFIG=/persist/home/user/nixos/talos-config/talosconfig
talosctl health --nodes 10.100.0.10,10.100.0.11 --endpoints 10.100.0.10

The cluster view:

kubectl get nodes
kubectl get pods -A
kubectl get applications -n argocd

CI runners register with Forgejo rather than with Kubernetes, so check both sides — the pod can be Running while the runner shows offline under Site Admin → Actions → Runners:

kubectl get pods -n ci
kubectl get hpa -n ci

Rebooting a Node

Drain first so pods are rescheduled instead of killed, and remember that a single-worker cluster has nowhere to reschedule to — expect downtime for anything on k8s-worker.

kubectl drain k8s-worker --ignore-daemonsets --delete-emptydir-data
talosctl reboot --nodes 10.100.0.11 --endpoints 10.100.0.10
kubectl uncordon k8s-worker

Updating Talos

Upgrade with the factory installer image so the pinned system extensions are preserved; a plain upstream installer would drop them. The current schematic (613e159…, carrying iscsi-tools and util-linux-tools) is recorded in talos-config/patches/install-image.yaml — bump the version there first, then upgrade each node:

talosctl upgrade \
  --nodes 10.100.0.10 \
  --endpoints 10.100.0.10 \
  --image factory.talos.dev/installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.X.Y \
  --preserve

--preserve keeps etcd data on the control plane. Upgrade the control plane and the worker one at a time, waiting for talosctl health to come back clean in between.

State Inventory

What has to exist to rebuild, and where it lives:

Item Location Contents
Talos secrets talos-config/secrets.yaml Cluster PKI, bootstrap tokens
Talos patches talos-config/patches/ WireGuard keys, install image, CIDRs
Kubeconfig ~/.kube/config Cluster access, regenerable from talosconfig
GitOps repos Forgejo gitops/apps, gitops/charts Every workload definition
S3 credentials SOPS ceph-s3/* RadosGW admin keys
Traefik kubeconfig SOPS kubernetes/traefik-kubeconfig Host ingress access to the API
Ceph CSI credential ceph-csi-rbd/csi-rbd-secret Pool-scoped Ceph user key
OpenBao seal key openbao/openbao-seal-key Auto-unseal key for the Raft store
OpenBao recovery keys Password manager Shamir 3-of-5 recovery shares, root token
ArgoCD admin password Password manager GitOps UI access

talos-config/secrets.yaml and talos-config/patches/wireguard-*.yaml hold private keys and are gitignored. They reach B2 through the server’s Restic job, which backs up the flake directory itself — reference Backblaze B2 Backups.

Recovery

Both K8s VMs lost, server intact — the VMs carry no unique state: etcd is rebuildable, and every workload definition is in gitops/apps and gitops/charts. Delete the VMs with incus delete k8s-master k8s-worker --force and run through Cluster Bootstrap. The RBD images of the old volumes still exist in the kubernetes pool, but the new cluster provisions fresh ones; to reuse an image, hand-write a PersistentVolume with its imageName before letting the consumer’s claim bind. The monitoring volumes are normally not worth recovering, since metrics and logs refill from live scraping.

Server hardware failure — this takes Ceph, Forgejo and Traefik with it, so rebuild the host first: reinstall NixOS from the flake and restore /persist and /mnt/atlas from B2, reference Initial Setup. That brings back Forgejo and with it both GitOps repos. Recreate the Ceph cluster and the kubernetes pool, reference Incus Cluster, then bootstrap the cluster. Volume data in the pool does not survive this.

Everything lost — recovery bottoms out at the last B2 snapshot. Rebuild the server from the flake in git, restore from B2, then rebuild the Ceph cluster and the Kubernetes cluster in that order. Data loss is bounded by the daily Restic schedule and, for in-cluster volumes, is total.