Talos reports node-level health over its own API, which is the only way to see problems below the Kubernetes layer:
export TALOSCONFIG=/persist/home/user/nixos/talos-config/talosconfig
talosctl health --nodes 10.100.0.10,10.100.0.11 --endpoints 10.100.0.10The cluster view:
kubectl get nodes
kubectl get pods -A
kubectl get applications -n argocdCI runners register with Forgejo rather than with Kubernetes, so
check both sides — the pod can be Running while the runner
shows offline under Site Admin → Actions → Runners:
kubectl get pods -n ci
kubectl get hpa -n ciDrain first so pods are rescheduled instead of killed, and remember
that a single-worker cluster has nowhere to reschedule to — expect
downtime for anything on k8s-worker.
kubectl drain k8s-worker --ignore-daemonsets --delete-emptydir-data
talosctl reboot --nodes 10.100.0.11 --endpoints 10.100.0.10
kubectl uncordon k8s-workerUpgrade with the factory installer image so the pinned system
extensions are preserved; a plain upstream installer would drop them.
The current schematic (613e159…, carrying
iscsi-tools and util-linux-tools) is recorded
in talos-config/patches/install-image.yaml — bump the
version there first, then upgrade each node:
talosctl upgrade \
--nodes 10.100.0.10 \
--endpoints 10.100.0.10 \
--image factory.talos.dev/installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.X.Y \
--preserve--preserve keeps etcd data on the control plane. Upgrade
the control plane and the worker one at a time, waiting for
talosctl health to come back clean in between.
What has to exist to rebuild, and where it lives:
| Item | Location | Contents |
|---|---|---|
| Talos secrets | talos-config/secrets.yaml |
Cluster PKI, bootstrap tokens |
| Talos patches | talos-config/patches/ |
WireGuard keys, install image, CIDRs |
| Kubeconfig | ~/.kube/config |
Cluster access, regenerable from talosconfig |
| GitOps repos | Forgejo gitops/apps, gitops/charts |
Every workload definition |
| S3 credentials | SOPS ceph-s3/* |
RadosGW admin keys |
| Traefik kubeconfig | SOPS kubernetes/traefik-kubeconfig |
Host ingress access to the API |
| Ceph CSI credential | ceph-csi-rbd/csi-rbd-secret |
Pool-scoped Ceph user key |
| OpenBao seal key | openbao/openbao-seal-key |
Auto-unseal key for the Raft store |
| OpenBao recovery keys | Password manager | Shamir 3-of-5 recovery shares, root token |
| ArgoCD admin password | Password manager | GitOps UI access |
talos-config/secrets.yaml and
talos-config/patches/wireguard-*.yaml hold private keys and
are gitignored. They reach B2 through the server’s Restic job, which
backs up the flake directory itself — reference Backblaze B2 Backups.
Both K8s VMs lost, server intact — the VMs carry no
unique state: etcd is rebuildable, and every workload definition is in
gitops/apps and gitops/charts. Delete the VMs
with incus delete k8s-master k8s-worker --force and run
through Cluster Bootstrap. The RBD images
of the old volumes still exist in the kubernetes pool, but
the new cluster provisions fresh ones; to reuse an image, hand-write a
PersistentVolume with its imageName before
letting the consumer’s claim bind. The monitoring volumes are normally
not worth recovering, since metrics and logs refill from live
scraping.
Server hardware failure — this takes Ceph, Forgejo
and Traefik with it, so rebuild the host first: reinstall NixOS from the
flake and restore /persist and /mnt/atlas from
B2, reference Initial Setup. That brings
back Forgejo and with it both GitOps repos. Recreate the Ceph cluster
and the kubernetes pool, reference Incus Cluster, then bootstrap
the cluster. Volume data in the pool does not survive this.
Everything lost — recovery bottoms out at the last B2 snapshot. Rebuild the server from the flake in git, restore from B2, then rebuild the Ceph cluster and the Kubernetes cluster in that order. Data loss is bounded by the daily Restic schedule and, for in-cluster volumes, is total.