Server Reinstall

Reinstalls the server from scratch, for example onto a new disk, and brings it back as the same host with all of its data. The backup is restored into the freshly partitioned disk before the first install, so the host keys, the sops age key and the UID/GID map are in place when the system first activates.

flowchart LR
    A[freeze services] --> B[final backup]
    B --> C[power off, swap disk]
    C --> D[live ISO]
    D --> E[disko]
    E --> F[restore into /mnt]
    F --> G[bootstrap install]
    G --> H[unlock over SSH]
    H --> I[rebuild push]
    I --> J[post-install]

The restore source is either the B2 repository or a local replica on a workstation reached over the LAN. A local replica restores about ten times faster (about 12 minutes for 120 GiB at 1 Gb/s) and costs nothing, while a B2 restore is billed per GiB downloaded past the daily free allowance. Keep the old disk untouched until Post-Install passes: it is the fallback.

Prepare

Freeze Services

Stop everything that writes state, so the final backup holds the last write of every service. Incoming mail waits in the senders’ queues, which retry for days

systemctl stop restic-backups-b2.timer
systemctl stop postfix dovecot forgejo keycloak opencloud trilium \
  anki-sync-server n8n navidrome miniflux memos slskd open-webui \
  openbao syncthing
systemctl stop 'podman-*'
systemctl stop postgresql

Pause the server device in Syncthing on every peer as well: until the restore is verified, the server must not push an empty or partial folder to them. The backup carries no Syncthing index, so on resume the server pulls anything the restore lacks from the peers rather than reporting it deleted.

Final Backup

B2

The backup unit rotates the read-only @restic-persist snapshot first, so every path comes from the same instant

systemctl start restic-backups-b2.service
journalctl -u restic-backups-b2.service -n 5 -o cat

Record the snapshot ID it prints.

Local Replica

The replica is a second restic repository on a workstation’s disk, written over SFTP with the same password and the same include and exclude lists as the B2 job. Check that both ends negotiated gigabit first: a NIC that reports downshifted in the kernel log has a bad wire pair and falls back to 100 Mb/s, which makes the transfer take ten times as long

cat /sys/class/net/<interface>/speed  # must print 1000
journalctl -k | grep -i downshift

Create the repository, then rotate the snapshot and back it up. The include and exclude files are the store paths the B2 unit already uses

REPO=sftp:user@<workstation_LAN_IP>:/mnt/aether/restic-server
SFTP="sftp.command=ssh -i /persist/home/user/.ssh/id_ed25519 -o BatchMode=yes user@<workstation_LAN_IP> -s sftp"
export RESTIC_PASSWORD_FILE=/run/secrets/restic/password
restic -r "$REPO" -o "$SFTP" init

systemctl start restic-persist-snapshot.service
systemctl show restic-backups-b2 -p ExecStart --value \
  | grep -oE '/nix/store/[^ ;]*(staticPaths|exclude-patterns)'
systemd-run --unit restic-replica -p Type=oneshot \
  -p BindReadOnlyPaths=/run/restic-persist:/persist \
  --setenv=RESTIC_PASSWORD_FILE=/run/secrets/restic/password \
  --setenv=RESTIC_CACHE_DIR=/var/cache/restic-replica \
  restic -r "$REPO" -o "$SFTP" backup \
  --files-from=<staticPaths> --exclude-file=<exclude-patterns> --no-scan
journalctl -fu restic-replica

Record the snapshot ID it prints.

Verify the Backup

Check the repository and spot-check the paths a reinstall cannot do without. restic ls does not descend into subdirectories, so pass --recursive when looking for nested files

restic check --read-data-subset=5%
for p in /persist/var/lib/nixos/uid-map /persist/etc/secrets/initrd \
  /persist/etc/ssh /persist/var/keys /persist/var/lib/postgresql \
  /persist/home/user/.config/syncthing; do
  restic ls <snapshot> "$p" | tail -n +2 | head -3
done
poweroff

Install

Live ISO

Boot the minimal NixOS image. The graphical image suspends itself when idle and freezes the session during long steps such as the restore

sudo -i
loadkeys es
passwd
dd if=/dev/urandom bs=1 count=32 | base64 > /tmp/pass

Store the passphrase in a password manager: it unlocks the disk on every boot.

ssh-copy-id root@<target_machine_IP>

Partition

The disk is addressed by model and serial, so read the path and set it as device in hosts/server/nixos/disko.nix, then commit it

ls -l /dev/disk/by-id/ | grep nvme-

nixos-anywhere skips kexec because the target already runs a NixOS installer, which keeps /tmp/pass in place

nix run nixpkgs#nixos-anywhere \
  --extra-experimental-features 'nix-command flakes' \
  -- --flake 'path:.#server' \
  --phases 'kexec,disko' \
  --generate-hardware-config nixos-facter hosts/server/facter.json \
  'root@<target_machine_IP>'

Disko leaves the tree mounted under /mnt. Confirm the layout

lsblk /dev/nvme0n1
lvs
findmnt -R /mnt

Restore

Paths are stored absolute, so /persist/... lands on /mnt/persist and /mnt/atlas on /mnt/mnt/atlas, where disko mounted each subvolume. The cache goes to the new disk because the installer’s tmpfs cannot hold it. Run the restore as a transient unit so a dropped SSH session does not kill it.

B2

The B2 key must be allowed to read files: a key created with write access only answers downloads with 403

export RESTIC_REPOSITORY=b2:yhkze-infra-backups:nixos
export B2_ACCOUNT_ID=<id> B2_ACCOUNT_KEY=<key>
read -s RESTIC_PASSWORD && export RESTIC_PASSWORD
nix run nixpkgs#restic -- snapshots
nix run nixpkgs#restic -- restore <snapshot> --target /mnt \
  --cache-dir /mnt/persist/var/cache/restic

Local Replica

The installer reads the replica over SFTP with a throwaway key. On the workstation the key is forced into a read-only sftp-server -R and limited to the target’s address, so it can neither open a shell nor modify the repository; --no-lock lets restic read without writing a lock file

ssh-keygen -t ed25519 -N "" -C restore-throwaway -f /root/.ssh/restore_key
cat /root/.ssh/restore_key.pub

Allow the key on the workstation. ~/.ssh/authorized_keys is otherwise unused, since declared keys live in /etc/ssh/authorized_keys.d

SFTP_SERVER=$(sudo sshd -T | awk '/^subsystem sftp/ {print $3}')
echo "restrict,from=\"<target_machine_IP>\",command=\"$SFTP_SERVER -R\" <pubkey>" \
  > ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
read -s P && printf %s "$P" > /root/.restic-pass && chmod 600 /root/.restic-pass
RESTIC=$(nix build nixpkgs#restic --print-out-paths --no-link)/bin/restic
systemd-run --no-block --unit restic-restore -p Type=oneshot \
  --setenv=RESTIC_PASSWORD_FILE=/root/.restic-pass --setenv=HOME=/root \
  $RESTIC --no-lock \
  -r sftp:user@<workstation_LAN_IP>:/mnt/aether/restic-server \
  -o "sftp.command=ssh -i /root/.ssh/restore_key -o StrictHostKeyChecking=accept-new user@<workstation_LAN_IP> -s sftp" \
  restore <snapshot> --target /mnt --cache-dir /mnt/persist/var/cache/restic
journalctl -fu restic-restore

Remove the key once the restore finishes

rm ~/.ssh/authorized_keys

An interrupted restore can be started again with the same command: files that already match the snapshot are skipped.

Verify the Restore

The summary must add up to the snapshot’s size (restored plus skipped). Then confirm the keys exist and that state directories are owned by the UIDs the restored map assigns; the setup guide’s ssh-keygen and sops key steps are skipped because both keys came back

cd /mnt/persist
ls -l etc/secrets/initrd etc/ssh var/keys var/lib/nixos
jq -c '{openbao,forgejo,opencloud,postgres}' var/lib/nixos/uid-map
stat -c '%n %u:%g' var/lib/openbao var/lib/forgejo var/lib/opencloud var/lib/postgresql

Bootstrap Install

BOOTSTRAP=1 installs the minimal feature list, reference values/bootstrap.nix. The first activation then reads the restored UID/GID map, so every service user gets back the UID that owns its files

BOOTSTRAP=1 nix run nixpkgs#nixos-anywhere \
  --extra-experimental-features 'nix-command flakes' \
  -- --flake 'path:.#server' \
  --option pure-eval false \
  --phases 'install,reboot' \
  'root@<target_machine_IP>'

Unlock and Converge

The restored initrd host key is unchanged, so known_hosts still matches. The passphrase is sent without a trailing newline

printf %s '<pass>' | ssh -p 2224 root@<target_machine_IP> systemd-tty-ask-password-agent
rebuild push <target_machine_IP>

Post-Install

Health first: systemctl --failed must be empty, and Keycloak login, Forgejo, mail send and receive, OpenCloud and Trilium must answer. Then rebuild what the backup deliberately leaves out.

open-terminal — its Nix store lives in a podman volume. The backup holds it as an exported tar, because the global excludes (**/build, **/cache and others) would corrupt a file-level copy

podman volume import open-terminal-nix /persist/var/lib/open-terminal/nix-volume.tar
rm /persist/var/lib/open-terminal/nix-volume.tar

Ceph — a fresh cluster with the same fsid, its OSD on the raw pool/ceph logical volume (ceph-volume lvm create --data pool/ceph). Reference incus-cluster.md Phase 1, then copy the admin keyring to the workstations again.

Incus — remove the stale member from another cluster member, then rejoin the server with a fresh token as in incus-cluster.md Phase 3

incus cluster remove server --force

Kubernetes — recreate the Talos VMs from talos-config/. Reference bootstrap.md.

Finish with a first backup from the new host (systemctl start restic-backups-b2.service), resume the server device in Syncthing on every peer, and only then reuse the old disk. Before wiping it, mount it read-only and list what it holds that the new host lacks (rsync -n -a --ignore-existing <old>/ <new>/): anything besides deliberate deletions is outside the backup paths or matched an exclude pattern.