Storage

Layout

One Ceph cluster on the server holds every block device and object store. Incus disks and Kubernetes volumes are RBD images in their own pools; object storage goes through RadosGW, which only the WireGuard mesh and the Incus bridge can reach.

flowchart LR
    subgraph server["server (10.100.0.1)"]
        subgraph ceph["Ceph cluster · MON, MGR, 1 OSD (500 GiB)"]
            pincus[("pool incus<br/>Incus VM disks")]
            pk8s[("pool kubernetes<br/>Kubernetes volumes")]
            prgw[("pools default.rgw.*<br/>buckets")]
        end
        rgw["RadosGW :7480<br/>wg0 and incusbr0 only"]
        incus["Incus"]
        thanos["Thanos"]
        restic["Restic → B2"]
    end
    subgraph k8s["Talos cluster (k8s-master, k8s-worker)"]
        csi["ceph-csi-rbd<br/>StorageClass ceph-rbd (default)"]
        pg["CloudNativePG Cluster<br/>rgxr-pg (test, prod)"]
        barman["Barman Cloud plugin"]
        runner["Forgejo runners"]
    end
    incus --> pincus
    csi -- "kernel rbd over WireGuard" --> pk8s
    pg -- "PVC" --> csi
    pg -- "WAL, base backups" --> barman
    barman -- "S3: rgxr-backups/&lt;namespace&gt;" --> rgw
    runner -- "S3: actions-cache" --> rgw
    thanos -- "S3: thanos" --> rgw
    rgw --> prgw
    restic -. "does not cover Ceph" .-> ceph
Bucket RGW user Used by
actions-cache admin Forgejo runner build cache
thanos thanos Thanos metric blocks, reference Observability
rgxr-backups rgxr-backup CloudNativePG backups of rgxr, prefix per namespace, reference PostgreSQL

Every pool keeps the Ceph default of two replicas, which one OSD cannot place, so the cluster reports HEALTH_WARN with undersized placement groups. Data is written and read normally; it has a single copy.

Ceph RBD CSI Driver

Persistent volumes are RBD images in the kubernetes pool of the server’s single-node Ceph cluster, reached over the WireGuard mesh. Using the cluster that already backs Incus keeps one storage system instead of two, and keeps volume data off the Talos VM disks.

The kernel rbd client is used — rbd, libceph and ceph are compiled into the Talos kernel, so no Talos system extension is needed.

Item Value
Chart ceph-csi-rbd 3.18.1 (https://ceph.github.io/csi-charts)
Namespace ceph-csi-rbd (privileged pod-security labels)
Monitor 10.100.0.1:6789
Cluster fsid fabbcb41-1efa-4385-94b8-8e48e4c155c7
Pool kubernetes
StorageClass ceph-rbd — cluster default; local-path remains for node-local volumes only
Filesystem ext4, Delete reclaim policy, expansion allowed
Image features layering only

Two constraints are worth knowing before changing values. The provisioner runs a single replica because the chart’s default of three leaves pods unschedulable on a two-node cluster, and imageFeatures is restricted to layering because exclusive-lock and object-map need a newer client than the kernel rbd driver offers. The mounter field is left empty, which selects the kernel mounter rather than rbd-nbd.

The nodeplugin DaemonSet tolerates the control-plane taint so a volume can still attach if a workload is ever scheduled on k8s-master.

The namespace’s privileged labels come from managedNamespaceMetadata in the ArgoCD Application rather than a plain CreateNamespace: Talos enforces pod security admission, and without those labels every nodeplugin pod is rejected.

Credentials — the csi-rbd-secret is created out of band and is not in git. The Ceph user is scoped to the one pool (mon 'profile rbd', osd 'profile rbd pool=kubernetes', mgr 'profile rbd pool=kubernetes'). The exact recipe lives in the header comment of ceph-csi-rbd/values.yaml in the charts repo; ArgoCD cannot sync the driver into a working state until the secret exists.

Volumes currently provisioned:

Claim Size
openbao/data-openbao-0 3 Gi
openbao/audit-openbao-0 1 Gi
rgxr-test/rgxr-pg-1, rgxr-prod/rgxr-pg-1 5 Gi each

The default flag lives on two charts, which change together: ceph-csi-rbd/values.yaml marks ceph-rbd default and local-path-provisioner marks local-path not default. Two defaults would make the admission plugin pick by creation time.

Ceph RadosGW (S3)

An S3-compatible endpoint on the same Ceph cluster, for workloads that want object storage rather than a block device.

The daemon needs a keyring before it will start. The entity is client.server, not client.rgw.server, because the NixOS unit runs radosgw --id server:

mkdir -p /var/lib/ceph/radosgw/ceph-server
ceph auth get-or-create client.server \
  osd 'allow rwx' mon 'allow rw' mgr 'allow rw' \
  -o /var/lib/ceph/radosgw/ceph-server/keyring
chown ceph:ceph /var/lib/ceph/radosgw/ceph-server/keyring
systemctl start ceph-rgw-server
curl http://10.100.0.1:7480   # returns XML when healthy

Create the admin user and its S3 key pair, then store both halves in SOPS under ceph-s3/:

radosgw-admin user create --uid=admin --display-name="Admin" \
  --caps="buckets=*;users=*;usage=*;metadata=*"
radosgw-admin key create --uid=admin --key-type=s3 --gen-access-key

Buckets are created on demand with any S3 client pointed at the endpoint — actions-cache backs the CI build cache:

export AWS_ACCESS_KEY_ID=<access-key>
export AWS_SECRET_ACCESS_KEY=<secret-key>
nix run nixpkgs#awscli2 -- --endpoint-url http://10.100.0.1:7480 \
  s3api create-bucket --bucket actions-cache

Backup Coverage

The server’s Restic job (modules/features/restic-backup.nix) backs up the declared filesystem paths to Backblaze B2 daily with GFS retention of 3d/2w/3m/2y. That list covers the NixOS flake, Forgejo state, the host databases and /mnt/atlas, so the cluster’s own definition — Talos configs, patches and the GitOps repos served by Forgejo — is recoverable from B2. Reference Backblaze B2 Backups.

RBD images in the kubernetes pool and the RadosGW buckets are not in that path list, so in-cluster volume data is not part of the B2 chain. Volumes are instead reconstructible from their source: OpenBao state is captured by Raft snapshots, reference OpenBao. CloudNativePG backups go to RadosGW, so they too have no off-site copy, reference PostgreSQL. Metrics, logs and traces are stored on the server, not in the cluster, reference Observability.