One Ceph cluster on the server holds every block device and object store. Incus disks and Kubernetes volumes are RBD images in their own pools; object storage goes through RadosGW, which only the WireGuard mesh and the Incus bridge can reach.
flowchart LR
subgraph server["server (10.100.0.1)"]
subgraph ceph["Ceph cluster · MON, MGR, 1 OSD (500 GiB)"]
pincus[("pool incus<br/>Incus VM disks")]
pk8s[("pool kubernetes<br/>Kubernetes volumes")]
prgw[("pools default.rgw.*<br/>buckets")]
end
rgw["RadosGW :7480<br/>wg0 and incusbr0 only"]
incus["Incus"]
thanos["Thanos"]
restic["Restic → B2"]
end
subgraph k8s["Talos cluster (k8s-master, k8s-worker)"]
csi["ceph-csi-rbd<br/>StorageClass ceph-rbd (default)"]
pg["CloudNativePG Cluster<br/>rgxr-pg (test, prod)"]
barman["Barman Cloud plugin"]
runner["Forgejo runners"]
end
incus --> pincus
csi -- "kernel rbd over WireGuard" --> pk8s
pg -- "PVC" --> csi
pg -- "WAL, base backups" --> barman
barman -- "S3: rgxr-backups/<namespace>" --> rgw
runner -- "S3: actions-cache" --> rgw
thanos -- "S3: thanos" --> rgw
rgw --> prgw
restic -. "does not cover Ceph" .-> ceph
| Bucket | RGW user | Used by |
|---|---|---|
actions-cache |
admin |
Forgejo runner build cache |
thanos |
thanos |
Thanos metric blocks, reference Observability |
rgxr-backups |
rgxr-backup |
CloudNativePG backups of rgxr, prefix per namespace, reference PostgreSQL |
Every pool keeps the Ceph default of two replicas, which one OSD
cannot place, so the cluster reports HEALTH_WARN with
undersized placement groups. Data is written and read normally; it has a
single copy.
Persistent volumes are RBD images in the kubernetes pool
of the server’s single-node Ceph cluster, reached over the WireGuard
mesh. Using the cluster that already backs Incus keeps one storage
system instead of two, and keeps volume data off the Talos VM disks.
The kernel rbd client is used — rbd,
libceph and ceph are compiled into the Talos
kernel, so no Talos system extension is needed.
| Item | Value |
|---|---|
| Chart | ceph-csi-rbd 3.18.1 (https://ceph.github.io/csi-charts) |
| Namespace | ceph-csi-rbd (privileged pod-security labels) |
| Monitor | 10.100.0.1:6789 |
| Cluster fsid | fabbcb41-1efa-4385-94b8-8e48e4c155c7 |
| Pool | kubernetes |
| StorageClass | ceph-rbd — cluster default; local-path
remains for node-local volumes only |
| Filesystem | ext4, Delete reclaim policy, expansion allowed |
| Image features | layering only |
Two constraints are worth knowing before changing values. The
provisioner runs a single replica because the chart’s default of three
leaves pods unschedulable on a two-node cluster, and
imageFeatures is restricted to layering
because exclusive-lock and object-map need a
newer client than the kernel rbd driver offers. The mounter
field is left empty, which selects the kernel mounter rather than
rbd-nbd.
The nodeplugin DaemonSet tolerates the control-plane taint so a
volume can still attach if a workload is ever scheduled on
k8s-master.
The namespace’s privileged labels come from
managedNamespaceMetadata in the ArgoCD Application rather
than a plain CreateNamespace: Talos enforces pod security
admission, and without those labels every nodeplugin pod is
rejected.
Credentials — the csi-rbd-secret is
created out of band and is not in git. The Ceph user is scoped to the
one pool (mon 'profile rbd',
osd 'profile rbd pool=kubernetes',
mgr 'profile rbd pool=kubernetes'). The exact recipe lives
in the header comment of ceph-csi-rbd/values.yaml in the
charts repo; ArgoCD cannot sync the driver into a working state until
the secret exists.
Volumes currently provisioned:
| Claim | Size |
|---|---|
openbao/data-openbao-0 |
3 Gi |
openbao/audit-openbao-0 |
1 Gi |
rgxr-test/rgxr-pg-1,
rgxr-prod/rgxr-pg-1 |
5 Gi each |
The default flag lives on two charts, which change together:
ceph-csi-rbd/values.yaml marks ceph-rbd
default and local-path-provisioner marks
local-path not default. Two defaults would make the
admission plugin pick by creation time.
An S3-compatible endpoint on the same Ceph cluster, for workloads that want object storage rather than a block device.
modules/features/ceph-radosgw.nix (server
only)wg0 and incusbr0
interfaces only, never from the internetceph-s3/access-key and
ceph-s3/secret-keyThe daemon needs a keyring before it will start. The entity is
client.server, not client.rgw.server, because
the NixOS unit runs radosgw --id server:
mkdir -p /var/lib/ceph/radosgw/ceph-server
ceph auth get-or-create client.server \
osd 'allow rwx' mon 'allow rw' mgr 'allow rw' \
-o /var/lib/ceph/radosgw/ceph-server/keyring
chown ceph:ceph /var/lib/ceph/radosgw/ceph-server/keyring
systemctl start ceph-rgw-server
curl http://10.100.0.1:7480 # returns XML when healthyCreate the admin user and its S3 key pair, then store both halves in
SOPS under ceph-s3/:
radosgw-admin user create --uid=admin --display-name="Admin" \
--caps="buckets=*;users=*;usage=*;metadata=*"
radosgw-admin key create --uid=admin --key-type=s3 --gen-access-keyBuckets are created on demand with any S3 client pointed at the
endpoint — actions-cache backs the CI build cache:
export AWS_ACCESS_KEY_ID=<access-key>
export AWS_SECRET_ACCESS_KEY=<secret-key>
nix run nixpkgs#awscli2 -- --endpoint-url http://10.100.0.1:7480 \
s3api create-bucket --bucket actions-cacheThe server’s Restic job
(modules/features/restic-backup.nix) backs up the declared
filesystem paths to Backblaze B2 daily with GFS retention of
3d/2w/3m/2y. That list covers the NixOS flake, Forgejo
state, the host databases and /mnt/atlas, so the cluster’s
own definition — Talos configs, patches and the GitOps repos served by
Forgejo — is recoverable from B2. Reference Backblaze B2 Backups.
RBD images in the kubernetes pool and the RadosGW
buckets are not in that path list, so in-cluster volume data is not part
of the B2 chain. Volumes are instead reconstructible from their source:
OpenBao state is captured by Raft snapshots, reference OpenBao. CloudNativePG backups go to RadosGW, so
they too have no off-site copy, reference PostgreSQL. Metrics, logs and traces are
stored on the server, not in the cluster, reference Observability.