Overview
Incus cluster with Ceph RBD storage for VM migration between hosts.
Server runs full Ceph (MON/MGR/OSD), workstations connect as RBD
clients.
Note: Live migration is not possible at the moment due to a QEMU 10
bug with Q35/ICH9LPC. Cold migration works fine.
Storage pools:
Pool
Driver
Purpose
local
btrfs
Offline-capable, native CPU
shared
Ceph RBD
Migration between hosts, requires server
CPU baseline is controlled by how you target the VM:
Target
CPU
Use case
--target=@migratable
qemu64 (portable)
VMs that migrate between hosts
--target=<hostname>
native
Maximum performance, single host
Planned VMs
VM
Pool
Purpose
WireGuard IP
Windows 10 LTSC
local
Windows workstation (native CPU)
—
k8s-master
local
Talos K8s control plane + etcd
10.100.0.10
k8s-worker
local
Talos K8s worker (all workloads)
10.100.0.11
k8s-master and k8s-worker are Talos Linux
VMs. Reference Cluster
Bootstrap for the bootstrap procedure. WireGuard peers for these VMs
are added to values/network-details.nix after Talos
bootstrap generates their keys.
Prerequisites
WireGuard VPN operational (10.100.0.0/24)
Server at 10.100.0.1
Dedicated disk(s) for Ceph OSD on server
Phase 1: Ceph Setup (Server)
Generate Cluster UUID
Every Ceph cluster needs a unique FSID.
Save this UUID for the next steps.
Update NixOS Configuration
Edit modules/features/ceph.nix, replace the
placeholder:
cephFsid = "your-generated-uuid-here" ;
sudo nixos-rebuild switch --flake 'path:.#server'
Create Keyrings
Ceph uses keyrings for authentication between daemons.
Set your FSID:
FSID = "your-generated-uuid-here"
Create the monitor keyring:
sudo ceph-authtool --create-keyring /tmp/ceph.mon.keyring \
--gen-key -n mon. --cap mon 'allow *'
Create the admin keyring (used for running ceph commands):
sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring \
--gen-key -n client.admin \
--cap mon 'allow *' \
--cap osd 'allow *' \
--cap mds 'allow *' \
--cap mgr 'allow *'
Import admin keyring into monitor keyring:
sudo ceph-authtool /tmp/ceph.mon.keyring \
--import-keyring /etc/ceph/ceph.client.admin.keyring
Create Monmap
The monmap tells Ceph where to find monitors.
sudo monmaptool --create --add server 10.100.0.1 --fsid $FSID /tmp/monmap
Initialize Monitor
The monitor maintains cluster state and coordinates operations.
Create monitor data directory:
sudo mkdir -p /var/lib/ceph/mon/ceph-server
sudo chown -R ceph:ceph /var/lib/ceph
Initialize monitor filesystem:
sudo ceph-mon --mkfs -i server --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
Fix ownership and start:
sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-server
sudo systemctl enable --now ceph-mon-server.service
Single-node Ceph requires adjusted replication settings. (Ceph
docs )
Allow replicas on the same host:
sudo ceph config set global osd_crush_chooseleaf_type 0
Set replica count to 2:
sudo ceph config set global osd_pool_default_size 2
sudo ceph config set global osd_pool_default_min_size 1
Start Manager
The manager provides monitoring and metrics.
Create manager directory:
sudo -u ceph mkdir -p /var/lib/ceph/mgr/ceph-server
Create manager keyring:
sudo ceph auth get-or-create mgr.server \
mon 'allow profile mgr' osd 'allow *' mds 'allow *' \
-o /var/lib/ceph/mgr/ceph-server/keyring
Fix ownership and start:
sudo chown -R ceph:ceph /var/lib/ceph/mgr/ceph-server
sudo systemctl enable --now ceph-mgr-server.service
Create Bootstrap Keyring
The bootstrap-osd keyring authorizes OSD creation.
sudo mkdir -p /var/lib/ceph/bootstrap-osd
sudo ceph auth get-or-create client.bootstrap-osd mon 'allow profile bootstrap-osd' \
-o /var/lib/ceph/bootstrap-osd/ceph.keyring
Create OSDs
OSDs store data on disks. One OSD per disk.
List available disks:
Look for unused disks without partitions.
Prepare the OSD (doesn’t try to enable incompatible systemd
units):
sudo ceph-volume lvm prepare --data /dev/sdX
Keyring search warnings are safe to ignore.
Activate and start via NixOS service:
sudo ceph-volume lvm activate --all
sudo systemctl enable --now ceph-osd-0.service
Repeat for additional disks (incrementing OSD number).
Harden Cluster
Disable insecure global_id reclaim (prevents CVE-2021-20288 auth
bypass):
sudo ceph config set mon auth_allow_insecure_global_id_reclaim false
Enable msgr2 protocol (encrypted, more efficient than legacy
msgr1):
sudo ceph mon enable-msgr2
Create Incus Pool
Create RBD pool with 32 placement groups:
sudo ceph osd pool create incus 32
Mark pool for RBD usage:
sudo ceph osd pool application enable incus rbd
Verify Cluster
Expected output
cluster:
id: your-uuid-here
health: HEALTH_WARN
OSD count 1 < osd_pool_default_size 2
services:
mon: 1 daemons, quorum server
mgr: server(active)
osd: 1 osds: 1 up, 1 in
data:
pools: 1 pools, 32 pgs
objects: 0 objects, 0 B
usage: ~30 MiB used, ~120 GiB avail
pgs: 32 active+clean
HEALTH_OK or HEALTH_WARN with only “OSD
count” warning is fine for single-node
All services should show daemons running
Pool should exist with active+clean PGs
Phase 2: Ceph Client
Setup (Desktop/Laptop)
Workstations only need keyring to connect. The ceph.conf
is managed by NixOS.
Copy the admin keyring to the persist location
sudo rsync -avz root@< server-ip> :/etc/ceph/ceph.client.admin.keyring /persist/etc/ceph/
sudo chmod 640 /persist/etc/ceph/ceph.client.admin.keyring
Rebuild the system to activate the bind mount
Desktop
sudo nixos-rebuild switch --flake 'path:.#desktop'
Laptop
sudo nixos-rebuild switch --flake 'path:.#laptop'
Phase 3: Incus Cluster Setup
Clean State (If Needed)
If Incus was previously deployed, reset it first.
sudo systemctl stop incus
sudo rm -rf /var/lib/incus
sudo systemctl start incus
Bootstrap Cluster (Server)
Prompts:
Clustering: yes
IP address: 10.100.0.1
Joining existing: no
Member name: server
Local storage pool: no
Remote storage pool: no
MAAS server: no
Network bridge: yes
Bridge name: incusbr0
IPv4 subnet: 10.200.3.1/24 (identical on every member,
reference Networking )
IPv4 NAT: yes
IPv6: auto
Auto-update images: yes
Create Storage Pools
NixOS preseed cannot create storage pools in clustered mode.
Each storage pool needs a source path pointing to where
data lives on disk. In a cluster, this path can differ between members
(e.g., different disks on each host). Incus calls this a
“member-specific” configuration. Since preseed applies cluster-wide, it
cannot set values that vary per member.
Create local btrfs pool:
incus storage create local btrfs source=/var/lib/incus/storage-pools/local
Create shared Ceph pool:
incus storage create shared ceph source=incus ceph.osd.pool_name=incus
Create Cluster Group
The migratable group sets a portable CPU baseline
(qemu64) for VMs that need to move between hosts. (Cluster
groups docs )
incus cluster group create migratable
incus cluster group set migratable instances.vm.cpu.x86_64.baseline=qemu64
Add all hosts to the migratable group:
incus cluster group add server migratable
Create Profiles
Profile for migratable VMs (shared storage):
incus profile create migratable
incus profile device add migratable root disk pool=shared path=/
incus profile device add migratable eth0 nic network=incusbr0 name=eth0
Profile for local VMs (fast storage, native CPU):
incus profile create local
incus profile device add local root disk pool=local path=/
incus profile device add local eth0 nic network=incusbr0 name=eth0
Generate Join Tokens
Tokens are single-use and expire in 3 hours:
incus cluster add desktop
incus cluster add laptop
Copy the output tokens.
Join Cluster (Desktop/Laptop)
Prompts:
Clustering: yes
IP address: 10.100.0.X
Joining existing: yes
Join token: <paste token>
Continue: yes
local pool source:
/var/lib/incus/storage-pools/local
shared pool source: (press enter to inherit)
Assign Groups (Run on Server)
incus cluster group add desktop migratable
incus cluster group add laptop migratable
Verification
incus cluster list
incus storage list
incus cluster group list
Usage
Migratable VM (Shared
Storage)
Create a VM that can move between hosts. Use
--target=@migratable to apply the portable CPU
baseline:
incus launch images:nixos/unstable devvm \
--vm \
--profile migratable \
--target = @migratable \
-c security.secureboot=false
Cold migrate to another host as follows.
Live migration is not possible at the moment due to a QEMU bug .
incus stop devvm
incus move devvm --target laptop
incus start devvm
The disk lives on Ceph, so both hosts already have access - no data
copy needed.
Local VM (Native CPU)
Create a VM on local storage for maximum performance. Target a
specific host to use native CPU:
incus launch images:nixos/unstable workvm \
--vm \
--profile local \
--target desktop \
-c security.secureboot=false
This VM runs with full native CPU capabilities. It cannot migrate
(local storage, native CPU) but runs faster and works offline.
Summary
Profile
Target
CPU
Storage
Migration
migratable
@migratable
qemu64
Ceph
Yes (cold)
local
<hostname>
native
btrfs
No
Management
Cluster Operations
Evacuate a member before maintenance (migrates its VMs away):
incus cluster evacuate desktop
incus cluster restore desktop
Remove offline member permanently:
incus cluster remove --force desktop
VM Operations
Start, stop, and restart:
incus start < vm>
incus stop < vm>
incus restart < vm>
Access the VGA console (works from any cluster member — Incus routes
to whichever host runs the VM):
incus console < vm> --type=vga
SSH into a VM (if SSH is configured inside it):
Show VM config and devices:
incus config show < vm>
incus config device show < vm>
List all VMs across the cluster:
Snapshots
Create a snapshot:
incus snapshot create < vm> < snapshot-name>
List snapshots:
Rename a snapshot:
incus snapshot rename < vm> < old-name> < new-name>
Restore:
incus snapshot restore < vm> < snapshot-name>
Delete:
incus snapshot delete < vm> < snapshot-name>
Storage
List storage volumes:
incus storage volume list shared
incus storage volume list local
Import an ISO as a storage volume (e.g. for VM installation
media):
incus storage volume import < pool> < file.iso> < volume-name> --type=iso
Delete a volume:
incus storage volume delete < pool> < volume-name>
Known Issues
Live migration fails with ICH9LPC error — QEMU 10
has a bug where live migration fails at restore with “Missing section
footer for ICH9LPC”. Cold migration (stop, move, start) works. Track
upstream: QEMU
#2046
Troubleshooting
Ceph Storage Operations Fail
If incus storage volume import shared or other Ceph
operations fail with signal: killed or connection errors,
the Ceph OSD may be down.
Check Ceph status
If you see 0 up in the OSD line, the OSD needs
reactivation. This happens after server reboots because ceph-volume uses
tmpfs mounts that don’t persist.
Reactivate OSD
sudo ceph-volume lvm activate --all
sudo systemctl start ceph-osd-0.service
Verify OSD is up
Should now show 1 up in the OSD line.
References