Incus Cluster with Ceph Storage

Overview

Incus cluster with Ceph RBD storage for VM migration between hosts. Server runs full Ceph (MON/MGR/OSD), workstations connect as RBD clients.

Note: Live migration is not possible at the moment due to a QEMU 10 bug with Q35/ICH9LPC. Cold migration works fine.

Storage pools:

Pool Driver Purpose
local btrfs Offline-capable, native CPU
shared Ceph RBD Migration between hosts, requires server

CPU baseline is controlled by how you target the VM:

Target CPU Use case
--target=@migratable qemu64 (portable) VMs that migrate between hosts
--target=<hostname> native Maximum performance, single host

Planned VMs

VM Pool Purpose WireGuard IP
Windows 10 LTSC local Windows workstation (native CPU) —
k8s-master local Talos K8s control plane + etcd 10.100.0.10
k8s-worker local Talos K8s worker (all workloads) 10.100.0.11

k8s-master and k8s-worker are Talos Linux VMs. Reference Cluster Bootstrap for the bootstrap procedure. WireGuard peers for these VMs are added to values/network-details.nix after Talos bootstrap generates their keys.

Prerequisites

Phase 1: Ceph Setup (Server)

Generate Cluster UUID

Every Ceph cluster needs a unique FSID.

uuidgen

Save this UUID for the next steps.

Update NixOS Configuration

Edit modules/features/ceph.nix, replace the placeholder:

cephFsid = "your-generated-uuid-here";
sudo nixos-rebuild switch --flake 'path:.#server'

Create Keyrings

Ceph uses keyrings for authentication between daemons.

Set your FSID:

FSID="your-generated-uuid-here"

Create the monitor keyring:

sudo ceph-authtool --create-keyring /tmp/ceph.mon.keyring \
  --gen-key -n mon. --cap mon 'allow *'

Create the admin keyring (used for running ceph commands):

sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring \
  --gen-key -n client.admin \
  --cap mon 'allow *' \
  --cap osd 'allow *' \
  --cap mds 'allow *' \
  --cap mgr 'allow *'

Import admin keyring into monitor keyring:

sudo ceph-authtool /tmp/ceph.mon.keyring \
  --import-keyring /etc/ceph/ceph.client.admin.keyring

Create Monmap

The monmap tells Ceph where to find monitors.

sudo monmaptool --create --add server 10.100.0.1 --fsid $FSID /tmp/monmap

Initialize Monitor

The monitor maintains cluster state and coordinates operations.

Create monitor data directory:

sudo mkdir -p /var/lib/ceph/mon/ceph-server
sudo chown -R ceph:ceph /var/lib/ceph

Initialize monitor filesystem:

sudo ceph-mon --mkfs -i server --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring

Fix ownership and start:

sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-server
sudo systemctl enable --now ceph-mon-server.service

Configure Single-Node Mode

Single-node Ceph requires adjusted replication settings. (Ceph docs)

Allow replicas on the same host:

sudo ceph config set global osd_crush_chooseleaf_type 0

Set replica count to 2:

sudo ceph config set global osd_pool_default_size 2
sudo ceph config set global osd_pool_default_min_size 1

Start Manager

The manager provides monitoring and metrics.

Create manager directory:

sudo -u ceph mkdir -p /var/lib/ceph/mgr/ceph-server

Create manager keyring:

sudo ceph auth get-or-create mgr.server \
  mon 'allow profile mgr' osd 'allow *' mds 'allow *' \
  -o /var/lib/ceph/mgr/ceph-server/keyring

Fix ownership and start:

sudo chown -R ceph:ceph /var/lib/ceph/mgr/ceph-server
sudo systemctl enable --now ceph-mgr-server.service

Create Bootstrap Keyring

The bootstrap-osd keyring authorizes OSD creation.

sudo mkdir -p /var/lib/ceph/bootstrap-osd
sudo ceph auth get-or-create client.bootstrap-osd mon 'allow profile bootstrap-osd' \
  -o /var/lib/ceph/bootstrap-osd/ceph.keyring

Create OSDs

OSDs store data on disks. One OSD per disk.

List available disks:

lsblk

Look for unused disks without partitions.

Prepare the OSD (doesn’t try to enable incompatible systemd units):

sudo ceph-volume lvm prepare --data /dev/sdX

Keyring search warnings are safe to ignore.

Activate and start via NixOS service:

sudo ceph-volume lvm activate --all
sudo systemctl enable --now ceph-osd-0.service

Repeat for additional disks (incrementing OSD number).

Verify OSDs:

sudo ceph osd tree

Harden Cluster

Disable insecure global_id reclaim (prevents CVE-2021-20288 auth bypass):

sudo ceph config set mon auth_allow_insecure_global_id_reclaim false

Enable msgr2 protocol (encrypted, more efficient than legacy msgr1):

sudo ceph mon enable-msgr2

Create Incus Pool

Create RBD pool with 32 placement groups:

sudo ceph osd pool create incus 32

Mark pool for RBD usage:

sudo ceph osd pool application enable incus rbd

Verify Cluster

sudo ceph -s
Expected output
  cluster:
    id:     your-uuid-here
    health: HEALTH_WARN
            OSD count 1 < osd_pool_default_size 2

  services:
    mon: 1 daemons, quorum server
    mgr: server(active)
    osd: 1 osds: 1 up, 1 in

  data:
    pools:   1 pools, 32 pgs
    objects: 0 objects, 0 B
    usage:   ~30 MiB used, ~120 GiB avail
    pgs:     32 active+clean
  • HEALTH_OK or HEALTH_WARN with only “OSD count” warning is fine for single-node
  • All services should show daemons running
  • Pool should exist with active+clean PGs

Phase 2: Ceph Client Setup (Desktop/Laptop)

Workstations only need keyring to connect. The ceph.conf is managed by NixOS.

Copy the admin keyring to the persist location

sudo rsync -avz root@<server-ip>:/etc/ceph/ceph.client.admin.keyring /persist/etc/ceph/
sudo chmod 640 /persist/etc/ceph/ceph.client.admin.keyring

Rebuild the system to activate the bind mount

Desktop

sudo nixos-rebuild switch --flake 'path:.#desktop'

Laptop

sudo nixos-rebuild switch --flake 'path:.#laptop'

Verify connectivity:

ceph -s

Phase 3: Incus Cluster Setup

Clean State (If Needed)

If Incus was previously deployed, reset it first.

sudo systemctl stop incus
sudo rm -rf /var/lib/incus
sudo systemctl start incus

Bootstrap Cluster (Server)

sudo incus admin init

Prompts:

Create Storage Pools

NixOS preseed cannot create storage pools in clustered mode.

Each storage pool needs a source path pointing to where data lives on disk. In a cluster, this path can differ between members (e.g., different disks on each host). Incus calls this a “member-specific” configuration. Since preseed applies cluster-wide, it cannot set values that vary per member.

Create local btrfs pool:

incus storage create local btrfs source=/var/lib/incus/storage-pools/local

Create shared Ceph pool:

incus storage create shared ceph source=incus ceph.osd.pool_name=incus

Create Cluster Group

The migratable group sets a portable CPU baseline (qemu64) for VMs that need to move between hosts. (Cluster groups docs)

incus cluster group create migratable
incus cluster group set migratable instances.vm.cpu.x86_64.baseline=qemu64

Add all hosts to the migratable group:

incus cluster group add server migratable

Create Profiles

Profile for migratable VMs (shared storage):

incus profile create migratable
incus profile device add migratable root disk pool=shared path=/
incus profile device add migratable eth0 nic network=incusbr0 name=eth0

Profile for local VMs (fast storage, native CPU):

incus profile create local
incus profile device add local root disk pool=local path=/
incus profile device add local eth0 nic network=incusbr0 name=eth0

Generate Join Tokens

Tokens are single-use and expire in 3 hours:

incus cluster add desktop
incus cluster add laptop

Copy the output tokens.

Join Cluster (Desktop/Laptop)

sudo incus admin init

Prompts:

Assign Groups (Run on Server)

incus cluster group add desktop migratable
incus cluster group add laptop migratable

Verification

incus cluster list
incus storage list
incus cluster group list

Usage

Migratable VM (Shared Storage)

Create a VM that can move between hosts. Use --target=@migratable to apply the portable CPU baseline:

incus launch images:nixos/unstable devvm \
  --vm \
  --profile migratable \
  --target=@migratable \
  -c security.secureboot=false

Cold migrate to another host as follows.

Live migration is not possible at the moment due to a QEMU bug.

incus stop devvm
incus move devvm --target laptop
incus start devvm

The disk lives on Ceph, so both hosts already have access - no data copy needed.

Local VM (Native CPU)

Create a VM on local storage for maximum performance. Target a specific host to use native CPU:

incus launch images:nixos/unstable workvm \
  --vm \
  --profile local \
  --target desktop \
  -c security.secureboot=false

This VM runs with full native CPU capabilities. It cannot migrate (local storage, native CPU) but runs faster and works offline.

Summary

Profile Target CPU Storage Migration
migratable @migratable qemu64 Ceph Yes (cold)
local <hostname> native btrfs No

Management

Cluster Operations

Evacuate a member before maintenance (migrates its VMs away):

incus cluster evacuate desktop
incus cluster restore desktop

Remove offline member permanently:

incus cluster remove --force desktop

VM Operations

Start, stop, and restart:

incus start <vm>
incus stop <vm>
incus restart <vm>

Access the VGA console (works from any cluster member — Incus routes to whichever host runs the VM):

incus console <vm> --type=vga

SSH into a VM (if SSH is configured inside it):

incus exec <vm> -- bash

Show VM config and devices:

incus config show <vm>
incus config device show <vm>

List all VMs across the cluster:

incus list -c ns4t

Snapshots

Create a snapshot:

incus snapshot create <vm> <snapshot-name>

List snapshots:

incus snapshot list <vm>

Rename a snapshot:

incus snapshot rename <vm> <old-name> <new-name>

Restore:

incus snapshot restore <vm> <snapshot-name>

Delete:

incus snapshot delete <vm> <snapshot-name>

Storage

List storage volumes:

incus storage volume list shared
incus storage volume list local

Import an ISO as a storage volume (e.g. for VM installation media):

incus storage volume import <pool> <file.iso> <volume-name> --type=iso

Delete a volume:

incus storage volume delete <pool> <volume-name>

Known Issues

Live migration fails with ICH9LPC error — QEMU 10 has a bug where live migration fails at restore with “Missing section footer for ICH9LPC”. Cold migration (stop, move, start) works. Track upstream: QEMU #2046

Troubleshooting

Ceph Storage Operations Fail

If incus storage volume import shared or other Ceph operations fail with signal: killed or connection errors, the Ceph OSD may be down.

Check Ceph status

ceph -s

If you see 0 up in the OSD line, the OSD needs reactivation. This happens after server reboots because ceph-volume uses tmpfs mounts that don’t persist.

Reactivate OSD

sudo ceph-volume lvm activate --all
sudo systemctl start ceph-osd-0.service

Verify OSD is up

ceph -s

Should now show 1 up in the OSD line.

References