LINSTOR DRBD
LINSTOR is a management layer, frequently deployed with DRBD which does the actual work of replicating a block device. It’s mostly known for being a very fast solution for virtualization clustering. It focuses on keeping a replica of a VMs disk up to date for quick migration and redundancy, with primary IO operating locally.
Install
DKMS Build Tools
https://linuxcontainers.org/incus/docs/main/howto/storage_linstor_setup/
This involves building the kernel modules from source and pulls in all the g++ and libs adding up to 575G of needed space.
Your instructed to use add-apt-repository but that’s an Ubuntu command and you can’t install it via software-properties-common anymore. So you’d next want to install from a repo provided by Linbit, but they only make part of their stack available for Trixie. To get the drbd module, You can bodge it by putting in a link to the proxmox repo that does have it all.
sudo apt update
sudo apt install curl gnupg
curl -fsSL https://packages.linbit.com/package-signing-pubkey.asc | gpg --dearmor | sudo tee /usr/share/keyrings/linbit-keyring.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/linbit-keyring.gpg] http://packages.linbit.com/public/ proxmox-9 drbd-9" | sudo tee /etc/apt/sources.list.d/linbit.list
You can then proceed.
# Note: If you're running the zabbly+ kernel, you'll need to add a .config link
sudo ln -s /boot/config-$(uname -r) /usr/src/linux-headers-$(uname -r)/.config
# In most cases the headers are already installed, but you can check with:
sudo apt install linux-headers-$(uname -r)
# Install the parts that are already build
sudo apt install lvm2 drbd-utils linstor-satellite
# Allow the dynamic kernel module system to do it's thing
sudo apt install drbd-dkms
Start the Satellite service on all nodes
sudo systemctl enable --now linstor-satellite
LINSTOR Controller
LINSTOR uses a single controller. On that system, run:
# Install the controller components
sudo apt install linstor-controller linstor-client python3-setuptools
# Enable the controller service
sudo systemctl enable --now linstor-controller
Configure
On the controller
TODO: I was able to add the node simply by simply omitting the IP address and entering the command “linstor node interface create shire2” and “linstor node interface create shire3”
# You will see a message about "Unsupported storage providers". This is expected as we have only installed LVM
linstor node create server01 <server_1> --node-type combined
linstor node create server02 <server_2> --node-type satellite
linstor node create server03 <server_3> --node-type satellite
# These command should now show all nodes and relevant features
linstor node list
linstor node info
Add the drives to linstore. You may want to run a wipefs -a /dev/sdX on the nodes to preempt any error messages.
# Change the server names such as "server01" and device names as appropriate.
linstor physical-storage create-device-pool --storage-pool nvme_pool --pool-name nvme_pool lvmthin server01 /dev/nvme1n1
linstor physical-storage create-device-pool --storage-pool nvme_pool --pool-name nvme_pool lvmthin server02 /dev/nvme1n1
linstor physical-storage create-device-pool --storage-pool nvme_pool --pool-name nvme_pool lvmthin server03 /dev/nvme1n1
# Double check that they are all up.
linstor storage-pool list
Tell Incus that you have a linstor volume
# Tell Incus where the linstor controller is
incus config set storage.linstor.controller_connection=http://<server_1>:3370
# Create a storage pool named 'remote' in incus on the individual nodes
incus storage create remote linstor --target server01
incus storage create remote linstor --target server02
incus storage create remote linstor --target server03
# link the incus storeage pool to the linstore storage pool named 'nvme_pool`
incus storage create remote linstor linstor.resource_group.storage_pool=nvme_pool
Launch an instance to test
sudo incus launch images:debian/13 c2 --storage remote
Launching c2
If this command hangs, you must cancel out (the process will keep trying for a while in the background) and check to make sure all the nodes see each other with the command sudo drbdadm status at each node.
Operations
Moving Instances to the CLuster
The normal thing to do is migrate existing containers. The only gotcha there is size. Incus creates a 10G volume on the remote disk. This is LVM thin provisioned, so it won’t use it unless it needs it, but it does set an upper limit on how big the instance can get. If you’re brining in an instance larger than 10G you’ll need to change the default volume size temporarily or you’ll get a cryptic error message about rsync and other issues.
# Check your size then move with an optional parameter to set the volume size if needed.
sudo du -sh /var/lib/incus/storage-pools/local/containers/
4.5G /var/lib/incus/storage-pools/local/containers/desktop
68G /var/lib/incus/storage-pools/local/containers/database
...
...
# This one is under the default 10GB so nothing extra is needed
incus move desktop --storage remote
# Since this on is 68G, set the size up, move it, then unset it
incus storage set remote volume.size 100Gib
incus move database --storage remote
incus storage unset remote volume.size
# Some resouces suggesting setting this option, but it fails
#incus move database --storage remote -d root,size=100GiB
Verifying its running locally.
When you move your instance it lands on two nodes. The nodes are chosen by linstore’s auto-place logic and incus doesn’t get to pick. So you’ll want to check instance is running on one of those two nodes for the best speed.
Let’s first look at a case where it lined up.
Take a look in incus to see where your instance is running. Here we can see it’s running on the server “shire2”
sudo incus list name=something
+-----------+---------+-----------------------+---------------------------------------------+-----------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+-----------+---------+-----------------------+---------------------------------------------+-----------+-----------+----------+
| something | RUNNING | 192.168.250.16 (eth0) | fd96:bd12:59eb:1:1266:6aff:fea3:b529 (eth0) | CONTAINER | 0 | shire2 |
+-----------+---------+-----------------------+---------------------------------------------+-----------+-----------+----------+
Now let’s ask linstore where it’s disk is at.
# Get the volume UUID
sudo linstor resource-definition list --show-props Aux/Incus/name | grep something
| incus-volume-38a5a7036bdf4001a073bc559ec06212 | remote | DRBD,STORAGE | ok | incus-volume-something
# See where its running at
linstor resource list
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Layers ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-volume-434bcfb9288d460fbfee6482dbf9670c ┊ shire1 ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ UpToDate ┊ 2026-03-21 20:47:02 ┊
┊ incus-volume-434bcfb9288d460fbfee6482dbf9670c ┊ shire2 ┊ DRBD,STORAGE ┊ InUse ┊ Ok ┊ UpToDate ┊ 2026-03-21 20:47:02 ┊
┊ incus-volume-434bcfb9288d460fbfee6482dbf9670c ┊ shire3 ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ TieBreaker ┊ 2026-03-21 20:47:02 ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
You can see that it’s InUse and UpToDate on shire2, the same server the instance is running on.
$ linstor resource list
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Layers ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-volume-38a5a7036bdf4001a073bc559ec06212 ┊ shire1 ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ UpToDate ┊ 2026-03-22 08:53:11 ┊
┊ incus-volume-38a5a7036bdf4001a073bc559ec06212 ┊ shire2 ┊ DRBD,STORAGE ┊ InUse ┊ Ok ┊ Diskless ┊ 2026-03-22 08:53:10 ┊
┊ incus-volume-38a5a7036bdf4001a073bc559ec06212 ┊ shire3 ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ UpToDate ┊ 2026-03-22 08:53:11 ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
You’ll notice the state Diskless on shire2…where it’s useage is InUse. That’s not ideal. Let’s fix that.
# Create a replica on the same node it's runing on. It will automatically start using it when it's done
linstor resource create shire2 incus-volume-38a5a7036bdf4001a073bc559ec06212 --storage-pool ssd_pool
linstor resource list
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Layers ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-volume-38a5a7036bdf4001a073bc559ec06212 ┊ shire1 ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ UpToDate ┊ 2026-03-22 08:53:11 ┊
┊ incus-volume-38a5a7036bdf4001a073bc559ec06212 ┊ shire2 ┊ DRBD,STORAGE ┊ InUse ┊ Ok ┊ SyncTarget(0.81%) ┊ 2026-03-22 08:53:10 ┊
┊ incus-volume-38a5a7036bdf4001a073bc559ec06212 ┊ shire3 ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ UpToDate ┊ 2026-03-22 08:53:11 ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
# After a while that will change to UpToDate and you can delete one of the other replicas
linstor resource delete shire3 incus-volume-38a5a7036bdf4001a073bc559ec06212
Troubleshooting
Building initial module drbd/9.3.0-1 for 6.18.8-zabbly+ grep: /lib/modules/6.18.8-zabbly+/build/.config: No such file or directory
Assuming you’ve already addressed the note about the missing .config file above, make sure you’re running the same version. If you’ve updated but not rebooted, you’ll see this error.
uname -a
If it’s trying to build a version that’s different than your running kernel, like an older version, you see a message something like
Building for 6.18.8-zabbly+ and 6.19.6-zabbly+
It can fail because you don’t have the old headers anymore. Simply purge the old kernel so it doesn’t try and build for it.
sudo apt purge *6.18.8-zabbly+*
DRBD not available.
If you check your note status you may see something like this
╭─────────────────────────────────────────────────────────────────────╮ ┊ Node ┊ DRBD ┊ LUKS ┊ NVMe ┊ Cache ┊ BCache ┊ WriteCache ┊ Storage ┊ ╞═════════════════════════════════════════════════════════════════════╡ ┊ shire1 ┊ + ┊ - ┊ - ┊ + ┊ - ┊ + ┊ + ┊ ┊ shire2 ┊ - ┊ - ┊ - ┊ + ┊ - ┊ + ┊ + ┊ ┊ shire3 ┊ - ┊ - ┊ - ┊ + ┊ - ┊ + ┊ + ┊ ╰─────────────────────────────────────────────────────────────────────╯
Only one of the nodes looks up. If you check the node that’s up it will show this:
cat /proc/drbd
version: 9.3.1 (api:2/proto:118-124)
If you’ve built the new DKMS module, but the other nodes show an older version such as version 8, try a reboot. It’s probably running default module. You may also be able to try a `modprobe -r drbd && modprobe drbd’ to load the new one
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.