CloudStack 4.22 – VM deployment on LINSTOR primary fails during ROOT volume population (qemu-img convert), while volume creation succeeds #12388
Replies: 8 comments
-
|
Thanks for opening your first issue here! Be sure to follow the issue template! |
Beta Was this translation helpful? Give feedback.
-
|
I am uploading the logs here. for the kvm's and the mgmt tar cvpzf ~/cskvm0*.logs.tar.gz /var/log/linstor-satellite/ /var/log/linstor-controller/ /var/log/cloudstack/ /var/log/libvirt/ /var/log/syslog /var/log/dmesg /var/log/kern.log tar cvpzf ~/csmgmt01.logs.tar.gz /var/log/cloudstack/ /var/log/syslog /var/log/dmesg /var/log/kern.log cskvm03.logs.tar.gz Also some settings of linstor root@cskvm01:~# linstor node list linstor sp l root@cskvm01:~# linstor sp l Resource Groupslinstor rg l root@cskvm01:~# linstor rg l root@cskvm01:~# On allno other parameter is set for linstor drbd. #Current Issue
But as a additional disk i can use linstor for existing vm created via nfs. it also migrated to another host without isssue live. can also create standalone volumes with linstor. I only can't create VM's with linstor before this on plain install and linstor as a only primary storage even system vm's where not making up. so this time i used nfs as primary than second primary linstor. VM spinup issue. i think this is mostly due to some template clone / raw / qcow2 issue as per some observation in logs. Update: I tried to deploy vm with ALPINE ISO and linstor and it worked. But it is not working with the default template from the cloudstack wizards (qcow2/img) 
root@cskvm01:/var/log/cloudstack/agent# linstor v l |
Beta Was this translation helpful? Give feedback.
-
|
In the Linbit forums I suspected that the wrong format in the Database might be a problem for the templates, So I simply suspect, that the templates images themselves are not working? Can you try with some other templates, like simple ubuntu cloud images? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @rp- ,
To resolve Point 2 that is deployment of template or Instance making issue i tried to convert the qcow2 image to raw didn't work than i made changes in the database of template to recognize it as raw but still failed. I tried vice versa still failed. In the linstor forum i was blocked to reply but i think it is now operational. That's why i was unable to reply there. If you haven't checked the log yet u can. I would suggest to check FIRST failures not last as u will find why i needed to try to fix it.. https://drive.google.com/file/d/1-RLNwX0lBIACjoB2luSxTncBFj3JHCtk/view?usp=drive_link Logs for the csmgmt01, cskvm01,cskvm02,cskvm03 #MGMT Currently i have installed HCI and checking it. Its working fine out of the box. Will again erase and install the same architecture mentioned here this weekend. And this time i will keep it as is. I mean will not touch much will report out of the box experience. |
Beta Was this translation helpful? Give feedback.
-
|
@bilalinamdar And then the first error is already this: But now as I look at it, I found it more interessting that there is already an error mounting the nfs secondary share and I'm a bit surprised that it then even tried to |
Beta Was this translation helpful? Give feedback.
-
|
@rp- Thanks for responding. |
Beta Was this translation helpful? Give feedback.
-
|
I'm also sharing here my "playbook" from Milan. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, This is initial Investigation. So i used virter and replicated exact env as yours from the playbook.
So i investigated further and renamed the .img of ubuntu 24.04 to qcow2 and reuploaded it as new template and its deployment worked. later i also used. Let me check further as this was virter env. but atleast i am happy for now. Will update soon as this is initial investigation 
remember the ubuntu 22 and 24 are from the official wizard download and not manual (.img) To summarize
extra command needed in the poc doc #On main I have seen similar .img issue in proxmox forum somewhere. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
sos_2026-01-02_15-03-28.tar.gz
###Architecture
┌──────────────────────────────────┐ ┌──────────────────────────────────┐ ┌──────────────────────────────────┐
│ KVM NODE 1 │ │ KVM NODE 2 │ │ KVM NODE 3 │
│ Hostname : cskvm01.poc.local │ │ Hostname : cskvm02.poc.local │ │ Hostname : cskvm03.poc.local │
│ IP : 10.50.11.101 │ │ IP : 10.50.11.102 │ │ IP : 10.50.11.103 │
│ │ │ │ │ │
│ CloudStack KVM Agent : 4.22.0.0 │ │ CloudStack KVM Agent : 4.22.0.0 │ │ CloudStack KVM Agent : 4.22.0.0 │
│ LINSTOR role : COMBINED │ │ LINSTOR role : SATELLITE │ │ LINSTOR role : SATELLITE │
│ LINSTOR Ctrl/GUI : Yes │ │ LINSTOR Ctrl/GUI : No │ │ LINSTOR Ctrl/GUI : No │
│ DRBD kernel : 9.3.0 │ │ DRBD kernel : 9.3.0 │ │ DRBD kernel : 9.3.0 │
│ Storage pool (LVMTHIN): lvm-thin-fast on each node (~223GiB free) │
└──────────────────────────────────┘ └──────────────────────────────────┘ └──────────────────────────────────┘
======================================================================
STORAGE (CloudStack Datastores)
(A) Primary Storage #1 (NFS Primary) [Shared NFS datastore for KVM]
┌──────────────────────────────────────────────────┐
│ NFS Primary Storage │
│ Server IP : 10.50.10.100 (csmgmt01) │
│ Export : /export/primary (example) │
│ Protocol : NFSv4.2 │
│ Used for : Primary volumes on NFS (non-Linstor) │
└──────────────────────────────────────────────────┘
(B) Primary Storage #2 (LINSTOR Primary) [Your “second primary”]
┌──────────────────────────────────────────────────────────────┐
│ LINSTOR Primary Storage (Pool name in CS: linstor-primary) │
│ CloudStack pool_type : Linstor │
│ LINSTOR controller : http://10.50.11.101:3370 │
│ Backend pools : LVM_THIN (lvm-thin-fast) on nodes │
│ Used for : Primary volumes on DRBD (/dev/drbdX) │
└──────────────────────────────────────────────────────────────┘
(C) Secondary Storage (NFS Secondary) [templates/isos/systemvms]
┌──────────────────────────────────────────────────┐
│ NFS Secondary Storage │
│ Server IP : 10.50.10.100 (csmgmt01) │
│ Export : /export/secondary │
│ Protocol : NFSv4.2 │
│ Used for : Templates / ISOs / SystemVM templates │
└──────────────────────────────────────────────────┘
problem
CloudStack 4.22 – VM deployment on LINSTOR primary fails during ROOT volume population (qemu-img convert), while volume creation succeeds
======================================================================
Deployment Topology
Management Node:
KVM + LINSTOR Cluster:
LINSTOR Deployment Model:
All three KVM nodes:
Storage Pools:
Secondary Storage:
======================================================================
Environment Summary
MANAGEMENT NODE (csmgmt01)
Role: CloudStack Management Server
OS: Ubuntu 22.04.5 LTS
CloudStack Version: 4.22.0.0
DB Schema Version: 4.22.0.0
Installed Packages:
Primary Storage (CloudStack DB):
KVM HOST (example: cskvm01)
Role: CloudStack KVM Hypervisor + LINSTOR Controller
OS: Ubuntu 22.04.5 LTS
Kernel: 5.15.0-164-generic
CloudStack Agent:
LINSTOR:
DRBD:
QEMU / libvirt:
Virtualization checks:
======================================================================
Templates
Templates registered in CloudStack DB:
Ubuntu 22.04
Ubuntu 24.04
On KVM hosts, template files stored on secondary NFS and named *.raw
are detected as QCOW2 via:
qemu-img info
Example:
Service Offering
Service offering used: testlinstor
ROOT disk sizes tested:
ROOT volume sizes verified in CloudStack DB match the offering.
======================================================================
Observed Problem
LINSTOR primary storage is detected as UP in CloudStack and volumes can
be created successfully across the 3-node LINSTOR cluster. However,
VM deployment fails specifically during ROOT volume population
from template.
Key behavior:
======================================================================
Relevant KVM Agent Log Excerpts
INFO Linstor: Creating volume for ROOT disk
INFO Linstor: Created DRBD device: /dev/drbd1001
INFO Executing qemu-img convert to DRBD device
ERROR qemu-img convert failed: output file is smaller than input file
WARN Template copy failed, cleaning up DRBD resource
INFO Linstor: Removed DRBD device and volume as part of cleanup
CloudStack Management Log Excerpts
ERROR Unable to find ObjectInDataStore mapping for TemplateObject on Linstor storage pool
WARN Failed to create ROOT volume for VM, marking volume as Destroy
======================================================================
Database Evidence
Failed instances:
i-2-12-VM
i-2-13-VM
LINSTOR storage pool remains in state = Up throughout.
======================================================================
Key Observation
LINSTOR and DRBD are functioning correctly across all three KVM nodes:
The failure occurs only at the template-to-root-volume population stage
(qemu-img convert writing to /dev/drbdX).
This suggests an issue in CloudStack’s LINSTOR integration or template
handling during ROOT volume deployment, rather than a LINSTOR or DRBD
volume provisioning problem.
Expected Behavior
CloudStack should successfully populate the ROOT volume on LINSTOR
primary storage from the template and continue VM deployment without
cleaning up the DRBD resource.
versions
The versions of ACS, hypervisors, storage, network etc..
the first one is from nfs all other are created using linstor tag in compute offering.
linstor based volume can be deployed standalone but it can’t be used for vm creation fails.
Beta Was this translation helpful? Give feedback.
All reactions