Two virtual machines limited in one physical machine above Eucalyptus Xen

Question

I have an Eucalyptus-based cloud using Xen with 3 nodes. When I run a VM more than twice the number of physical machines in the xen cluster, the extra VMs never boot up, they always change state from pending to terminate, and the log file shows the node controller shutoff the extra VM after it check the instances number. When I manually shutdown some VM, then I can easily start up some VM exactly as the number I've shutdown.

The log shows this error:

ERROR: [Tue Jan 17 12:51:05 2012][004231][EUCAERROR ] libvirt: POST operation failed: xend_post: error from xen daemon: (xend.err 'Device 2049 (vbd) could not be connected. /etc/xen/scripts/block failed; error detected.') (code=11) [Tue Jan 17 12:51:05 2012][004231][EUCADEBUG ] doDescribeResource() invoked [Tue Jan 17 12:51:06 2012][004231][EUCADEBUG ] doDescribeInstances() invoked [Tue Jan 17 12:51:06 2012][004231][EUCADEBUG ] doDescribeInstances(): instanceId=i-3BB00757 publicIp=192.168.4.165 privateIp=192.168.4.165 mac=d0:0d:3B:B0:07:57 vlan=-1 networkIndex=-1 [Tue Jan 17 12:51:06 2012][004231][EUCADEBUG ] doDescribeInstances(): instanceId=i-473807EE publicIp=192.168.4.112 privateIp=192.168.4.112 mac=d0:0d:47:38:07:EE vlan=-1 networkIndex=-1 [Tue Jan 17 12:51:06 2012][004231][EUCADEBUG ] doDescribeInstances(): instanceId=i-348A06AE publicIp=0.0.0.0 privateIp=0.0.0.0 mac=d0:0d:34:8A:06:AE vlan=-1 networkIndex=-1 [Tue Jan 17 12:51:06 2012][004231][EUCAERROR ] libvirt: POST operation f ailed: xend_post: error from xen daemon: (xend.err 'Device 2049 (vbd) could not be connected. /etc/xen/scripts/block failed; error detected.') (code=11) [Tue Jan 17 12:51:08 2012][004231][EUCAERROR ] libvirt: POST operation failed: xend_post: error from xen daemon: (xend.err 'Device 2049 (vbd) could not be connected. /etc/xen/scripts/block failed; error detected.') (code=11) [Tue Jan 17 12:51:09 2012][004231][EUCAERROR ] libvirt: POST operation failed: xend_post: error from xen daemon: (xend.err 'Device 2051 (vbd) could not be connected. /etc/xen/scripts/block failed; error detected.') (code=11) [Tue Jan 17 12:51:10 2012][004231][EUCAERROR ] libvirt: POST operation failed: xend_post: error from xen daemon: (xend.err 'Device 2051 (vbd) could not be connected. /etc/xen/scripts/block failed; error detected.') (code=11) [Tue Jan 17 12:51:10 2012][004231][EUCAFATAL ] hypervisor failed to start domain [Tue Jan 17 12:51:10 2012][004231][EUCADEBUG ] state change for instance i-348A06AE: Staging -> Shutoff (Extant) [Tue Jan 17 12:51:10 2012][004231][EUCAERROR ] libvirt: Domain not found: xenUnifiedDomainLookupByName (code=42) [Tue Jan 17 12:51:10 2012][004231][EUCAINFO ] cleaning up state for instance i-348A06AE

Is this a limitation for CentOS 5.* and RedHat 5. ? Is it fixed on CentOS 6. ?

http://docs.vmd.citrix.com/XenServer/5.0.0/1.0/en_gb/guest.html#rhel5_limitations

"Only 3 virtual network interfaces are supported in versions below 5.2. For 5.2 and above, 7 virtual network interfaces are supported."

I opened this topic in both CentOS and Eucalyptus forums without luck:

• https://www.centos.org/modules/newbb/viewtopic.php?topic_id=35491&forum=38
• http://open.eucalyptus.com/forum/unable-run-more-4-instances-even-extra-resources

Thanks, Leandro

Answer 1

The solution was to increase number of loop devices in CentOS 5, as detailed in the following link:

http://open.eucalyptus.com/participate/wiki/increase-number-loop-devices-centos-5

As the link is no longer valid, you have to go to archive.org or see this similar answer:

how to increase the loop devices number

Answer 2

The error you are seeing seems related to disk not to network (Device 2049 (vbd) could not be connected. /etc/xen/scripts/block failed; error detected). I don't see that script on a CentOS 5.7 install I have here, so you may want to dig into that script to see if there is some obvious error.