Instance OS cannot find attached volume

Question

I have been trying to figure this out since 2 days without success. Let me know if you have faced similar issue and how to fix it. Or any pointers will be of help.

Here is what we do.

1. Create virtual instance.
2. Create blank volume.
3. Create volume from snapshot.
4. Attach blank volume to instance.
5. Attach volume from snapshot to instance.
6. Other IP address, security group stuff.

All these are done in separate threads but with dependencies added. Ex. 4 depends on 2 (4->2). 2->1. 5->3->1.

The problem is that sometimes:

1. One of the devices don't appear in the device list. Openstack shows both attached.
2. Both device don't appear in the device list. Openstack shows both attached.
3. Both appear but, disks at specified devices get switched! Ex. We attached 1GB disk on /dev/vdb and 10GB disk on /dev/vdc. But when we do fdisk –l /dev/vdc we get 1G and /dev/vdb shows 10G.
4. Works!!!

I have checked the code and all components are doing what they are supposed to do. That is leading me to suspect openstack itself! (I know… that's kind of strange.)

I am using allinone with icehouse and RDO packstack setup.

[Update 1]

I am using JClouds APIs to attach the volumes. Also, I am specifying the device names (/dev/xvdb, /dev/xvdc). Yet, I am checking for device names in the response after the call and using them instead.

NovaAPI nova = ContextBuilder
            .newBuilder("openstack-nova")
            .endpoint(endpoint)
            .credentials(getIdentity(), getPassword())
            .modules(modules).buildApi(NovaApi.class);
VolumeAttachmentApi attachment = nova.getVolumeAttachmentExtensionForZone(CONSTANT_REGION).get();
VolumeAttachment attachment.attachVolumeToServerAsDevice(volumeIdExisting, instanceId, deviceDevXVDB);

[Update 2] I created a sample program to isolate the issue. This sample program does the following:

    /-> 2 -> 3 >\
1 -> -> 6 -> -> x
    \-> 4 -> 5 >/

Basically the main thread creates the instance and launches two threads. One thread creates blank volume (10GB) and attaches it. While the other creates a volume from snapshot (1GB) and attaches it. The main thread then creates IP and attaches it and waits for the two threads to complete.

The observation is rather strange. Every single time, the attach volume response says the blank volume got attached to /dev/vdc, and the volume from snapshot attached to /dev/vdb.

But when I checked on the instance

fdisk -l /dev/vdb

returns 10GB and

fdisk -l /dev/vdc

returns 1GB

What could I be missing?? Any pointers/suggestions would be helpful.

Answer 1

Just for the record. Attaching volumes simultaneously will not work. The device names assigned to attached volume is dependent on which gets attached first. Even though we may specify the name, it may not be honored by the underlying driver.

Finally, we did away with the parallel attachment of volumes to sequential and now everything works as expected.