How to Copy Azure Containers & Blobs

Question

I am attempting to copy all the blobs to different storage:

CloudBlobClient srcblobClient = sourceStorageAccount.CreateCloudBlobClient();
CloudBlobClient targetBlobClient = targetStorageAccount.CreateCloudBlobClient();

foreach (CloudBlobContainer cont in srcblobClient.ListContainers())
{
    foreach (IListBlobItem srcBlob in cont.ListBlobs(useFlatBlobListing: true))
    {                        
        var targetContainer = targetBlobClient.GetContainerReference(cont.Name);
        targetContainer.CreateIfNotExists();

        Uri thisBlobUri = srcBlob.Uri;
        var serverBlob = srcblobClient.GetBlobReferenceFromServer(thisBlobUri);

        ICloudBlob targetBlob = targetContainer.GetBlobReferenceFromServer(serverBlob.Name);

        targetBlob.StartCopyFromBlob(thisBlobUri);
    }
}

I am able to see the listing of blobs & copy method is being invoked targetBlob.StartCopyFromBlob(thisBlobUri);

However copy is not actually happening. Any Idea?

PS I am using Azure Storage SDK 4.3 & target storage is development storage.

EDIT 2:

For remote azure storage copy above code works fine.

However for Emulated storage I get 400 BadRequest error, when trying to create container: targetContainer.CreateIfNotExists();

My emulated storage version is 3.0, it seems there is conflict between azure SDK & emulator version.

Which version of storage client library works well with storage emulator 3.0?

Answer 1

The reason you're getting this error is indeed because of version mismatch. If I am not mistaken, Storage Emulator version 3.0 uses REST API version 2013-08-15 where as the latest version of storage client library uses REST API version 2014-02-14 (Ref: http://msdn.microsoft.com/en-us/library/azure/dn744252.aspx ). What you could is make use of an older version of storage client library. You can install appropriate version via Nuget. For example, if you want to install Storage Client Library version 3.2.1, you can do so by doing the following:

Install-Package WindowsAzure.Storage -Version 3.2.1 (Ref: http://www.nuget.org/packages/WindowsAzure.Storage/3.2.1 )

Please try it out and see if that fixes the problem.

Also looking at your code, I would also recommend some changes:

• I would not recommend changing the permission on Blob Container to Public . It kind of exposes your blob storage and makes it available via anonymous access. What I would recommend is that you create SAS URLs with Read Permission on your source blobs and copy using those SAS URLs. Since the blob copy is asynchronous, I would recommend keeping the SAS URL valid for 7 days (maximum time allocated for copy operation).
• I see that you're doing GetBlobReferenceFromServer on both source blob and target blob. This method is not recommended for source blob as it actually makes a network call thus for each blob you already got through listing. It is not recommended on target blob because this method will throw a Not Found (404) if your target blob does not exist.

Instead what I would recommend is that you cast the blobs you got through listing into appropriate blob type (Block or Page) and then get SAS URL. If you know that all of your blobs are block blobs, you can simply cast them into CloudBlockBlob object without worrying about the casting.

One thing I am not sure of is how page blobs will get copied. When you're copying between storage accounts, a page blob gets copied as a page blob. However I have not tried copying from a storage account to development storage account. But if you don't have page blobs, you don't have to worry about it :).