Creating Azure blob containers from a powershell cmdlet fails

Question

I have a small bit of C# code that creates an Azure Blob container just fine when run via a console-mode executable, but fails when the same code is part of a PowerShell cmdlet (also in C#). The failure mode is really odd: it creates the container, but then keeps retrying for a coupla minutes before it errors out. I don't get it.

The code, which is built in Visual Studio using .NET Framework 4.8, and Azure.Storage.Blobs v12.4.1:

using System.Management.Automation;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;

namespace BlobTest.Commands
{
    [Cmdlet(VerbsData.Initialize, "AzContainer")]
    public class Initialize_AzContainer : Cmdlet
    {
        protected override void ProcessRecord()
        {
            base.ProcessRecord();

            const string AzStorageAccount = "stevestorageaccount";
            const string AzAccessKey = "*****==";

            string connectionString = $"DefaultEndpointsProtocol=http" // http intentional: for wireshark
                                      + $";AccountName={AzStorageAccount}"
                                      + $";AccountKey={AzAccessKey}"
                                      + ";EndpointSuffix=core.windows.net";

            const string containerName = "my-test-container-2";

            var bsclient = new BlobServiceClient(connectionString);

            WriteVerbose($"Creating container {containerName}");
            var rc = bsclient.CreateBlobContainer(containerName, PublicAccessType.None);

            WriteVerbose($"Result = {rc}");
        }
    }
}

This hangs for a coupla minutes before barfing:

PS> Import-Module .\BlobTest.Commands.dll
PS> Initialize-AzContainer -Verbose
VERBOSE: Creating container my-test-container-2

[hung out for 7 minutes]

Initialize-AzContainer : Retry failed after 6 tries.
At line:1 char:1
+ Initialize-AzContainer -Verbose
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Initialize-AzContainer], AggregateException
    + FullyQualifiedErrorId : System.AggregateException,BlobTest.Commands.Initialize_AzContainer

I intentionally use insecure http while troubleshooting so I can sniff the conversation, and Wireshark shows really odd behavior. The // in the traces are my notes, not part of the trace:

// MY END SENDS: this looks normal
PUT /my-test-container-2?restype=container HTTP/1.1
x-ms-version: 2019-07-07
x-ms-client-request-id: acc4991b-33ab-4764-9f20-a33edb5cc4a3
x-ms-return-client-request-id: true
User-Agent: azsdk-net-Storage.Blobs/12.4.1 (.NET Framework 4.8.4150.0; Microsoft Windows 10.0.18363 )
x-ms-date: Wed, 29 Apr 2020 17:42:27 GMT
Authorization: SharedKey <secret key info>
Host: stevestorageaccount.blob.core.windows.net
Content-Length: 0

// AZURE REPLIES: all ok!
HTTP/1.1 201 Created
Content-Length: 0
Last-Modified: Wed, 29 Apr 2020 17:42:28 GMT
ETag: "0x8D7EC64AF7BD68A"
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 5ebb740e-201e-001a-354d-1e15bd000000
x-ms-client-request-id: acc4991b-33ab-4764-9f20-a33edb5cc4a3
x-ms-version: 2019-07-07
Date: Wed, 29 Apr 2020 17:42:27 GMT

// MY END TRIES AGAIN - huh?
PUT /my-test-container-2?restype=container HTTP/1.1
x-ms-version: 2019-07-07
x-ms-client-request-id: acc4991b-33ab-4764-9f20-a33edb5cc4a3
x-ms-return-client-request-id: true
User-Agent: azsdk-net-Storage.Blobs/12.4.1 (.NET Framework 4.8.4150.0; Microsoft Windows 10.0.18363 )
x-ms-date: Wed, 29 Apr 2020 17:42:29 GMT
Authorization: SharedKey <secret key info>
Host: stevestorageaccount.blob.core.windows.net
Content-Length: 0

// AZURE REPLIES: sorry already exists
HTTP/1.1 409 The specified container already exists.
Content-Length: 230
Content-Type: application/xml
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 5ebb7805-201e-001a-224d-1e15bd000000
x-ms-client-request-id: acc4991b-33ab-4764-9f20-a33edb5cc4a3
x-ms-version: 2019-07-07
x-ms-error-code: ContainerAlreadyExists
Date: Wed, 29 Apr 2020 17:42:29 GMT
...<?xml version="1.0" encoding="utf-8"?>
<Error>
<Code>ContainerAlreadyExists</Code>
<Message>The specified container already exists.
RequestId:5ebb7805-201e-001a-224d-1e15bd000000
Time:2020-04-29T17:42:29.6128339Z</Message></Error>

(repeats a few times)

The same code works fine in a console-mode executable, and it behaves the same way whether I use async or regular synchronous as above. I've tried various other combinations, adding a cancellation token, etc. and nothing impacts this.

Note: I know how to create a container with the native Azure cmdlets, but the above is part of additional processing, so I'm trying to get to the bottom of the failure, not find some other way to create a container.

I've spent quite a few hours at this, and am at a loss to see what's going on.

Edit: I've tried rebooting my system, no difference, and this is all with PowerShell 5.1.18362.752 on Windows 10 with all latest updates.

Answer 1

It seems that the retry policy is triggered here. But the weird thing is that in a console project, it does not retry if operation is successful.

As a workaround, in your code, you can set the retry policy, sample code like below:

[Cmdlet(VerbsData.Initialize, "AzContainer")]
public class Initialize_AzContainer : Cmdlet
{
    protected override void ProcessRecord()
    {
        base.ProcessRecord();

       //other code

       //set the retry policy here
        var options = new BlobClientOptions();
        options.Retry.MaxRetries = 0;

        var bsclient = new BlobServiceClient(connectionString, options);

        WriteVerbose($"Creating container {containerName}");

        var rc = bsclient.CreateBlobContainer(containerName, PublicAccessType.None);
        WriteVerbose($"Result = {rc}");


    }

}