azure queue performance

Question

For the windows azure queues the scalability target per storage is supposed to be around 500 messages / second ( http://msdn.microsoft.com/en-us/library/windowsazure/hh697709.aspx ). I have the following simple program that just writes a few messages to a queue. The program takes 10 seconds to complete (4 messages / second). I am running the program from inside a virtual machine (on west-europe) and my storage account also is located in west-europe. I don't have setup geo replication for my storage. My connection string is setup to use the http protocol.

       // http://blogs.msdn.com/b/windowsazurestorage/archive/2010/06/25/nagle-s-algorithm-is-not-friendly-towards-small-requests.aspx
        ServicePointManager.UseNagleAlgorithm = false;

        CloudStorageAccount storageAccount=CloudStorageAccount.Parse(ConfigurationManager.AppSettings["DataConnectionString"]);

        var cloudQueueClient = storageAccount.CreateCloudQueueClient();

        var queue = cloudQueueClient.GetQueueReference(Guid.NewGuid().ToString());

        queue.CreateIfNotExist();
        var w = new Stopwatch();
        w.Start();
        for (int i = 0; i < 50;i++ )
        {
            Console.WriteLine("nr {0}",i);
            queue.AddMessage(new CloudQueueMessage("hello "+i));    
        }

        w.Stop();
        Console.WriteLine("elapsed: {0}", w.ElapsedMilliseconds);
        queue.Delete();

Any idea how I can get better performance?

EDIT:

Based on Sandrino Di Mattia's answer I re-analyzed the code I've originally posted and found out that it was not complete enough to reproduce the error. In fact I had created a queue just before the call to ServicePointManager.UseNagleAlgorithm = false; The code to reproduce my problem looks more like this:

        CloudStorageAccount storageAccount=CloudStorageAccount.Parse(ConfigurationManager.AppSettings["DataConnectionString"]);

        var cloudQueueClient = storageAccount.CreateCloudQueueClient();

        var queue = cloudQueueClient.GetQueueReference(Guid.NewGuid().ToString());

        //ServicePointManager.UseNagleAlgorithm = false; // If you change the nagle algorithm here, the performance will be okay.
        queue.CreateIfNotExist();
        ServicePointManager.UseNagleAlgorithm = false; // TOO LATE, the queue is already created without 'nagle'
        var w = new Stopwatch();
        w.Start();
        for (int i = 0; i < 50;i++ )
        {
            Console.WriteLine("nr {0}",i);
            queue.AddMessage(new CloudQueueMessage("hello "+i));    
        }

        w.Stop();
        Console.WriteLine("elapsed: {0}", w.ElapsedMilliseconds);
        queue.Delete();

The suggested solution from Sandrino to configure the ServicePointManager using the app.config file has the advantage that the ServicePointManager is initialized when the application starts up, so you don't have to worry about time dependencies.

Answer 1

I answered a similar question a few days ago: How to achive more 10 inserts per second with azure storage tables .

For adding 1000 items in table storage it took over 3 minutes, and with the changes I described in my answer it dropped to 4 seconds (250 requests/sec). In the end, table storage and storage queues aren't all that different. The backend is the same, data is simply stored in a different way. And both table storage and queues are exposed through a REST API, so if you improve the way you handle your requests, you'll get a better performance.

The most important changes:

• expect100Continue : false
• useNagleAlgorithm : false (you're already doing this)
• Parallel requests combined with connectionManagement/maxconnection

Answer 2

Also, ServicePointManager.DefaultConnectionLimit should be increased before making a service point. Actually Sandrino's answer says the same thing but using config.

Turn off proxy detection even in the cloud. Auto detect in proxy config element. Slows initialisation.

Choose distributed partition keys.

Collocate your account near to compute, and customers.

Design to add more accounts as needed.

Microsoft set the SLA at 2,000 tps on queues and tables as of 07 2012.

I didn't read Sandrino's linked answer, sorry, just was on this question as I was watching Build 2012 session on exactly this.