Azure Long-running external web service calls

Question

I've written a windows service in C# that interfaces with a third party via external web service calls (SOAP). Some of the calls respond quickly and some slowly

(for example, quick = retrieving a list of currencies; slow = retrieving a historical value of currencies per product over many years)

The external service works fine when I run it on my local machine - the quick calls runs for about 20 seconds; the slow calls runs for about 30 minutes. I can't do anything about the speed of the 3rd party service and I don't mind the time it takes to return an answer..

My problem is that when I deploy my service to my Azure Virtual Machine: The quick call works perfectly like it does locally; the slow ones just never returns anything. I have tried exception handling, logging to files, logging to eventLog.

There is no clear indication of what goes wrong - it just seems that for whatever reason - the long running web service calls, never return successfully to Azure.

I've read somewhere that there is some sort of connection-recycling happening every 4 minutes, which I suspect is somehow causing the external web service response to land up somewhere in a void with the load balancer or whatever not knowing any longer whom requested the content.

I start by creating the request with the relevant SOAP envelope, like this:

HttpWebRequest tRequest = (HttpWebRequest)WebRequest.Create(endpoint);

Then i set the stuff, like this:

tRequest.ClientCertificates.Add(clientCertificate);
tRequest.PreAuthenticate = true;
tRequest.KeepAlive = true;
tRequest.Credentials = CredentialCache.DefaultCredentials;
tRequest.ContentLength = byteArray.Length;
tRequest.ContentType = @"text/xml; charset=utf-8";
tRequest.Headers.Add("SOAPAction", @"http://schemas.xmlsoap.org/soap/envelope/");
tRequest.Method = "POST";
tRequest.Timeout = 3600000;
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; }; //the SSL certificate is bad

Stream requestStream = tRequest.GetRequestStream(); 
requestStream.Write(byteArray, 0, byteArray.Length);
requestStream.Close();
requestStream.Dispose(); //works fine up to this point.  

WebResponse webResponse = tRequest.GetResponse();  //The slow calls never make it past this. Fast one does

Anyone else experienced something similar and any suggestions how to solve it please?

Many thanks

http://www.fourtimesfour.co.za

Answer 1

When you deploy to Azure (Cloud Service, Virtual Machine) there is always the Azure Load Balancer, which sits between your VMs and the Internet.

In order to keep resources equally available to all the cloud users, Azure Load Balancer will kill idle connections. What idle for Azure LB is - default is 4 minutes of no communication sent over established channel. So if you call a web service and there is absolutely no response on the pip for 4 minutes - your connection will get terminated.

You can configure this timeout, but I would really argue the need for keeping connection open that long. Yes, you can do nothing about it, besides looking for a service which has better design (ie either returns responses faster, or implements asynchronous calls where the first call to the service will just give you a task id, using which you can poll periodically to get a result)

Here is a good article on how to configure the Azure Load Balancer timeout . Be aware, the maximum timeout for Azure LB is 30 minutes.