Azure ML web service times out

Question

I have created a simple experiment in Azure ML and trigger it with an http client. In Azure ML workspace, everything works ok when executed. However, the experiment times out and fails when I trigger the experiment using an http client. Setting a timeout value for the http client does not seem to work.

Is there any way we can set this timeout value so that the experiment does not fail?

Answer 1

Make sure you're setting the client timeout value correctly. If the server powering the web service times out, then it will send back a response with the HTTP status code 504 BackendScoreTimeout (or possibly 409 GatewayTimeout ). However, if you simply never receive a response, then your client isn't waiting long enough.

You can find out a good amount of time by running your experiment in ML Studio. Go to the experiment properties to find out how long it ran for, and then aim for about twice that amount of time as a timeout value.

Answer 2

I've had similar problems with an Azure ML experiment published as a web service. Most of the times it was running ok, while sometimes it returned with a timeout error. The problem is that the experiment itself has a 90 seconds running time limit. So, most probably your experiment has a running time over this limit and returns with a timeout error. hth

Answer 3

Looks like it isn't possible to set this timeout based on a feature request that is still marked as "planned" as of 4/1/2018 .

The recommendation from MSDN forums from 2017 is to use the Batch Execution Service, which starts the machine learning experiment and then asynchronously asks whether it's done.

Here's a code snippet from the Azure ML Web Services Management Sample Code (all comments are from their sample code):

        using (HttpClient client = new HttpClient())
        {
            var request = new BatchExecutionRequest()
            {

                Outputs = new Dictionary<string, AzureBlobDataReference> () {
                    {
                        "output",
                        new AzureBlobDataReference()
                        {
                            ConnectionString = storageConnectionString,
                            RelativeLocation = string.Format("{0}/outputresults.file_extension", StorageContainerName) /*Replace this with the location you would like to use for your output file, and valid file extension (usually .csv for scoring results, or .ilearner for trained models)*/
                        }
                    },
                },    

                GlobalParameters = new Dictionary<string, string>() {
                }
            };

            client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", apiKey);

            // WARNING: The 'await' statement below can result in a deadlock
            // if you are calling this code from the UI thread of an ASP.Net application.
            // One way to address this would be to call ConfigureAwait(false)
            // so that the execution does not attempt to resume on the original context.
            // For instance, replace code such as:
            //      result = await DoSomeTask()
            // with the following:
            //      result = await DoSomeTask().ConfigureAwait(false)

            Console.WriteLine("Submitting the job...");

            // submit the job
            var response = await client.PostAsJsonAsync(BaseUrl + "?api-version=2.0", request);

            if (!response.IsSuccessStatusCode)
            {
                await WriteFailedResponse(response);
                return;
            }

            string jobId = await response.Content.ReadAsAsync<string>();
            Console.WriteLine(string.Format("Job ID: {0}", jobId));

            // start the job
            Console.WriteLine("Starting the job...");
            response = await client.PostAsync(BaseUrl + "/" + jobId + "/start?api-version=2.0", null);
            if (!response.IsSuccessStatusCode)
            {
                await WriteFailedResponse(response);
                return;
            }

            string jobLocation = BaseUrl + "/" + jobId + "?api-version=2.0";
            Stopwatch watch = Stopwatch.StartNew();
            bool done = false;
            while (!done)
            {
                Console.WriteLine("Checking the job status...");
                response = await client.GetAsync(jobLocation);
                if (!response.IsSuccessStatusCode)
                {
                    await WriteFailedResponse(response);
                    return;
                }

                BatchScoreStatus status = await response.Content.ReadAsAsync<BatchScoreStatus>();
                if (watch.ElapsedMilliseconds > TimeOutInMilliseconds)
                {
                    done = true;
                    Console.WriteLine(string.Format("Timed out. Deleting job {0} ...", jobId));
                    await client.DeleteAsync(jobLocation);
                }
                switch (status.StatusCode) {
                    case BatchScoreStatusCode.NotStarted:
                        Console.WriteLine(string.Format("Job {0} not yet started...", jobId));
                        break;
                    case BatchScoreStatusCode.Running:
                        Console.WriteLine(string.Format("Job {0} running...", jobId));
                        break;
                    case BatchScoreStatusCode.Failed:
                        Console.WriteLine(string.Format("Job {0} failed!", jobId));
                        Console.WriteLine(string.Format("Error details: {0}", status.Details));
                        done = true;
                        break;
                    case BatchScoreStatusCode.Cancelled:
                        Console.WriteLine(string.Format("Job {0} cancelled!", jobId));
                        done = true;
                        break;
                    case BatchScoreStatusCode.Finished:
                        done = true;
                        Console.WriteLine(string.Format("Job {0} finished!", jobId));
                        ProcessResults(status);
                        break;
                }

                if (!done) {
                    Thread.Sleep(1000); // Wait one second
                }
            }
        }