Geocode api OVER QUERY LIMIT due to parallel for

Question

I have a requirement where i have to validate a lot of addresses (approx 5000) using Geocode API on server side. Therefore i decided to use parallel for to send multiple requests to the API at a time (in order to reduce execution time) This is my code

    //Address object
    public class AddressModel 
        {
            public int AddressID { get; set; }
            public string AddressLineOne { get; set; }
            public string AddressLineTwo { get; set; }
            public string City { get; set; }
            public string State { get; set; }
            public string ZipCode { get; set; }
        }

    //Main code starts here
    List<AddressModel> Addresses = GetAddressesFromFile();  //This will add around 5k addresses to the list
    Parallel.For(0, Addresses.Count, i =>
                    {
                        System.Net.WebResponse response = ResolveAddressFromGoogle(Addresses[i]);
                        //Process google response 
                    });


    public System.Net.WebResponse ResolveAddressFromGoogle(AddressModel address)
            {
                System.Net.WebResponse response = null;
                try
                {
            string urlGoogleApi = "https://maps.googleapis.com/maps/api/geocode/xml?address={0}&sensor=false&key=mykey&components=postal_code:{1}";
                    urlGoogleApi = string.Format(urlGoogleApi, address.AddressLineOne, address.Zip.ZipCode);
                    System.Net.WebRequest req = System.Net.WebRequest.Create(urlGoogleApi );
                    req.Method = "GET";
                    response = req.GetResponse();
                }
                catch (Exception ex)
                {
                    response = null;
                }
                return response;
            }

But the problem is that more than 50 requests are being sent to the API per second due to which i get an over query limit response for a lot of addresses. Is there a way to limit requests to 50 per second?

I tried a way where i tried delaying the task for 1 second if the response contains over_query_limit string, like this

Parallel.For(0, Addresses.Count, i =>
                    {
                        label:
                        System.Net.WebResponse response = ResolveAddressFromGoogle(Addresses[i]);
                        //If response contain over_query_limit
                        Task.delay(1000);
                        goto label;
                        //else process google response normally
                    });

But it doesn't seem to work. Any help will be appreciated

Answer 1

Try specifying the MaxDegreeOfParallelism in ParallelOptions and await the response from the API. I'm not sure it will give you 50 connections, that is just the maximum it is permitted to give if you specify 50. I have used 5 to illustrate it working. Given 20 elements, a maximum speed of 1 second per element and a maximum of 5 concurrent, it should take at least 4 seconds to complete the for loop.

        List<string> addresses = new List<string>();
        for (int i = 0; i < 20; i++)
        {
            addresses.Add(string.Empty);
        }

        Parallel.For(0, 
                     addresses.Count, 
                     new ParallelOptions{MaxDegreeOfParallelism = 5}, 
                     i =>
                    {
                        Console.WriteLine(DateTime.UtcNow.ToString("HH:mm:ss"));
                        Task.Delay(1000).Wait(); // simulating a long running event
                    }
        );