Azure Function runs into System.Net.Sockets.SocketException

Question

I have AzureFunctions app with 2 HTTP trigered functions. Both are direved from same class but use different urls to fetch data. Every day Azure Data Factory pipeline triggers 1st HTTP function and another pipeline calls 2nd function within 1 minute

Each function makes around 1300 HTTP requests to 3rd party website and stores each response as separate json file in Blob storage.

The problem is almost each time (but not always) 2nd function throws System.Net.Sockets.SocketException because few outbound requests run into common 21 seconds TCP timeout. Strange thing I've notices - it is likely Azure throttles for some reason my outbound requests: first batch takes near 300ms, next sequence takes 4.3sec, then 9.5sec and next batch reaches 21 sec with exception

Here is image of timing increasing of outbound requests

Exception stacktrace:

System.Net.Http.HttpRequestException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) at System.Threading.Tasks.ValueTask 1.get_Result() at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Threading.Tasks.ValueTask 1.get_Result() at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Threading.Tasks.ValueTask 1.get_Result() at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask 1 creationTask) at System.Threading.Tasks.ValueTask 1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at FunctionApp.BaseFunc.<>c__DisplayClass7_2.<b__0>d.MoveNext() in E:\vsts-agent-win-1_work\339\s\Services\Host\Controllers\BaseFunc.cs:line 102 --- End of stack trace from previous location where exception was thrown --- at FunctionApp.BaseFunc.ProcessRun(ILogger log, String runId) in E:\vsts-agent-win-1_work\339\s\Services\Host\Controllers\BaseFunc.cs:line 122.

FunctionApp is hosted on AppService plan S1 so there is no limit of outbound connections of 600 (I believe so)

Metrics of TCP connections during exception (max was 498): Metrics of AzureFunction App

TCP Connections from "Solve problem" helper of AzureFunction App Max TCP connections in all states was 502

CPU and Memory of App service plan during exception: App Service Plan metrics

App is.Net Core 2.2

I didn't manage to reproduce this on my local PC. But on Azure it happens almost every day on each environment (dev, test, prod). After such fail Azure Data Factory makes retry in 5 minutes and it is successfull each time.

Here is code of Base class that is used by both functions:

 public abstract class BaseFunc
{
    protected abstract string BlobFolderName { get; }
    protected TelemetryClient telemetryClient;
    private static HttpClient _httpClient;

    static BaseFunc()
    {
        HttpClientHandler handler = new HttpClientHandler();
        handler.MaxConnectionsPerServer = 300;
        _httpClient = new HttpClient(handler);
    }
    protected async Task ProcessRun(ILogger log, string runId)
    {
        int processedItems = 0;
        try
        {
            Stopwatch sw = Stopwatch.StartNew();
            ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

            await Authentication("url", log, runId); //sets default Authorization header

            string getIdeaResult = await _httpClient.GetStringAsync("url");
            JObject jsonObject = JObject.Parse(getIdeaResult);
            int ideaCount = (int)jsonObject.SelectToken("total_count");

            List<Task> tasks = new List<Task>();
            string DataPulledDate = DateTime.Now.ToString("dd-MMM-yyyy");
            CloudStorageAccount storageAccount = CloudStorageAccount.Parse("connection string");
            CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
            CloudBlobContainer cloudBlobContainer = cloudBlobClient.GetContainerReference("container");

            string getIdsUri = "url" + $"&limit={batchSize}&offset=";
            int iterations = (int)Math.Ceiling((decimal)ideaCount/batchSize);

            for (int i = 0; i < iterations; i++)
            {
                string result = await _httpClient.GetStringAsync("url" + i * 50);
                JObject jsonIdsObject = JObject.Parse(result);
                int[] ideaIds = jsonIdsObject["content"].Children().Values<int>("id").ToArray();
                foreach (int id in ideaIds)
                {
                    tasks.Add(Task.Run(async () =>
                    {
                        string content = null;
                        using (var response = await _httpClient.SendAsync(new HttpRequestMessage(HttpMethod.Get, "url"+ id))) //Exception is thrown on this line
                        {
                            content = await response.Content.ReadAsStringAsync();
                            response.EnsureSuccessStatusCode();
                        }
                        CloudBlockBlob cloudBlockBlob = cloudBlobContainer.GetBlockBlobReference($"{DataPulledDate}/{BlobFolderName}/ideaId-{id}.json");
                        await cloudBlobContainer.CreateIfNotExistsAsync();
                        await cloudBlockBlob.UploadTextAsync(content);
                        Interlocked.Increment(ref processedItems);
                    }));
                }
            }
            await Task.WhenAll(tasks);
            sw.Stop();
        }
        catch (Exception ex)
        {
            log.LogError(ex, "{RunId}: Run failed. {Items} items processed successfully, Exception: {Exception}.", runId, processedItems, ex.ToString());
            throw;
        }
        finally
        {
            if (telemetryClient != null)
            {
                telemetryClient.Flush();
                Thread.Sleep(3000);
            }
        }
    }
}

Code of Function itself:

namespace FunctionApp
{
    public class GetIdeas : BaseFunc
    {
        public GetIdeas(TelemetryClient telemetryClient)
        {
            this.telemetryClient = telemetryClient;
        }

        protected override string BlobFolderName { get => "folder"; }
        protected override string GetItemUrl { get => "url"; }

        [FunctionName("GetIdeasFn")]
        public async Task Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)] HttpRequest req, ILogger log)
        {
            await ProcessRun(log, $"GetIdeasFn - {DateTime.UtcNow.Ticks}");
        }
    }
}

Appreciate any help.

Answer 1

we had the same issue but in our case, we had a code patch having an infinite while loop that is creating hundreds of requests to Microsoft GraphApi and without responding to one request it was creating another. we correct it and the issue fixed! Maybe that help someone