Azure Table Storage query failed with authentication error (Received:Forbidden) 不时出现

Question

Environment: Azure app service.

Azure storage SDK: WindowsAzure.Storage (9.3.3)

Invocation (pseudo code):

void QueryAzureTable(){
    while(true){
        var tableClient = new AzureTable();
        var resp = tableClient.Query('table','pk','rk');
        // ...
    }
}

var tasks = new List<Task>();
for (var i = 0; i < 5; i++)
{
    tasks.Add(QueryAzureTable());
}
await Task.WhenAll(tasks).ConfigureAwait(false);

Authorization method for QueryAzureTable: tried with both clientId/secret and managed identity/MSI, same result for both.

Observation:

1. Around half of the requests failed due to anth issue in QueryAzureTable() (see detailed error msg below).
2. If I restart the azure app service instance, the auth error will be gone for ~12 hours.

Error Message:

1. Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
2. Unexpected response code, Expected:OK or NotFound, Received:Forbidden

I have checked and tried with almost every solutions mentioned in this stackoverflow thread , but no luck. Guess this specific auth issue might be related with multi-tasks.

Answer 1

Kind of figured out the solution on my own: adding a retry logic to renew the token .

void query(...){
    int cnt=0;
    while(true){
        try{
            _client.queryTable(...);
        }
        catch(AuthException ex){
            log.error(ex...);
            var token=new Token(...);
            _client = new AzureTableClient(token);
            cnt++;
            if(cnt==3) throw;
        }
    }
}

The first clue to this solution was whenever there was a app service release, deployment or restart of the app service, the query table function worked well for a while, and then after around 12 hours, errors started showing up. But not 100% failure rate.

If there is any explanation or conclusion that helps to root cause this, please share your opinions. Thanks in advance: My blind guess is that it has something to do with muti-tasks. WindowsAzure.Storage (9.3.3) does not do a good job of renewing token for muti-tasks.

Hope this could help you.