AuthenticationFailed Azure Table Storage

Question

We are getting a random error from table storage when accessing data:

System.Data.Services.Client.DataServiceClientException: <?xml version="1.0" encoding="utf-8" standalone="yes"?>
<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
 <code>AuthenticationFailed</code>
 <message xml:lang="en-US">Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:67cd9503-7a10-48a9-8c97-fee3906ac8cb
Time:2012-06-19T08:20:42.0670051Z</message>
</error>
  at System.Data.Services.Client.QueryResult.Execute()
  at System.Data.Services.Client.DataServiceRequest.Execute[TElement](DataServiceContext context, QueryComponents queryComponents)

Here are some facts about the error and our web application:

• We have 5 medium webservers hosting our site
• At any given time there are 200-500 visiors on our site. And they are constantly clicking around.
• Data is loaded from table storage on every click, it may be saved as well.
• The error only happens 20-50 times a day.

What puzzles me is the infrequent occurrence of this error compared to the massive amount of page loads and AJAX callbacks going on.

What is the cause of this error? We have read that there may be a time stamp issue if the server time is off but why would the time be wrong on our live server and why isn't the error happening constantly then?

Answer 1

Ok, I finally found out what was causing my problem. I had a bad combination of static and non static internal variables in my DataSource class:

private static CloudStorageAccount storageAccount;
private CharacterContext _context;

Since the storageAccount variable was static, concurrent requests would compete at changing it to different storage accounts when instantiating a DataSource object:

public CharacterDataSource()
{
      storageAccount = CloudStorageAccount.FromConfigurationSetting(ServiceConfigurationHelper.GetTableStorageConnectionStringConfigKey(GameInstanceName));
      _context = new CharacterContext(storageAccount.TableEndpoint.AbsoluteUri, storageAccount.Credentials);
}

Because of this little issue, the instantiation of the _context object could actually be using one storeage account when accessing storageAccount.TableEndpoint.AbsoluteUri and another when accessing storageAccount.Credentials. And this was causing the AuthenticationFailed error.

There was absolutely no reason for the storageAccount variable to be static so the solution was simply to remove this modifier and we have had no more errors since then.

The hint I got from Microsoft was that we were using the wrong credentials for the storage account, something we were unable to see in the stack trace we were getting. Don't know where the guy from Microsoft found the information but I'm glad i contacted them.

Answer 2

The server time being off is a common reason for this type of issues, but since your application is running in Azure and it doesn't happen that much I would exclude this.

Like you state in the question you have between 200 and 500 users on your site at any given time. This means you'll have thousands of transactions that work correctly, and only a very small number of errors. I have 2 suggestions:

1. Call Microsoft and provide them with your RequestId and time so that they can investigate the real cause.
2. Implement a retry policy. As you might know, Table Storage already comes with a retry policy . But in your case this won't be sufficient because this retry policy doesn't retry on 403 (and all other 400) error codes. That's why you could look at building a custom retry policy . People might not agree, but I would see this type of issue as a transient fault since it happens randomly on probably < 0,5% of all your transactions. And in that case I would handle it by using a retry policy.