Anti-Fogery token cannot be decrypted when deployed behind a load balancer

Question

I've setup AntiForgery tokens in my Asp.Net Core 1.0 site as follows:

  services.AddAntiforgery(options =>
        {
            options.HeaderName = "X-XSRF-Token"; //Angular's default header name for sending the xsrf token
        });

---

app.Use(next => context =>
        {
            //2 tokens are generated.  A Cookie token, which goes in an ASP cookie called something like .AspNetCore.Antiforgery.****** and a request token.  The request token gets passed in the http request
            //headers back to the server when executing a POST and is validated agains't the cookie token.  By taking the request token and putting it in an HTTP-Only cookie called XSRF-TOKEN, Angular will 
            //automatically take the cookie and add it to an X-XSRF-Token header for each request.   The request token received in the header is then validated against the cookie token. In ConfigureServices() above, 
            //ASP's antiforgery system is setup to look for the request token in the X-XSRF-Token header.  For form posts (i.e. not posts done by Angular), the form has a token associated with it that gets passed in the POST
            // which is used in place of the header token  
            var tokens = antiforgery.GetAndStoreTokens(context);
            context.Response.Cookies.Append("XSRF-TOKEN", tokens.RequestToken, new CookieOptions() { HttpOnly = false });

            return next(context);
        });

        app.UseAntiForgeryValidation();

---

public class AntiForgeryValidation
{
    private readonly RequestDelegate _next;
    IAntiforgery _antiforgery;
    private ILogger<AntiForgeryValidation> _logger;

    public AntiForgeryValidation(RequestDelegate next, IAntiforgery antiforgery, ILoggerFactory loggerFactory)
    {
        _next = next;
        _antiforgery = antiforgery;
        _logger = loggerFactory.CreateLogger<AntiForgeryValidation>();
    }

    public async Task Invoke(HttpContext context)
    {
        //don't need to validate anti-forgery tokens for GET
        if (string.Equals("POST", context.Request.Method, StringComparison.OrdinalIgnoreCase)
            || string.Equals("PUT", context.Request.Method, StringComparison.OrdinalIgnoreCase)
            || string.Equals("DELETE", context.Request.Method, StringComparison.OrdinalIgnoreCase))
        {

            await _antiforgery.ValidateRequestAsync(context);
        }

        await _next(context);
    }
}

public static partial class MiddlewareExtensionMethods
{
    public static void UseAntiForgeryValidation(this IApplicationBuilder builder)
    {
        builder.UseMiddleware<AntiForgeryValidation>();
    }
}

In my dev environment, and when published to IIS, this works well. There is a load balancer in front of my site though and when I access it through the VIP provided by the load balancer my anti-forgery validation starts failing with the following errors:

{7f02de42-e781-4c27-a2e5-fce932f4b7a4} was not found in the key ring. Unprotect operation cannot proceed

and

An unhandled exception was thrown by the application.The antiforgery token could not be decrypted.

(more of the log is below).

I should also point out that even though there's a load balancer, there is only 1 web server behind it so there shouldn't be any issue with the requests being split between servers. If anyone has any idea why this would be occurring when accessed through the load balancer and not when accessed directly (via a hosts entry to bypass the load balancer) I would love some assistance.

Log 2016-11-01 15:28:01.3761|22|TRACE|Microsoft.Extensions.Logging.LoggingExtensions.PerformingUnprotectOperationToKeyWithPurposes|Performing unprotect operation to key {4441518f-1b1d-49a8-b06a-29724cb692ae} with purposes ('C:\\temp\\wizard', 'Microsoft.AspNetCore.Antiforgery.AntiforgeryToken.v1'). 2016-11-01 15:28:01.4074|22|TRACE|Microsoft.Extensions.Logging.LoggingExtensions.PerformingProtectOperationToKeyWithPurposes|Performing protect operation to key {4441518f-1b1d-49a8-b06a-29724cb692ae} with purposes ('C:\\temp\\wizard', 'Microsoft.AspNetCore.Antiforgery.AntiforgeryToken.v1'). 2016-11-01 15:28:01.4074|22|DEBUG|Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgery.GetAndStoreTokens|An antiforgery cookie token was reused. 2016-11-01 15:28:01.4230|22|TRACE|Microsoft.Extensions.Logging.LoggingExtensions.PerformingUnprotectOperationToKeyWithPurposes|Performing unprotect operation to key {7f02de42-e781-4c27-a2e5-fce932f4b7a4} with purposes ('C:\\temp\\wizard', 'Microsoft.AspNetCore.Antiforgery.AntiforgeryToken.v1'). 2016-11-01 15:28:01.4230|22|TRACE|Microsoft.Extensions.Logging.LoggingExtensions.KeyWasNotFoundInTheKeyRingUnprotectOperationCannotProceed|Key {7f02de42-e781-4c27-a2e5-fce932f4b7a4} was not found in the key ring. Unprotect operation cannot proceed. 2016-11-01 15:28:01.4230|22|ERROR|Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame`1.RequestProcessingAsync|Connection id "0HL02KLA3LKUJ": An unhandled exception was thrown by the application.The antiforgery token could not be decrypted. 2016-11-01 15:28:01.4230|22|INFO|Microsoft.AspNetCore.Hosting.Internal.HostingLoggerExtensions.RequestFinished|Request finished in 44.7391ms 200 2016-11-01 15:28:01.4230|22|DEBUG|Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Connection.Microsoft.AspNetCore.Server.Kestrel.Internal.Http.IConnectionControl.End|Connection id "0HL02KLA3LKUJ" completed keep alive response.

Update

I've added

 services.AddDataProtection()
            .PersistKeysToFileSystem(new DirectoryInfo(@"c:\temp\"));

and a key is persisted to c:\\temp\\ but I've still having the issue. The updated log is below. The key: {f5087f37-37e3-4e52-b40b-77e32a285f3e} is written to c:\\temp and is selected as the default key, however there's still some mysterious key {4441518f-1b1d-49a8-b06a-29724cb692ae} that the app is trying to use to decrypt the anti-fogery token which can't be found.

2016-11-02 07:19:28.7357|1|DEBUG|Microsoft.AspNetCore.Hosting.Internal.HostingLoggerExtensions.Starting|Hosting starting 2016-11-02 07:19:28.8450|1|DEBUG|Microsoft.AspNetCore.Hosting.Internal.HostingLoggerExtensions.Started|Hosting started 2016-11-02 07:19:29.0326|7|DEBUG|Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Connection.Start|Connection id "0HL035D0UCHEL" started. 2016-11-02 07:19:29.1107|3|INFO|Microsoft.AspNetCore.Hosting.Internal.HostingLoggerExtensions.RequestStarting|Request starting HTTP/1.1 POST http://apidev.brewster.ca/ application/x-www-form-urlencoded 218 2016-11-02 07:19:29.1263|3|DEBUG|Microsoft.AspNetCore.HttpOverrides.ForwardedHeadersMiddleware.ApplyForwarders|Parameter count mismatch between X-Forwarded-For and X-Forwarded-Proto. 2016-11-02 07:19:29.1732|3|TRACE|Microsoft.Extensions.Logging.LoggingExtensions.PerformingUnprotectOperationToKeyWithPurposes|Performing unprotect operation to key {4441518f-1b1d-49a8-b06a-29724cb692ae} with purposes ('C:\\webpublish\\wizard', 'Microsoft.AspNetCore.Antiforgery.AntiforgeryToken.v1'). 2016-11-02 07:19:29.1888|3|DEBUG|Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository.ReadElementFromFile|Reading data from file 'c:\\temp\\key-f5087f37-37e3-4e52-b40b-77e32a285f3e.xml'. 2016-11-02 07:19:29.1888|3|DEBUG|Microsoft.Extensions.Logging.LoggingExtensions.FoundKey|Found key {f5087f37-37e3-4e52-b40b-77e32a285f3e}. 2016-11-02 07:19:29.2200|3|DEBUG|Microsoft.Extensions.Logging.LoggingExtensions.ConsideringKeyWithExpirationDateAsDefaultKey|Considering key {f5087f37-37e3-4e52-b40b-77e32a285f3e} with expiration date 2017-01-31 14:11:02Z as default key. 2016-11-02 07:19:29.2357|3|DEBUG|Microsoft.Extensions.Logging.LoggingExtensions.OpeningCNGAlgorithmFromProviderWithChainingModeCBC|Opening CNG algorithm 'AES' from provider '' with chaining mode CBC. 2016-11-02 07:19:29.2513|3|DEBUG|Microsoft.Extensions.Logging.LoggingExtensions.OpeningCNGAlgorithmFromProviderWithHMAC|Opening CNG algorithm 'SHA256' from provider '' with HMAC. 2016-11-02 07:19:29.2670|3|DEBUG|Microsoft.Extensions.Logging.LoggingExtensions.UsingKeyAsDefaultKey|Using key {f5087f37-37e3-4e52-b40b-77e32a285f3e} as the default key. 2016-11-02 07:19:29.2670|3|TRACE|Microsoft.Extensions.Logging.LoggingExtensions.KeyWasNotFoundInTheKeyRingUnprotectOperationCannotProceed|Key {4441518f-1b1d-49a8-b06a-29724cb692ae} was not found in the key ring. Unprotect operation cannot proceed. 2016-11-02 07:19:29.2670|3|ERROR|Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgery.GetCookieTokenDoesNotThrow|An exception was thrown while deserializing the token.The antiforgery token could not be decrypted. 2016-11-02 07:19:29.2982|3|TRACE|Microsoft.Extensions.Logging.LoggingExtensions.PerformingProtectOperationToKeyWithPurposes|Performing protect operation to key {f5087f37-37e3-4e52-b40b-77e32a285f3e} with purposes ('C:\\webpublish\\wizard', 'Microsoft.AspNetCore.Antiforgery.AntiforgeryToken.v1'). 2016-11-02 07:19:29.2982|3|TRACE|Microsoft.Extensions.Logging.LoggingExtensions.PerformingProtectOperationToKeyWithPurposes|Performing protect operation to key {f5087f37-37e3-4e52-b40b-77e32a285f3e} with purposes ('C:\\webpublish\\wizard', 'Microsoft.AspNetCore.Antiforgery.AntiforgeryToken.v1'). 2016-11-02 07:19:29.3138|3|DEBUG|Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgery.GetAndStoreTokens|A new antiforgery cookie token was created. 2016-11-02 07:19:29.3296|3|DEBUG|VTR.Common.Utilities.Middleware.AntiForgeryValidation.Invoke|AntiForgeryCookie: [.AspNetCore.Antiforgery.2mRY6wjt_Lw, CfDJ8I9RQUQdG6hJsGopcky2kq4ayXKUx4xErDElnyDij0J31qSLHyt3oKyqJ1ocoRHYoIkK7WSpze9SVEzOan0LQTFs3SwwtvMUw_e6EUPvaPxWjH_1_pQ5DiT8hu7TM8UNWjjFT_XSkNZz-uBVHdh2CmY] 2016-11-02 07:19:29.3296|3|DEBUG|VTR.Common.Utilities.Middleware.AntiForgeryValidation.Invoke|XSRF cookie: CfDJ8I9RQUQdG6hJsGopcky2kq45_O5j5bzBP5QyxbzIKXaSbb8K04mez2Czsa_OdCYn84bvSz2v8M-nkN_O6yorN8qfyy4mV3HdGJV3BQgWSHSFrziJfQonBKmiF4fsrmpHVX7jA6iugirWL8yNJPO35xY 2016-11-02 07:19:29.3607|3|DEBUG|VTR.Common.Utilities.Middleware.AntiForgeryValidation.Invoke|Form Token: CfDJ8ELeAn-B5ydMouX86TL0t6R3a1YFIaw2K3Wf6xpsbQPGztGb2uACGCdnofeFKd-woGFkwTjWQEC4Qag1E2A2RCxtvry4DJAvPSZnbKZGMd248exLwt9CX1bhWsYj8Cs6N9MvEQ38gqEXmEArDXFYMM0 2016-11-02 07:19:29.3607|3|DEBUG|VTR.Common.Utilities.Middleware.AntiForgeryValidation.Invoke|Header Token: 2016-11-02 07:19:29.3768|3|TRACE|Microsoft.Extensions.Loggi ng.LoggingExtensions.PerformingUnprotectOperationToKeyWithPurposes|Performing unprotect operation to key {7f02de42-e781-4c27-a2e5-fce932f4b7a4} with purposes ('C:\\webpublish\\wizard', 'Microsoft.AspNetCore.Antiforgery.AntiforgeryToken.v1'). 2016-11-02 07:19:29.3768|3|TRACE|Microsoft.Extensions.Logging.LoggingExtensions.KeyWasNotFoundInTheKeyRingUnprotectOperationCannotProceed|Key {7f02de42-e781-4c27-a2e5-fce932f4b7a4} was not found in the key ring. Unprotect operation cannot proceed. 2016-11-02 07:19:29.3768|3|ERROR|Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame`1.RequestProcessingAsync|Connection id "0HL035D0UCHEL": An unhandled exception was thrown by the application.The antiforgery token could not be decrypted. 2016-11-02 07:19:29.3768|3|INFO|Microsoft.AspNetCore.Hosting.Internal.HostingLoggerExtensions.RequestFinished|Request finished in 286.4265ms 200 2016-11-02 07:19:29.4233|3|DEBUG|Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Connection.Microsoft.AspNetCore.Server.Kestrel.Internal.Http.IConnectionControl.End|Connection id "0HL035D0UCHEL" completed keep alive response.

Answer 1

The issue was caused by a caching problem on the load balancer. Once the cache was flushed the missing key problem went away