X509Certificate constructor takes >6 seconds to execute for a particular user

Question

I've developed a C#, .NET4.5.2 client/server system which uses TLS/SSL to communicate. The certificates are loaded from a file. I created the certificate files using 'MakeCert' utitilty to create a .pvk and .cer file then I combined them into a .pfx using 'pvk2pfx' utility.

To use the certificates I load them by using the X509Certificate2 constructor with the overload to pass in the file path and password as strings:

X509Certificate2(string filePath, string password)

I've noticed over time that the loading of certificates has become very slow. I'm not sure if there was an 'event' that made them slow or if it has been gradual but it's now up to about 6 seconds to load the PFX file. Loading the CER file is no problem and takes ~0.1 seconds.

I'm running Windows 8.1 and the problem is only for my user login on the laptop.

I wrote the following test app to verify the problem:

    private const string filePath = @"c:\testcert.pfx";
    private const string password = "testpassword";
    static void Main(string[] args)
    {
        var stopwatch = new Stopwatch();
        try
        {
            Console.WriteLine("About to create certificate. Press Enter.");
            Console.ReadLine();

            stopwatch.Start();
            var cert = new X509Certificate(filePath, password);
            stopwatch.Stop();

            Console.WriteLine(stopwatch.Elapsed);
            Console.WriteLine("Certificate created. About to reset.  Press Enter.");
            Console.ReadLine();

            cert.Reset();

            Console.WriteLine("Certificate reset.  Press Enter.");
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
        }
        Console.ReadLine();
    }

I asked some colleagues to run the program on their PCs. I also tried running within a VM and then setting up a new user on my own laptop. In all cases it ran in ~0.1 seconds but for my normal user login it runs in >6 seconds.

At first I wasn't calling 'Reset()' on the certificate so thought perhaps there was a problem with some temporary files somewhere so I used procmon to find out what was happening. I identified that some temporary files were being created in the following directory (although they got tidied up when the application exited even without the call to Reset()):

C:\Users\<username>\AppData\Roaming\Microsoft\Crypto\RSA\<SID>

Just to be sure I've tried deleting the files in this directory but it doesn't make any difference.

Using procmon I can see that there are 2 gaps in file/registry activity during the loading of the certificate which don't occur in a system where the load is fast. First is just after it tries to use 'dpapi.dll'. The second is after a read to the following 'C:\\Extend\\$UsnJrnl:$J:$DATA'. DPAPI.dll is the interface for Windows Data Protection. The latter file is the USN Journal for NTFS which records file changes. I'm no expert on either and I'm not sure if either is relevant!

I've then tried using API Monitor http://www.rohitab.com/apimonitor to observe system calls. Again I'm no expert but I trawled through to see what was happening just before the pauses. There's a lot in there I don't understand that may or may not be relevant and I welcome comment on any of it to help focus in on the problem.

The last call before a 2 second gap is a memcpy with the following call stack:

#   Module  Address Offset  Location
1   RPCRT4.dll  0x74f6378b  0x2378b I_RpcSendReceive + 0x1bb
2   RPCRT4.dll  0x74f6367b  0x2367b I_RpcSendReceive + 0xab
3   RPCRT4.dll  0x74f594df  0x194df NdrServerInitializeNew + 0x83f
4   RPCRT4.dll  0x74f63619  0x23619 I_RpcSendReceive + 0x49
5   RPCRT4.dll  0x74f6398b  0x2398b NdrSendReceive + 0x2b

Higher up the potentially interesting lines seem to be:

#   Time of Day Thread  Module  API Return Value    Error   Duration
64922   6:38:44.348 AM  1   DPAPI.dll   SystemFunction040 ( 0x00ac5a30, 8, RTL_ENCRYPT_OPTION_SAME_LOGON )  STATUS_SUCCESS      0.0000402
64923   6:38:44.349 AM  1   CRYPTBASE.dll   RtlInitUnicodeString ( 0x0090e5a8, "\Device\KsecDD" )           0.0000004

64949   6:38:44.349 AM  1   RPCRT4.dll  RtlInitUnicodeString ( 0x0090e0b0, "\RPC Control\protected_storage" )           0.0000000

I find it difficult to trace the callstack but I think these are ultimately coming from a function called CryptQueryObject.

I've found the following article which may be relevant but it's not Windows8.1. I've deleted the %windir%\\Temp folder just in case but also hasn't helped.

https://support.microsoft.com/en-gb/kb/931908

I remember finding an article somewhere that suggested a delay might be something to do with ActiveDirectory calls from CryptQueryObject but I can't find the link.

I'm really looking for:

1. How to fix my user login so it doesn't take 6 seconds to load a certificate
2. How to ensure my code is OK so it doesn't happen again or to other people using the system

Thanks for any help.

Answer 1

This issue has now resolved and I think I understand why!

When you load an X509Certificate with a private key, Windows stores the key to a file and encrypts the file using 'Data Protection' (hence the calls to DPAPI I had monitored).

https://technet.microsoft.com/en-us/library/cc962112.aspx

The key used to encrypt the file is called a 'Master Key'. It is based on your user login and gets renewed whenever you change your use password. It also automatically expires after 90 days.

http://www.passcape.com/index.php?section=docsys&cmd=details&id=28#33

As described in the above link, the master key is stored in the following directory:

%APPDATA%/Microsoft/Protect/%SID%

I could see that my MasterKey was in fact older than 90 days by looking at the 'last modified' date on the files in that directory. So, whenever I was loading a certificate, it was trying to store the private key in an encrypted file and noticed that the Master Key was out of date so it would then try and update it.

I reran ProcMon but stopped filtering for activity just from my test app. I could then see that a number of UDP calls to our Active Directory server were being made during the 'gaps' by a process called lsass.exe which is to do with security and password updates (can't post another link due to stack overflow restrictions but easy to search for).

I think that the process of trying to update the master key must involve some interaction to the ActiveDirectory service.

I work remotely and connect to my work network via VPN. The VPN does not allow access to the ActiveDirectory server so I think the delays were caused by trying and failing to access that server.

I waited till my next visit to the office and as soon as I was connected to the network and could access the ActiveDirectory server then the delay in loading certificates disappeared. I confirmed that my MasterKey files have also been updated and are no longer out of date.

I retested loading certificates while connected via the VPN and the delay is still gone so this does seem to confirm that it is OK as long as the Master Key isn't out of date.

Unfortunately, I couldn't change my password over the VPN due to the same problems of not being able to access the Active Directory service.

My problem solved. I appreciate it was probably fairly unique but I hope the details of my investigation help somebody else at some point!