SSL connection encryption negotiation

Question

I've installed a .Net-based website on multiple virtual machines (Windows 2012 R2), using the same version of the same installer on both. Part of this installation includes generation of a self-signed ssl certificate. When accessing the two websites over https, one is indicating "secure connection failed" when using Firefox, however the other server/website works fine. Both load in IE fine.

When looking at the page properties (from IE), the failing site indicates a connection of "TLS 1.0, AES with 128 bit encryption (High); RSA with 2048 bit exchange", while the successful one indicates "TLS 1.2, AES with 256 bit encryption (High); ECDH_P256 with 256 bit exchange".

What server settings should I be looking at to determine the difference between the two?

Answer 1

I tracked this down to an incompatibility between the self-signed certificate, which was using SHA512, and TLS. Changing the certificate to use SHA256 resolved the issue. The reason for the failure on one VM vs another was a registry setting, which dictated whether or not to support SHA512. The succeeding machine supported 512, the failing one did not.

https://blogs.technet.microsoft.com/silvana/2014/03/14/schannel-errors-on-scom-agent/