JasyptStringDigester with SHA2 becomes very slow suddenly

Question

In my web application, login passwords are hashed and saved with JasyptStringDigester with SHA256. During login, password input by user will be hashed with same digester for comparsion.

However, after the application runs about 2 days, login becomes very slow suddenly. Once it happens, I have to restart the server to recover.

With Thread dump, I found out that the slowdown is caused by the digester and it uses up CPU resources. I have tried to change JCE provider from default one to bouncycastle but it didn't help.

I have also checked the momery usage in JVM when this problem occurs, but there are plenty of them.

Environment:

JDK 7u60

JBoss 7.1.1 Final

Digester configuration(used as singleton):

 <bean id="jasyptStringDigester" class="org.jasypt.digest.StandardStringDigester"> <property name="provider" ref="bouncyCastleProvider" /> <property name="algorithm" value="SHA-256" /> <property name="iterations" value="100000" /> <property name="saltGenerator"> <bean id="zeroSaltGenerator" class="org.jasypt.salt.ZeroSaltGenerator"/> </property> <property name="saltSizeBytes" value="10"/> </bean> <bean id="bouncyCastleProvider" class="org.bouncycastle.jce.provider.BouncyCastleProvider"/> 

Thread dump:

  "ajp--10.88.90.34-8009-22" daemon prio=10 tid=0x00007ff2100ad800 nid=0xc7e runnable [0x00007ff1a9ae4000] java.lang.Thread.State: RUNNABLE at org.bouncycastle.crypto.digests.SHA256Digest.Sum0(Unknown Source) at org.bouncycastle.crypto.digests.SHA256Digest.processBlock(Unknown Source) at org.bouncycastle.crypto.digests.GeneralDigest.finish(Unknown Source) at org.bouncycastle.crypto.digests.SHA256Digest.doFinal(Unknown Source) at org.bouncycastle.jcajce.provider.digest.BCMessageDigest.engineDigest(Unknown Source) at java.security.MessageDigest.digest(MessageDigest.java:353) at java.security.MessageDigest.digest(MessageDigest.java:399) at org.jasypt.digest.StandardByteDigester.digest(StandardByteDigester.java:979) - locked <0x0000000748e4a9c0> (a org.bouncycastle.jcajce.provider.digest.SHA256$Digest) at org.jasypt.digest.StandardByteDigester.digest(StandardByteDigester.java:933) 

Would anyone help please? I have been stuck into this problem for a long time. A similar issue was found in https://bugs.openjdk.java.net/browse/JDK-8023983 but I couldn't find any solution.

Thanks.

Answer 1

I had exactly the same problem, entropy is not the cause. SHA256 digest doesn't need random. The problem is JIT (Just in time) compilation mechanism that allow to compile some Hotspot function directly in native code. There is a known issue in JDK7 see : http://www.oracle.com/technetwork/java/javase/documentation/javase7supportreleasenotes-1601161.html that disable the native compiler when code cache is full. In this case SHA digest is not executed natively and become very long !

To reproduce just disable the compiler :

-Djava.compiler=NONE 

The solution is to migrate to Java 8, a workaround for Java 7 is to increase the CodeCache size according to your needs by using the JVM option.

-XX:ReservedCodeCacheSize=300m

You can monitor your code cache through JConsole :

I hope it will help :)

Eric

Answer 2

Which OS?

If it is Linux, once it slows down take a look at the available entropy ( cat /proc/sys/kernel/random/entropy_avail ). If that number is less than 250 or so, you need more randomness on the system. It is a common issue on servers. Network traffic or disk IO can help.

To make it more difficult to diagnose, you might actually get a brief speed up as you log in. That is because the log in itself generates some network traffic which helps add some randomness to the entropy pool. On the other hand, SSHing into the box may be painfully slow as it needs some randomness for the SSH session keys. In either case (speed up after log in or slow log in), entropy is a likely issue.

You can consider adding some additional sources of randomness, such as haveged or rng-tools .

You may see suggestions to copy some "randomness" from /dev/urandom to /dev/random , which will appear to fix the problem. But /dev/urandom does not actually have any randomness, it is just a PRNG seeded with randomness. So this just avoids the issue by compromising the security of the crypto you are using. Just say no.