Playframework unusual CPU load

Question

Recently we started using PlayFramework and seeing some unusual activity in CPU load.

Machine details and other configurations:

32G Machine
12  Cores
PlayFramework 2.2.0
java -Xms1024m -Xmx1024m -XX:MaxPermSize=256m -XX:ReservedCodeCacheSize=128m
java applications are running within a docker container(Docker version 0.8.0).

There are 6 play server running behind nginx

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
31752 root      20   0 7876m 1.2g  14m S  716  3.8 150:55.28 java
26282 root      20   0 7862m 1.2g  14m S   48  3.8 310:51.65 java
56449 root      20   0 7789m 389m  13m S    2  1.2   0:33.10 java
40006 root      20   0 7863m 1.2g  14m S    2  3.8  17:56.41 java
42896 root      20   0 7830m 1.2g  14m S    1  3.8  15:10.30 java
52119 root      20   0 7792m 1.2g  14m S    1  3.7   8:48.38 java

The request rate is at max 100Req/s.

Has anyone faced this similar issues before? Please let me know.

Answer 1

I am experiencing a similar issue. However, I am only having trouble with the dev setup on my machine:

• play! application started via play run
• within a docker container
• that is running inside a VM

-> 100% CPU load inside the VM

Problem does not exist in our live environment, where the application is run pre-packaged via the start script. We also have it running in a docker container there (which in turn runs in an EC2 VM).

So it seems like in my case play run makes a difference - are you using play run to start your app inside the docker container?

EDIT: Also, there seems to be a bug in Netty related to epoll selectors : http://www.electrotank.com/forums/showthread.php?12927-Well-Known-JDK-bug-in-Netty https://code.google.com/p/spymemcached/issues/detail?id=279

The second link describes a workaround that involves setting the -XX:CMSInitiatingOccupancyFraction JVM parameter to 80 or something.

In my case, this did not help however...

You might consider profiling your java process eg using

sudo strace -cf -p <pid>

and see if anything suspicious shows up.

Answer 2

How are you running the play app? I have experienced a similar high load when running with play ~run , because it watches files for changes constantly.