System.err.println fails under threads pressure in Java

Question

I'm writing a program using java, where I use some error printing statements for debugging. My program generates about 2000 threads. The program runs fine till the moment when a large number of threads access this statement:

System.err.println("Some error message");

When this happens one of my threads successfully manages to get access to println function, while the other threads have status:

State in JVM: Waiting for synchronized block

Digging deeper in the debugging statement I noticed that the thread which managed to access println function is stopped at this function:

private native void writeBytes(byte b[], int off, int len , boolean append) throws IOException;

and it has the following stack trace:

java.io.FileOutputStream.write(FileOutputStream.java:327)

java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)

java.io.PrintStream.write(PrintStream.java:482)

sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)

sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)

sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)

java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)

java.io.PrintStream.write(PrintStream.java:527)

java.io.PrintStream.print(PrintStream.java:669)

java.io.PrintStream.println(PrintStream.java:806)

fetcher.responseHandler.ExtendedResponseHandler500.handleResponse(ExtendedResponseHandler500.java:20)

fetcher.FetchWorker.run(FetchWorker.java:79)

java.lang.Thread.run(Thread.java:745)

While the other threads are stopped at the first line of the println function (inside the java core code):

synchronized(this)

Is this problem caused by me? or is this error related to JVM? Can I do anything about this issue?

Answer 1

The most likely cause is that the output stream of the process isn't being consumed by the parent process, so the stdout buffer fills up and then the next call to System.err.println just hangs forever.

This is common when one process is used to launch another, but doesn't set up "flushing" threads to drain the child's stdout and stderr streams.

Note that this doesn't have anything in particular to do with "threading" - but launching many threads can certainly increase the rate at which errors are generated (and perhaps cause more total errors if something else fails due to contention downstream) which means your output buffer fills up faster and hangs earlier.

Answer 2

It is perfectly normal for 2000 threads to be waiting to acquire lock on the println call.

Your stack trace shows that you are getting some HTTP 500 errors. Probably the majority your threads got this error and now are all in line to report it on the standard error. What you are seeing is the consequence of your problem, not the cause.

2000 threads is an insane number, it will not improve performances for about any reasonable scenario and will most likely degrade performances. Start with something like 4 and see if incrementally doubling that value gives you any improvement. The JVM can handle this number of threads (so this is NOT the source of your problem) but it is just useless. Using more JVM will not fix the problem (that is probably a simple HTTP 500 and/or network timeouts). Also check the server side logs.

If you need to maximize performances (in a real high concurrency scenario) consider asynchio, but normal IO is extremely good for all common cases and seems fine here.

Update: My hypotesis is this:

1. a thread does a remote call
2. it gets a 500 error
3. it joins the line to report the error behind other 2000 threads
4. it succedes and goes back to point 1

The step 3 may take a lot of time so even taking multiple thread dumps you will see the very same thread that appears always locaked (I mean that you need to be very lucky to see the Thread-1352 during the steps 1 or 2). I'm assuming that you checked the thread name and that the locked thread we are discussing is always the same. Do you see any log while the program "freeze" (does it freezes, right?) or everything is still? How many thread dumps did you took, how much time in between?