Synchronizing several threads writing to the same file in java

Question

Of course there is the obvious way of using "synchronized". But I'm creating a system designed for running on several cores and writing to that file various times at the same milisecond. So I believe that using synchronize will hurt performance badly. I was thinking of using the Pipe class of java (but not sure if it will help) or having each thread write to a different file and an additional thread collecting those writings, creating the final result. I should mention that the order of the writings isn't important and it is timestamped in nanotime anyway. What is the better idea of those two? have any other suggestions? thanks.

Answer 1

Using some sort of synchronization (eg. single mutexes) is quite easy to implement.

If I had enough RAM I would create a queue of some sort for each log-producer thread , and a log-consumer thread to read from all the queues in a round-robin fashion (or something like that).

Answer 2

Not a direct answer to your question, but the logback project has synchronization facilities built into it for writing to the same file from different threads, so you might try to use it if it suits your needs, or at least take a look at it's source code. Since it's built for speed, I'm pretty sure the implementation isn't to be taken for granted.

Answer 3

You are right to be concerned, you are not going to be able to have all the threads write to the same file without a performance problem.

When I had this problem (writing my own logging, way back before Log4j) I created two fixed-size buffers in memory and had all the producer threads write to one buffer while a dedicated consumer thread read from the other buffer and wrote to a file. That way the writer threads had to synchronize only on getting and incrementing the index to the buffer and when the buffers were being swapped, and it only blocked when the current buffer was full. It was memory-intensive but fast.

For other ideas you could check out how loggers like Log4j and Logback work, they would have had to solve this problem.

Answer 4

Try to use JMS. All your processes running on different machines may send JMS message instead of writing to file. Create only one queue receiver that receives messages and writes them to file. Log4J already has this functionality: see JMSAppender.