Serialising takes too long with multiple process trying to write at same time

Question

We are facing a performance issue while serialising file by multiple processes.

Here is the issue: We are creating multiple processes (in distributed computing env) to do our calculation and then write down output of each processes in a file. Main process use these files, merge it in memory and do the further calculation.

We have limited no of servers. What happens is 2-3 processes can be created in same server. When this happens (2-3 processes created in same server), we are having a scenario when those processes try to write\\serialize computed file (file size is approx 80-90 MB) on the disk at the same time. When this happens, it takes around 3m to serialise the file. In normal case, that 80-90 MB takes only 30s.

We monitored this scenario in performance monitor (and our log timings) and could see because of 2-3 processes trying to write at the same time, it takes around 6 times longer than normal timing.

Any suggestion to improve timings (of 3mins scenario) is appreciated.

We use .Net framework and code is written in c#.

Answer 1

You can try forcing the processes to write into different files and then just read all files in a folder. For example you can have the following structure

|-C:\experiments\current
|--- output_{UNIQUE_SUFFIX}.bin 
|--- output_0.bin
|--- output_1.bin
|--- output_nwvpqnfj.bin
|--- output_jhfjqhfew.bin