Can multiple programs write to STDOUT at the same time?

Question

I'm currently using GNU Parallel to run a Python script on multiple large files simultaneously. I have a master Python script that sets up the files I need to process, and then dispatches Parallel to run the same worker script on these files.

I need to get the data back into the master thread when all the workers have finished, and I'm currently just writing to files and pulling them back out. Is it possible for the workers to dump their pickled contents onto STDOUT for the master to collect, or is there a chance that two workers could start writing to STDOUT at the same time and interleave with each other?

Basically, will STDOUT block until another program has finished?

Answer 1

In a shell environment, starting various jobs in the background, all writing to stdout , has a high chance of interleaving that output, as there is no lock on stdout .

However, GNU Parallel can redirect stdout for the various jobs it starts and prevent this interleaving. There are several commmand line switches and various options.

By default output is grouped:

--group
Group output. Output from each jobs is grouped together and is only printed when the command is finished. stderr (standard error) first followed by stdout (standard output). This takes some CPU time. In rare situations GNU parallel takes up lots of CPU time and if it is acceptable that the outputs from different commands are mixed together, then disabling grouping with -u can speedup GNU parallel by a factor of 10.

--group is the default. Can be reversed with -u .

But other options, including directing to files, are available.