Submit a new script after all parallel jobs in R have completed

Question

I have an R script that creates multiple scripts and submits these simultaneously to a computer cluster, and after all of the multiple scripts have completed and the output has been written in the respective folders, I would like to automatically launch another R script that works on these outputs.

I haven't been able to figure out whether there is a way to do this in R: the function 'wait' is not what I want since the scripts are submitted as different jobs and each of them completes and writes its output file at different times, but I actually want to run the subsequent script after all of the outputs appear.

One way I thought of is to count the files that have been created and, if the correct number of output files are there, then submit the next script. However to do this I guess I would have to have a script opened that checks for the presence of the files every now and then, and I am not sure if this is a good idea since it probably takes a day or more before the completion of the first scripts.

Can you please help me find a solution?

Thank you very much for your help -fra

Answer 1

I think you are looking at this the wrong way:

• Not an R problem at all, R happens to be the client of your batch job.

• This is an issue that queue / batch processors can address on your cluster.

• Worst case you could just wait/sleep in a shell (or R script) til a 'final condition reached' file has been touched

• Inter-dependencies can be expressed with make too