Retreive output files in HTCondor as they are created

Question

I am using HTCondor to run a job that takes a lot of time (tens of hours) and produces files periodically (tens of minutes). I want HTCondor to transfer the files to my submit directory as soon as the files are created. How can I accomplish this?

Is there any option native for condor to allow me to do this? Is there an alternative method?

I could also try to scp or mv the files as they are created but I don't know how to transfer files manually from condor to my personal space

Answer 1

The condor_tail command can do this, from the submit side, or the condor_chirp command can do it from the job itself.

For example, if you are logging into the submit machine (the same machine you submitted the job from, in all likelyhood), and you are curious about the state of any output files for a job that is still running, you can look at the last 1024 bytes of the file by running

$ condor_tail <the_job_id> the_file_name

To see more than 1024 bytes, you could run

$ condor_tail <the_job_id> -maxbytes some_number_more_than_1024 the_file_name

Or, if you are on Linux, and your system admininstrator allows you, you may be able to run

$ condor_ssh_to_job <the_job_id>

which gives you a login shell to the machine the job is running on, in the scratch directory of the job, with as much of the job's environment as condor can give it.

Finally, if the job itself can run a command, say periodically, after every completed step, and the job has enabled remoteio with the submit file option

+WantRemoteIO = true

the job could run

$ condor_chirp put local_file_name file_name_on_submit_machine

and the file name local_file_name from the job will be copied to the submit machine into the directory the job started from.