HTCondor 分散聚集

Question

I'm developing a project where I need to reduce a video resolution from 1080p to 720p. That part is pretty straight forward with the command "ffmpeg -I inputVideo.mp4 -r 60 -s hd720 outputVideo.mp4".

Now, the idea is using HTCondor, so every Resource receives one part of the video ( Scatter ), reduces it quality, and returns it, so in the end all parts can be put together to make the full video 720p ( Gather ). The issue is, every node returns its own segment, so how can I make that the return is the total junction video. I thought of using a.dag file, so after the execution, it would run a script that would merge them, but, for what I understand from the documentation that is run on each node and not in a place where all segments are, which is the client, I guess.

So I would like to ask, what do you suggest so I can merge the output video files.

Answer 1

I think you are on the right track with dagman. Start from a diamond dag like in the docs:

https://htcondor.readthedocs.io/en/stable/users-manual/dagman-applications.html#the-dag-input-file-basic-commands

But maybe you want more than 2 nodes to do the processing -- the scattering. As each of these HTCondor jobs finishes, the output will be sent back to the submitting machine. You can either send all of these out to a HTCondor worker node for the final assembly, or, if the assembly is not a cpu-heavy activity, just run it on the submit machine where all the intermediate outputs are. In HTCondor,a "local universe" job looks just like a normal job, but runs on the submit machine. Perhaps the final node that depends on all the scatter nodes could be a local universe job.

At a training session for dagman, there was a similar image processing example with the mandlebrot set. This link may give you some good ideas:

https://opensciencegrid.org/user-school-2019/#materials/day4/part4-ex1-simple-dag/