Download multiple files in parallel with wget

Question

I want to check a URL list and if the file (.pdf) exists, download it. I use this command and it's working;

for i in `cat url.txt`; do if wget -q --method=HEAD $i; then wget $i; fi ; done

But this is really slow, I have a lot of URLs to check. Is there a way to "multithread" this command?

Answer 1

Use perl parallel

eg

for i in `cat url.txt`; do if wget -q --method=HEAD $i; then 
   echo "wget '$i'"; fi ; done \
   | parallel