How to curl all files recursively

Question

Here's the code I'm currently using:

xargs -n 1 curl -s -o /dev/null -w "%{http_code} - %{url_effective}\n" < 'file.txt'

This works for curling all the URLs that are in file.txt and getting the status code and the URL that was curled. However, I need to do this for every file recursively.

I've tried a few things, like this, but none of them worked:

xargs -n 1 curl -s -o /dev/null -w "%{http_code} - %{url_effective}\n" < *

There's about 3000 files in multiple directories. Is there any way to do this recursively?

Answer 1

Assume that your file list's name has a rule like this: file.txt, file1.txt, file2.txt under the current working directory including the subdirectories.

You could merge all the links in those files with cat **/file*.txt command, then combine it with the xargs :

cat **/file*.txt | xargs -n 1 curl -s -o /dev/null -w "%{http_code} - %{url_effective}\n"

Good luck!