WGET - Download specific files (by extension or mime-type) from third party websites

Question

I need to get ALL ".js" extension files from a website by using wget, including third party ones, but it's not always being done.

I use the next code:

wget -H -p -A "*.js" -e robots=off --no-check-certificate https://www.quantcast.com

For example, if I execute wget to " https://www.stackoverflow.com " I want to get all "*.js" files from stackoverflow.com but also third party websites, such as "scorecardresearch.com", "secure.quantserve.com" and others.

Is something missing in my code?

Thanks in advance!

Answer 1

Wget with the -p flag will only download simple page requirements like scripts with a src , links with an href or images with a src .

Third party scripts are often loaded dynamically using script snippets (such as Google Tag Manager https://developers.google.com/tag-manager/quickstart ). These dynamically loaded script will not be downloaded by Wget, since they need to run the JavaScript to actually load. To get absolutely everything, you would likely need something like Pupeteer or Selenium to load the page and scrape the contents.