wget，下载具有特定结尾的链接文件

Question

Want to download all wordlists from this site https://wiki.skullsecurity.org/Passwords

I tried

wget https://wiki.skullsecurity.org/Passwords --no-check-certificate --accept "*.bz2" -r

but does only download the targeted page..

What didn't work either:

wget https://downloads.skullsecurity.org/passwords/ --no-check-certificate -m

(tried different combinations of -m and -r)

Tried also with --user-agent to prevent wget preventing from downloading

Tried -l 3 , still no success..facepalm

Answer 1

This works for me:

 wget -e robots=off -r -np -nH --accept "*.bz2"  http://downloads.skullsecurity.org/passwords/

Read about Robot Exclusion

If you know what you are doing and really really wish to turn off the robot exclusion, set the robots variable to 'off'

Site http://downloads.skullsecurity.org/ contains robot.txt with content

User-agent: *
Disallow: /

Explanation

The Disallow: / tells the robot that it should not visit any pages on the site.