Download file using wget

Question

I need to download fits files from a webpage. I'm doing this using -i wget options: I store the download file in a list.txt file which contains URL1, URL2... and then

$ wget -i list.txt

Do you know if there is the possibility to do the same thing using a python script? Thanks.

Answer 1

Assuming your file contains one URL per line, you can do this:

import urllib2
with open('list.txt') as my_list:
    for line in my_list:
        response = urllib2.urlopen(line)
        html = response.read()
        # now process the page's source

Answer 2

with open('list.txt') as my_list:
    for url in my_list:
        wget.download(url)

Answer 3

If you get SSL: CERTIFICATE_VERIFY_FAILED:

import wget
import ssl

ssl._create_default_https_context = ssl._create_unverified_context
with open('list.txt') as my_list:
    for url in my_list:
        wget.download(url)

Using the OS library:

import os

with open('list.txt') as my_list:
    for url in my_list:
        os.system('wget ' + url)