尝试从 Python 中的一系列 URL 下载数据（文本）

Question

Sorry for the possibly dull question. I am trying to download text from a range of URLs with Python all at once. They follow a very straightforward structure:

" http://example.com/01000/01000/01000.htm "; " http://example.com/01000/01001/01001.htm ";

and so on, up to 01099.

After getting the text, I would need to analyze it with the nltk toolkit. I have tried to use wget on Windows, but did not work in command line. I am wondering if there is a way, similar to glob module for URLs to download data from this range all at once.

(There is also some blank URLs in the range.)

Thanks a lot for your help.

Answer 1

Once you've got the URL using string manipulation (seeing that you know the structure of the URL) you can use the Requests module

Example;

import requests

base_url = "http://example.com/01000/01001/0"
for i in range(1000, 1100):
    target_url = base_url + str(i) + ".htm"
    r = requests.get(target_url)

    print(r.text) # python 3 only

Answer 2

You could try my python3-wget module . Heres an example of use;

#!/usr/bin/python3
#-*- coding:utf-8 -*-

import wget

urls = 'http://example.com/01000/01000/0'
for x in range(1000, 1099):
    url = urls + str(x) + '.htm' 
    filename = wget.download(url)

That will download all the files, if you need to exract specific text from the pages you will need to look into creating a simple web scraper with Requests and BeautifulSoup4.

Answer 3

Thanks a lot for your help. In the end, that's how my code looks like:

import requests
base_url = "http://example.com/01000/0"
for i in range(1000, 1100):
    target_url = base_url + str(i) + '/' + '0' + str(i) + ('.htm')
    r = requests.get(target_url)
    print(target_url)

    with open(str(i) + ".htm", 'w', encoding="iso-8859-1") as f:
    f.write(r.text)

 #The encoding is due to language specific text. 
#It downloaded all the files in the given range: http://example.com/01000/01000/01000.htm 
#to /01000/01099/01099.htm.