使用 Python 从 URL 刮取子文件

Question

A webpage I would like to scrape consists of several files:

I'm interested of scraping only the highlighted file, that is: mboxFrame.

My method of scraping pages

import requests
from bs4 import BeautifulSoup

webPage = requests.get(URL, verify=False)

soup = BeautifulSoup(webPage.content, "html.parser" )

is able to scrape only the file mail.html. Is there a way to scrape only what I want?

I would appreciate any hints or tips.

Answer 1

The way to open a file from a server is to request it with a URL. In fact, in the beginnings of the world wide web this was the only way to get content: content creators would put various files on servers and clients would open or download those files. The dynamic processing of URIs and parameters is a later invention. That is why commenters are asking for the URL you use. We want to see it and modify accordingly to help you see what parts need changing in order to get that particular file. You can omit the password, or replace it with some other string of letters.

In general, the file you want would be under the url you use, but ending with the file name. If the startong URL is www.example.com/mail/ , then this file would be at www.example.com/mail/mbox.msc .

Please note that any parameters should follow the path, so www.example.com/mail?user=hendrra&password=hendras_password would turn into www.example.com/mail/mbox.msc?user=hendrra&password=hendras_password