如何从 URL 中提取数据？

Question

I have a xlsx file where a lot of URLs are stored along with their serial ids. Each of these URLs redirects to a webpage where there is article written. My question is how do I scan all the URLs using python and store the title and the texts of the article in a new text file with the URL serial id as its file name?

Answer 1

You can do this using webscraping.

As you said, you have a xlsx containing tuples (ids, url) .

You could start by loading this into python with :

import pandas as pd

urls = pd.read_excel(filename)

Then to read the content of each URL you can use one of the most famous Web scraping library in python : BeautifulSoup .

from bs4 import BeautifulSoup
import requests

# get the raw HTML from the request
content = requests.get(url).content

# build the soup
soup = BeautifulSoup(content)

# get the title
title_tag = soup.find("title") # shows <title>ActualTitle</title>

title = soup.find("title").string # show ActualTitle


# You can get the whole text contained in the page 
text_content = soup.get_text()