如何从链接中带有或不带有“.html”的 URL 链接收集文本数据？

Question

I am trying to collect some text data from a URL like https://scikit-learn.org/stable/modules/linear_model.html .

I would like to get the following text data from the html

 1.1. Linear Models¶
 The following are a set of methods intended for regression in which the target value is 
 expected to be a linear combination of the features. In mathematical notation, if 
 is the predicted value.

My code:

import urllib
from bs4 import BeautifulSoup
link = "https://scikit-learn.org/stable/modules/linear_model.html"
f = urllib.request.urlopen(link)
html = f.read()
soup = BeautifulSoup(html)
print(soup.prettify()) 

How to navigate into the embedded body of the html to get the above text data ?

Also, I need to do the similar things for some links without ".html", I use the same code but no anything of the text data is returned from the link.

I cannot see anything of the text data when I printed it out by

 print(soup.prettify())

The return status is

  200

What could be the reason ?

thanks

Answer 1

When creating a BeautifulSoup object, you have to specify the parser that you want to use. Apart from that, I also recommend you to use requests instead of urllib , but it is completely your wish. Here is how you extract the text that you want:

div = soup.find('div', class_ = "section") #Finds the div with class section

print(div.h1.text) #Prints the text within the first h1 tag within the div

print(div.p.text) #Prints the text within the first p tag within the div

Output:

1.1. Linear Models¶
The following are a set of methods intended for regression in which
the target value is expected to be a linear combination of the features.
In mathematical notation, if \(\hat{y}\) is the predicted
value.

Here is the full code:

import urllib
from bs4 import BeautifulSoup
link = "https://scikit-learn.org/stable/modules/linear_model.html"
f = urllib.request.urlopen(link)
html = f.read()
soup = BeautifulSoup(html,'html5lib')

div = soup.find('div', class_ = "section")

print(div.h1.text)

print(div.p.text)