如何获取python网页的前3句？

Question

I have an assignment where one of the things I can do is find the first 3 sentences of a webpage and display it. Find the webpage text is easy enough, but I'm having problems figuring out how I find the first 3 sentences.

import requests
from bs4 import BeautifulSoup

url = 'https://www.troyhunt.com/the-773-million-record-collection-1-data-reach/'
res = requests.get(url)
html_page = res.content
soup = BeautifulSoup(html_page, 'html.parser')
text = soup.find_all(text=True)

output = ''
blacklist = [
      '[document]',
      'noscript',
      'header',
      'html',
      'meta',
      'head',
      'input',
      'script'
]

for t in text:
  if (t.parent.name not in blacklist):
    output += '{} '.format(t)

tempout = output.split('.')
for i in range(tempout):
  if (i >= 3):
    tempout.remove(i)

output = '.'.join(tempout)

print(output)

Answer 1

Finding sentences out of text is difficult. Normally you would look for characters that might complete a sentence, such as '.' and '.'. But a period (',') could appear in the middle of a sentence as in an abbreviation of a person's name. for example, I use a regular expression to look for a period followed by either a single space or the end of the string, which works for the first three sentences. but not for any arbitrary sentence.

import requests
from bs4 import BeautifulSoup
import re

url = 'https://www.troyhunt.com/the-773-million-record-collection-1-data-reach/'
res = requests.get(url)
html_page = res.content
soup = BeautifulSoup(html_page, 'html.parser')

paragraphs = soup.select('section.article_text p')
sentences = []
for paragraph in paragraphs:
    matches = re.findall(r'(.+?[.!])(?: |$)', paragraph.text)
    needed = 3 - len(sentences)
    found = len(matches)
    n = min(found, needed)
    for i in range(n):
        sentences.append(matches[i])
    if len(sentences) == 3:
        break
print(sentences)

Prints:

['Many people will land on this page after learning that their email address has appeared in a data breach I\'ve called "Collection #1".', "Most of them won't have a tech background or be familiar with the concept of credential stuffing so I'm going to write this post for the masses and link out to more detailed material for those who want to go deeper.", "Let's start with the raw numbers because that's the headline, then I'll drill down into where it's from and what it's composed of."]

Answer 2

To scrape the first three sentences, just add these lines to ur code:

section = soup.find('section',class_ = "article_text post") #Finds the section tag with class "article_text post"

txt = section.p.text #Gets the text within the first p tag within the variable section (the section tag)

print(txt)

Output:

Many people will land on this page after learning that their email address has appeared in a data breach I've called "Collection #1". Most of them won't have a tech background or be familiar with the concept of credential stuffing so I'm going to write this post for the masses and link out to more detailed material for those who want to go deeper.

Hope that this helps!

Answer 3

Actually using beautify soup you can filter by the class "article_text post" seeing source code:

myData=soup.find('section',class_ = "article_text post")
print(myData.p.text)

And get the inner text of p element

Use this instead of soup = BeautifulSoup(html_page, 'html.parser')