如何在python中使用硒和beautifulsoup從網站上抓報紙？

Question

我正在嘗試從報紙上收集日期，標題和內容（紐約時間）。

我有日期和標題，但沒有完整的文章。 下面是我用來抓取日期和標題的代碼。

import time
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

base = "https://www.nytimes.com"
browser = webdriver.Chrome('C:/chromedriver_win32/chromedriver.exe')
wait = WebDriverWait(browser, 10)
browser.get('https://www.nytimes.com/search?endDate=20190331&query=cybersecurity&sort=newest&startDate=20180401')

while True:
    try:
        time.sleep(1)
    show_more = wait.until(EC.element_to_be_clickable((By.XPATH, '//button[@type="button"][contains(.,"Show More")]')))
        show_more.click()
    except Exception as e:
            print(e)
            break

soup = BeautifulSoup(browser.page_source,'lxml')
search_results = soup.find('ol', {'data-testid':'search-results'})

links = search_results.find_all('a')
for link in links:
    title = link.find('h4').text
    date = link.find_next('time').text
    print(date + ': '+ title)

print("Complete")

browser.quit()

除了日期，標題外，我還想抓全文。

Answer 1

有可能的。 您必須獲取文章的每個鏈接，然后將其拉出。 我在下面添加了它，還整理了一下文章，因為有多余的空格，當您將它們全部合並在一起時，文本中沒有空格。

import time
import requests
from bs4 import BeautifulSoup
import json
import string
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

base = "https://www.nytimes.com"
browser = webdriver.Chrome('C:/chromedriver_win32/chromedriver.exe')
wait = WebDriverWait(browser, 10)
browser.get('https://www.nytimes.com/search?endDate=20190331&query=cybersecurity&sort=newest&startDate=20180401')

while True:
    try:
        time.sleep(1)
        show_more = wait.until(EC.element_to_be_clickable((By.XPATH, '//button[@type="button"][contains(.,"Show More")]')))  
        show_more.click()
    except Exception as e:
            print(e)
            break    

soup = BeautifulSoup(browser.page_source,'lxml')
search_results = soup.find('ol', {'data-testid':'search-results'})

links = search_results.find_all('a')
for link in links:
    link_url = link['href']

    title = link.find('h4').text
    date = link.find_next('time').text
    print(date + ': '+ title + '\n')

    response = requests.get(base + link_url)
    soup_link = BeautifulSoup(response.text, 'html.parser')
    scripts = soup_link.find_all('script')
    for script in scripts:
        if 'window.__preloadedData = ' in script.text:
            jsonStr = script.text
            jsonStr = jsonStr.split('window.__preloadedData = ')[-1]
            jsonStr = jsonStr.rsplit(';',1)[0]

            jsonData = json.loads(jsonStr)

            article = []
            for k, v in jsonData['initialState'].items():
                w=1
                try:
                    if v['__typename'] == 'TextInline':
                        article.append(v['text'])
                        #print (v['text'])
                except:
                    continue
            article = [ each.strip() for each in article ]
            article = ''.join([('' if c in string.punctuation else ' ')+c for c in article]).strip()
    print (article + '\n')

print("Complete")

browser.quit()

輸出：

March 31: Bezos’ Security Consultant Accuses Saudis of Hacking the Amazon C.E.O.’s Phone

SEATTLE — Jeff Bezos’ security consultant accused the Saudi government of gaining unauthorized access to the Amazon chief executive’s phone, as part of an effort to harm the world’s richest man. In an opinion article in The Daily Beast on Saturday, Gavin de Becker, Mr. Bezos’ security chief, alleged the Saudis wanted to hurt Mr. Bezos because he owns The Washington Post. The Post has aggressively reported on the murder of Jamal Khashoggi, one of its columnists, who was killed last year in Turkey. United States officials have concluded Mr. Khashoggi, who was critical of Saudi leaders, was killed on the orders of the Saudi crown prince, Mohammed bin Salman. Mr. de Becker said he had turned over his findings about the Saudis and their role against Mr. Bezos to law enforcement. “Our investigators and several experts concluded with high confidence that the Saudis had access to Bezos’ phone, and gained private information,” Mr. de Becker wrote. Mr. de Becker’s claims about the Saudis are difficult to verify and raise many questions. Throughout his article, the security consultant was vague on details. He did not reveal direct evidence of his accusations and wrote that he would not speak again publicly on the issue. On Sunday, American Media Inc., the parent company of The National Enquirer, which first reported earlier this year that Mr. Bezos was having an extramarital affair, issued a statement denying “the false and unsubstantiated claims of Mr. de Becker.” Amazon did not immediately have a comment about Mr. Bezos. An official with the Saudi embassy in Washington declined to comment on Saturday. Mr. de Becker’s allegations are the latest twist in a bizarre situation that has embroiled the largest U.S. tabloid publisher and Mr. Bezos, who runs Amazon and owns The Post. The unusual set of circumstances began in January, when Mr. Bezos announced that he and his wife, MacKenzie Bezos, were divorcing. The next day, The Enquirer published an exposé revealing that Mr. Bezos was romantically involved with Lauren Sanchez, a former Los Angeles TV anchor. Then in February, Mr. Bezos published a post on Medium accusing AMI of “extortion and blackmail.” Mr. Bezos said AMI had threatened to publish graphic photographs of him, including a “below-the-belt selfie,” if he did not publicly affirm that The Enquirer’s reporting on his affair was not motivated by political concerns. He said AMI, which has had ties to the Saudis, was “apoplectic” about The Post’s reporting on the Saudis. Mr. Bezos added that he had asked Mr. de Becker, his longtime security consultant, to investigate who had leaked information and photos about him. Earlier this month, The New York Times reported that two people with direct knowledge of The Enquirer’s reporting said that everything the tabloid received on Mr. Bezos’ affair, including the “below-the-belt selfie,” came from a single source. The Wall Street Journal later reported that AMI had paid Ms. Sanchez’s brother, Michael Sanchez, $200,000 for the texts. In its statement on Sunday, AMI said directly that its source was Mr. Sanchez. “The fact of the matter is, it was Michael Sanchez who tipped the National Enquirer off to the affair on Sept. 10, 2018, and over the course of four months provided all of the materials for our investigation.” AMI added, “There was no involvement by any other third party whatsover.” Mr. de Becker on Saturday said that the effort against Mr. Bezos went beyond Mr. Sanchez and also involved the Saudis. Mr. de Becker pointed to an article published on Saturday by The New York Post in which Mr. Sanchez said The Enquirer “had seen text exchanges between the secret couple” before he was in touch with the tabloid on the matter. “Reality is complicated, and can’t always be boiled down to a simple narrative like ‘the brother did it,’” Mr. de Becker wrote. Mr. de Becker said his investigation included interviews with cybersecurity experts and “people who personally know the Saudi Crown Prince Mohammed bin Salman.” But he stopped short of saying what methods he believed the Saudis may have used to access Mr. Bezos’ personal information. He added that AMI had wanted him to make a public statement “saying that my investigation had concluded they hadn’t relied upon ‘any form of electronic eavesdropping or hacking in their news-gathering process,’” and that the tabloid’s story was not “instigated, dictated or influenced in any manner by external forces, political or otherwise.” Mr. de Becker said he told AMI in a recorded call that those claims were “not my truth.”

March 29: In Ukraine, Russia Tests a New Facebook Tactic in Election Tampering

Campaigning for Ukraine’s presidential election had just begun to heat up when the authorities announced they had thwarted a Russian plot to use Facebook to undermine the vote. Unlike the 2016 interference in the United States, which centered on fake Facebook pages created by Russians in faraway St. Petersburg, the operation in Ukraine this year had ...

...

Answer 2

您只觀察搜索的第一頁。 您擁有文章列表的位置。 要獲取文章的內容，您必須向該文章發送請求並從那里獲取內容。

在這里，我獲取標題，作者，發布日期，內容並將它們存儲在列表中。 如果需要，我們可以從該列表中稍后創建一個DataFrame。

newyork_times_list = []
for a in search_results.find_all('a', href=True):

    newyork_times = {}
    page_url = "https://www.nytimes.com" + a['href']

    try:
        # URL
        newyork_times['URL'] =  page_url

        # Invoke URL
        page = requests.get(page_url)
        page_soup = BeautifulSoup(page.content, 'lxml')

        # Title
        newyork_times['Title'] = page_soup.find('title').text

        # Content
        page_content = ''
        page_soup_div = page_soup.find_all("div", {"class":"StoryBodyCompanionColumn"})
        for p_content in page_soup_div:
            page_content = page_content + p_content.text

        # Content
        newyork_times['Content'] =  page_content

        # Date Time
        page_soup_span = page_soup.find_all("time")
        newyork_times['Publish Date'] = page_soup_span[0].text

        # Author
        page_soup_span = page_soup.find_all("span", {"itemprop": "name"})
        newyork_times['Author'] =  page_soup_span[0].text

        newyork_times_list.append(newyork_times)

        print('Processed', page_url)
    except:
        print('ERROR!', page_url)

print('Done')

Answer 3

為了抓取報紙文章，您可以簡單地使用簡單優雅的鵝庫。 這也會導致您也清除了文章的文本和標題。 對於Date，您可以嘗試使用BeautifulSoup。

from goose import Goose
from requests import get

response = get('http://www.nytimes.com/2015/05/19/health/study-finds-dense-breast-tissue-isnt-always-a-high-cancer-risk.html?src=me&ref=general')
extractor = Goose()
article = extractor.extract(raw_html=response.content)
text = article.cleaned_text
title = article.title