在 Python 中抓取多个页面

Question

I am trying to scrape a page that includes 12 links. I need to open each of these links and scrape all of their titles. When I open each page, I face multiple pages in each link. However, my code could only scrape the first page in all of these 12 links

By below code, I can print all the 12 links URLs that exist on the main page.

url = 'http://mlg.ucd.ie/modules/COMP41680/assignment2/index.html'
res = requests.get (url)
soup = BeautifulSoup(res.text, 'html.parser') 
links = soup.find_all("a")
all_urls = []
for link in links[1:]:
    link_address ='http://mlg.ucd.ie/modules/COMP41680/assignment2/' + link.get("href")
    all_urls.append(link_address)

Then, I looped in all of them.

for i in range(0,12):
    url = all_urls[i]
    res = requests.get (url)
    soup = BeautifulSoup(res.text, 'html.parser') 

The title could be extracted by below lines:

title_news = []
news_div = soup.find_all('div', class_ = 'article')    
for container in news_div:
        title = container.h5.a.text 
        title_news.append(title)

The output of this code only includes the title for the first page of each of these 12 pages, while I need my code to go through multiple pages in these 12 URLs. The below gives me the links of all the pages that exist in each of these 12 links if it defines in an appropriate loop. ( It reads the pagination section and look for the next page URL link)

page = soup.find('ul', {'class' : 'pagination'}).select('li', {'class': "page-link"})[2].find('a')['href']

How I should use a page variable inside my code to extract multiple pages in all of these 12 links and read all the titles and not only first-page titles.

Answer 1

You can use this code to get all titles from all the pages:

import requests
from bs4 import BeautifulSoup

base_url = "http://mlg.ucd.ie/modules/COMP41680/assignment2/"
soup = BeautifulSoup(
    requests.get(base_url + "index.html").content, "html.parser"
)

title_news = []
for a in soup.select("#all a"):
    next_link = a["href"]

    print("Getting", base_url + next_link)

    while True:
        soup = BeautifulSoup(
            requests.get(base_url + next_link).content, "html.parser"
        )
        for title in soup.select("h5 a"):
            title_news.append(title.text)

        next_link = soup.select_one('a[aria-label="Next"]')["href"]

        if next_link == "#":
            break

print("Length of title_news:", len(title_news))

Prints:

Getting http://mlg.ucd.ie/modules/COMP41680/assignment2/month-jan-001.html
Getting http://mlg.ucd.ie/modules/COMP41680/assignment2/month-feb-001.html
Getting http://mlg.ucd.ie/modules/COMP41680/assignment2/month-mar-001.html
Getting http://mlg.ucd.ie/modules/COMP41680/assignment2/month-apr-001.html
Getting http://mlg.ucd.ie/modules/COMP41680/assignment2/month-may-001.html
Getting http://mlg.ucd.ie/modules/COMP41680/assignment2/month-jun-001.html
Getting http://mlg.ucd.ie/modules/COMP41680/assignment2/month-jul-001.html
Getting http://mlg.ucd.ie/modules/COMP41680/assignment2/month-aug-001.html
Getting http://mlg.ucd.ie/modules/COMP41680/assignment2/month-sep-001.html
Getting http://mlg.ucd.ie/modules/COMP41680/assignment2/month-oct-001.html
Getting http://mlg.ucd.ie/modules/COMP41680/assignment2/month-nov-001.html
Getting http://mlg.ucd.ie/modules/COMP41680/assignment2/month-dec-001.html
Length of title_news: 16226