使用python抓取网站时获取最大页面数

Question

I am very new to python and have to scrape a website for some data for a course at university:

Xrel

I am able to get the information i need. The problem is i need it for every entry(page, month, year).

The amount of pages differs for every month. Is there any way to extract the maximum pagenumber so i can store it and use it for a loop?

I would appreciate any help. Thanks!

Answer 1

For loops are nice but you can't always use them. In this case I would just repeatedly follow the link in the 'next page' button until there is no such button. Something like this:

url = <first page>
while True:
    # extract data
    if <there is a next page button>:
        url = <href of the button>
    else:
        break

Answer 2

This will get all your pages, yielding a BeautifulSoup object for each, the link to the next page is in the anchor tag with the class forward :

import requests
from urlparse import urljoin


def get_pages(base, url):
    soup = BeautifulSoup(requests.get(url).content)
    yield soup
    next_page = soup.select_one("a.forward")
    for page in iter(lambda: next_page, None):
        soup = BeautifulSoup(requests.get(urljoin(base, page["href"])).content)
        yield soup
        next_page = soup.select_one("a.forward")



for soup in get_pages("https://www.xrel.to/", "https://www.xrel.to/games-release-list.html?archive=2016-01"):
    print(soup)