如何使用 python 从 html 中的动态表中刮取所有行

Question

Here's link for scraping: http://5000best.com/websites/Games/

I tried almost everything I can. I'm a beginner in web scraping.

My code:

import requests
from  urllib.request import  urlopen
from urllib.error import  HTTPError
from urllib.error import  URLError
from bs4 import  BeautifulSoup
import pandas as pd
import csv


try:
    html = urlopen("http://5000best.com/websites/Games/")

except HTTPError as e:
    print(e)

except URLError as u:
    print(u)

else:
    soup = BeautifulSoup(html,"html.parser")
    table = soup.findAll('div',{"id":"content"})[0]
    tr = table.findAll(['tr'])[0:]
    csvFile = open('games.csv','wt', newline='',encoding='utf-8')
    writer = csv.writer(csvFile)
    try:   
        for cell in tr:
            th = cell.find_all('th')
            th_data = [col.text.strip('\n') for col in th]
            td = cell.find_all('td')
            row = [i.text.replace('\n','') for i in td]
            writer.writerow(th_data+row)      

    finally:   
        csvFile.close()

This code only scrape the first page of the table... I want all the pages. I inspected the web page but I didn't saw any url changes while toggling the page numbers, So it's completely dynamic.

Answer 1

You can read it directly using pandas.read_html() function as a DataFrame which will do it easily for you.

import pandas as pd


def main(url):
    for item in range(1, 4):
        df = pd.read_html(url.format(item))[1]
        print(df)


main("http://5000best.com/websites/Games/{}/")

Sample of output:

CSV edit:

import pandas as pd


def main(url):
    for item in range(1, 4):
        df = pd.read_html(url.format(item))[1]
        print(f"Saving Page {item}")
        df.to_csv(f"page{item}.csv", index=False)


main("http://5000best.com/websites/Games/{}/")

Code updated for single DataFrame :

import pandas as pd


def main(url):
    goal = []
    for item in range(1, 4):
        df = pd.read_html(url.format(item))[1]
        goal.append(df)
    final = pd.concat(goal)
    print(final)


main("http://5000best.com/websites/Games/{}/")

Answer 2

Looking at the network inspector for that page reveals that it makes requests to

• http://5000best.com/?wc&xml=1&ta=9&p=1&sortby=0&ise=&h=01
• http://5000best.com/?wc&xml=1&ta=9&p=2&sortby=0&ise=&h=02
• http://5000best.com/?wc&xml=1&ta=9&p=3&sortby=0&ise=&h=03

when you change pages. You may want to just scrape those instead.

Answer 3

Let me try to help you understand.

Have you used the developer tools in your browser? Open that (Use F12 or right click > inspect element) and select the network tab. Now while keeping the tab open, click on the next page link. A request shows up in the Network Tab.

This is what you are looking for. All dynamic thing on a web page can be viewed here.

Hope this helps you learn something. Cheers!