从多个页面抓取表格

Question

我想从这个网站上抓取数据。 该表可扩展到多个页面，但是url没有类似page=3 ，因此我无法遍历各个页面来获取数据。

我还有另一种方式可以抓取这样的数据而无需遍历各个页面吗？ 如何从python访问不同的页面以抓取数据？

Answer 1

使用@SeanBreckenridge提供的直接JSON链接，您可以创建一个包含所有数据的CSV文件，如下所示：

import requests
import json
import csv

header = ['University', 'Department', 'Overall Pay', 'LW Ratio', 'Academic Year', 'Program Year', 'Comment', 'V1', 'V2', 'V3', 'V4']

with open('output.csv', 'w', newline='') as f_output:
    csv_output = csv.writer(f_output)
    csv_output.writerow(header)

    for index in range(30):
        url = 'http://www.phdstipends.com/data/{}'.format(index)
        print(url)
        r = requests.get(url)

        if len(r.text) < 20:
            break

        csv_output.writerows(json.loads(r.text)['data'])

哪个开始：

University,Department,Overall Pay,LW Ratio,Academic Year,Program Year,Comment,V1,V2,V3,V4
University of California - Santa Barbara (UCSB),Chemical Engineering,"$31,125",1.27,2014-2015,1st,,"$31,250",,,$125
Pennsylvania State University (Penn State),Chemistry,"$20,500",1.11,2014-2015,2nd,,"$20,500",,,
University of California - San Diego (UCSD),Mathematics,"$20,500",0.87,2013-2014,4th,,,"$18,000","$2,500",