Python：Webscrape 从网站下载所有数据集

Question

我需要从这个网站下载所有数据集： https://www.electionreturns.pa.gov/ReportCenter/Reports （如果你不是从美国搜索，你将被重定向到另一个网站）这个网站有一个下拉菜单，基于您选择的下拉菜单，它会返回复选框。 我需要编写一个程序来访问该网站，从下拉列表中选择所有可能的选项，选中所有复选框并下载下拉列表和复选框后返回的数据集。

下拉看起来像这样： 选择其中一个下拉选项后的复选框如下所示：

到目前为止，这就是我的开始方式：

html_text=requests.get('https://www.electionreturns.pa.gov/ReportCenter/Reports').content
soup=BeautifulSoup(html_text,'lxml')
options=soup.find_all('option')

这里的问题是，如果我打印选项，它不会返回下拉选项。 相反，它返回占位符：

<select class="form-control ng-invalid ng-invalid-required ng-touched bg-border-mandatory" name="ddlElections" ng-change="rpt.electionChagned(rpt.SelectedElection)" ng-model="rpt.SelectedElection" ng-options="value as value.ElectionName for value in rpt.electionList" placeholder="Select Election">
<option value="">Select Election</option>
</select>

Answer 1

我使用 USA vpn 访问该站点。 我创建了脚本来保存 select 的所有选举，并选中所有复选框，包括地区报告类型。 只能通过使用请求库。 * 确保在当前工作目录中创建了Elections文件夹（或更改路径）。 2015 Special Election 5th Senatorial District没有数据，因此文件未下载。

希望我的程序有所帮助。

import requests
import json

def normalize(data: bytes) -> bytes:
    while True:
        try:
            data = json.loads(data)
        except:
            break

    return data

session = requests.session()

page = session.get("https://www.electionreturns.pa.gov/api/Reports/GetElectionList")
page.raise_for_status()

data = normalize(page.content)

count = len(data["Table"])

print(f"Got {count} elections ...")
for index, election in enumerate(data["Table"]):
    election_name = election["ElectionName"]
    election_id = election["Electionid"]
    election_type = election["ElectionType"]

    print(f"{round(index/count*100)}% Processing '{election_name}' ...")

    page = session.get(f"https://www.electionreturns.pa.gov/api/Reports/GetFilterData?electionId={election_id}&electionsubtype={election_type}")
    page.raise_for_status()

    filter_data = normalize(page.content)

    #print(filter_data)

    page = session.get(f"https://www.electionreturns.pa.gov/api/Reports/GetFilterData?electionId={election_id}&electionsubtype={election_type}&reportType=D")
    page.raise_for_status()

    filter_data_districts = normalize(page.content)

    #print(filter_data_districts)

    data = {
        "ElectionID": election_id,
        "ElectionsubType": election_type,
        "OfficeIds": [value["OfficeID"] for value in filter_data["Table"]],
        "RetOfficeIds": [],
        "PartyIds": [value["PartyID"] for value in filter_data["Table1"]],
        "DistrictIds": [value["DistrictID"] for value in filter_data_districts["Table1"]],
        "CandidateIds": [value["CandidateID"] for value in filter_data["Table2"]],
        "CountyIds": [],
        "ReferendumIds": [],
        "ReferendumDetailIds": [],
        "ExportType": "C", # C = CSV; E = Excel; T = Txt; 
        "ReportType": "D", # D = Districts; C = County; S = Statewide;
        "FileName": "UnOfficial"
    }

    # {
    #     "ElectionID": election_id,
    #     "ElectionsubType": election_type,
    #     "OfficeIds": [
    #         13
    #     ],
    #     "RetOfficeIds": [],
    #     "PartyIds": [
    #         3,
    #         4
    #     ],
    #     "DistrictIds": [
    #         104
    #     ],
    #     "CandidateIds": [
    #         21065,
    #         21066
    #     ],
    #     "CountyIds": [],
    #     "ReferendumIds": [],
    #     "ReferendumDetailIds": [],
    #     "ExportType": "C",
    #     "ReportType": "D",
    #     "FileName": "UnOfficial"
    # }

    #print(data)
    
    page = session.post("https://www.electionreturns.pa.gov/api/Reports/GenerateReport",json=data)
    
    if page.ok:
        print(f"Downloading into './Elections/{election_name}.csv'")
        with open(f"./Elections/{election_name}.csv", "wb") as file:
            file.write(page.content)
    else:
        print(f"Error {page.status_code}! {page.content.decode()}")