如何修复此 python 代码以获取我在代码中引用的表中的所有人员姓名？

Question

I am trying to get python to give me the names of the State Senators from Alabama on Ballotpedia. However, the code I put together is only giving me the title I requested from the url but I am not getting any names. Here is my current python code:

import requests
from bs4 import BeautifulSoup
import pandas as pd

list = ['https://ballotpedia.org/Alabama_State_Senate']

temp_dict = {}

for page in list:
    r = requests.get(page)
    soup = BeautifulSoup(r.content, 'html.parser')

    temp_dict[page.split('/')[-1]] = [item.text for item in 
soup.select("table.bptable gray sortable tablesorter 
jquery-tablesorter a")]

df = pd.DataFrame.from_dict(temp_dict, 
orient='index').transpose()

I believe my error is in this line:

    temp_dict[page.split('/')[-1]] = [item.text for item in soup.select("table.bptable gray sortable tablesorter jquery-tablesorter a")]

Thank you.

Answer 1

This seems to work for me:

import requests
from bs4 import BeautifulSoup

url = "https://ballotpedia.org/Alabama_State_Senate"

response = requests.get(url)
response.raise_for_status()

soup = BeautifulSoup(response.content, "html.parser")

for row in soup.find(id="officeholder-table").select("tr:not([colspan])"):
    name = row.select_one("td:nth-of-type(2)").text
    print(name)

Answer 2

Another way to get all info:

holders = soup.select("#officeholder-table")
targets = holders[0].select('tr')
for target in targets:
    print(target.text)

Output:

Alabama State Senate District 20
Linda Coleman-Madison
Democratic 
2006

etc.