BS4在Python中找不到元素

Question

I am somewhat new to Python and can't for the life of me figure out why the following code isn't pulling the element I am trying to get.

It currently returns:

for player in all_players:

    player_first, player_last = player.split()
    player_first = player_first.lower()
    player_last = player_last.lower()
    first_name_letters = player_first[:2]
    last_name_letters = player_last[:5]

    player_url_code = '/{}/{}{}01'.format(last_name_letters[0], last_name_letters, first_name_letters)
    player_url = 'https://www.basketball-reference.com/players' + player_url_code + '.html'
    print(player_url) #test
    req = urlopen(player_url)
    soup = bs.BeautifulSoup(req, 'lxml')
    wrapper = soup.find('div', id='all_advanced_pbp')
    table = wrapper.find('div', class_='table_outer_container')


    for td in table.find_all('td'):
        player_pbp_data.append(td.get_text())

Currently returning:

--> for td in table.find_all('td'):
        player_pbp_data.append(td.get_text()) #if this works, would like to 

AttributeError: 'NoneType' object has no attribute 'find_all'

Note: iterating through children of the wrapper object returns:

< div class="table_outer_container" > as part of the tree.

Thanks!

Answer 1

尝试改为显式传递html：

bs.BeautifulSoup(the_html, 'html.parser')

Answer 2

Make sure that table contains the data you expect.

For example https://www.basketball-reference.com/players/a/abdulka01.html doesn't seem to contain a div with id='all_advanced_pbp'

Answer 3

I trie to extract data from the url you gave but it did not get full DOM. after then i try to access the page with browser with javascrip and without javascrip, i know website need javascrip to load some data. But the page like players it need not. The simple way to get dynamic data is using selenium

This is my test code

import requests
from bs4 import BeautifulSoup
from selenium import webdriver

player_pbp_data = []

def get_list(t="a"):
    with requests.Session() as se:
        url = "https://www.basketball-reference.com/players/{}/".format(t)
        req = se.get(url)
        soup = BeautifulSoup(req.text,"lxml")
        with open("a.html","wb") as f:
            f.write(req.text.encode())
        table = soup.find("div",class_="table_wrapper setup_long long")
        players = {player.a.text:"https://www.basketball-reference.com"+player.a["href"] for player in table.find_all("th",class_="left ")}


def get_each_player(player_url="https://www.basketball-reference.com/players/a/abdulta01.html"):

    with webdriver.Chrome() as ph:
        ph.get(player_url)
        text = ph.page_source

    '''
    with requests.Session() as se:
        text = se.get(player_url).text
    '''

    soup = BeautifulSoup(text, 'lxml')
    try:
        wrapper = soup.find('div', id='all_advanced_pbp')
        table = wrapper.find('div', class_='table_outer_container')
        for td in table.find_all('td'):
            player_pbp_data.append(td.get_text())
    except Exception as e:
        print("This page dose not contain pbp")



get_each_player()