从基于下拉菜单更改的数据表中抓取“ li”标签

Question

I am trying to scrape data from a data table on this website: [ http://www.oddsshark.com/ncaab/lsu-alabama-odds-february-18-2017-744793]

The site has multiple tabs, which changes the html (I am working in the 'matchup' tab). Within that matchup tab, there is a drop-down menu that changes the data table that I am trying to access. The items in the table that I am trying to access are 'li' tags within an unordered list. I just want to scrape the data from the "Overall" category of the drop-down menu.

I have been unable to access the data that I want. The item that I'm trying to access is coming back as a 'noneType'. Is there a way to do this?

url = "http://www.oddsshark.com/ncaab/lsu-alabama-odds-february-18-2017-
744793"  
html_page = requests.get(url)
soup = BeautifulSoup(html_page.content, 'html.parser')

dataList = []
for ultag in soup.find_all('ul', {'class': 'base-list team-stats'}):
    print(ultag)
    for iltag in ultag.find_all('li'):
        dataList.append(iltag.get_text())

Answer 1

So the problem is that the content of the tab you are trying to pull data from is dynamically loaded using React JS. So you have to use the selenium module in Python to open a browser to click the list element "Matchup" programmatically then get the source after clicking it.

On my mac I installed selenium and the chromewebdriver using these instructions:

https://gist.github.com/guylaor/3eb9e7ff2ac91b7559625262b8a6dd5f

Then signed the python file, so that the OS X firewall doesn't complain to us when trying run it, using these instructions: Add Python to OS X Firewall Options?

Then ran the following python3 code:

import os
import time
from selenium import webdriver
from bs4 import BeautifulSoup as soup

# Setup Selenium Chrome Web Driver
chromedriver = "/usr/local/bin/chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)

# Navigate in Chrome to specified page.
driver.get("http://www.oddsshark.com/ncaab/lsu-alabama-odds-february-18-2017-744793")

# Find the matchup list element using a css selector and click it.
link = driver.find_element_by_css_selector("li[id='react-tabs-0'").click()

# Wait for content to load
time.sleep(1)

# Get the current page source.
source = driver.page_source

# Parse into soup() the source of the page after the link is clicked and use "html.parser" as the Parser.
soupify = soup(source, 'html.parser')

dataList = []
for ultag in soupify.find_all('ul', {'class': 'base-list team-stats'}):
    print(ultag)
    for iltag in ultag.find_all('li'):
        dataList.append(iltag.get_text())

# We are done with the driver so quit.
driver.quit()

Hope this helps as I noticed this was a similar problem to the one I just solved here - Beautifulsoup doesn't reach a child element