Selenium - Inserting Values Into Dropdown Menus (Where No 'Select' Tag Exists)

Question

Im trying to scrape daily fantasy baseball projections from here: https://www.numberfire.com/mlb/daily-fantasy/daily-baseball-projections/batters

I want to navigate through the 'Platform' table dropdown option to get projections for DraftKings (FanDuel is set as the table default) using selenium. Following this answer Selenium (Python): How to insert value on a hidden input? I'm attempting to change the hidden value for class 'dfs-option-drop-value'.

from selenium import webdriver

driver = webdriver.Chrome('/Applications/chromedriver')
URL = 'https://www.numberfire.com/mlb/daily-fantasy/daily-baseball-projections/batters'
driver.get(URL)

elem = driver.find_element_by_xpath(
    '//i[@class="nf-icon icon-caret-down active"]'
    '/following-sibling::input[@type="hidden"]')

value = driver.execute_script('return arguments[0].value;', elem)
print("Before update, hidden input value = {}".format(value))

driver.execute_script('''
    var elem = arguments[0];
    var value = arguments[1];
    elem.value = value;
''', elem, '4')

value = driver.execute_script('return arguments[0].value;', elem)
print("After update, hidden input value = {}".format(value))


# Then using beautiful soup
page = driver.page_source
soup = BeautifulSoup(page.content, 'html.parser')

The dfs-option-drop-value is changed as desired, but the page still renders FanDuel projections, so, clearly this isn't the appropriate table change. Note that this particular table doesn't have 'select' tags with option values, so answers such as Selenium dropdown menu and python - scraping tables by navigating different options in drop down list would not work.

Any ideas?

Answer 1

To open that drop-down and select the 'FanDuel' you can do the following:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

#This will open the drop-down dialog
WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.XPATH, "//span[@class='title' and(contains(text(),'Platform'))]/../div[@class='custom-drop']"))).click()

#This will select DraftKings from the open drop-down dialog.
WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.XPATH, "//ul[contains(@class,'active')]//li[@data-value='4']"))).click()