我怎样才能刮掉精确的值/文本 <td> 在使用硒选择下拉列表后使用其xpath？

Question

I am trying to extract a specific value from a table for each of the five days that are shown in the drop-down menu.

I need to be able to get each days' settle value on a recurring basis(each week user will scrape for the five new prices). Currently my script will only retrieve today's posted value in the table.

I had similar issues when using lxml to extract the xpath, which is why I thought it must be a javascript thing, so I am trying out Selenium now. Any help or direction is appriciated.

from selenium import webdriver

path_to_chromedriver = '/Users/Daniel/Desktop/chromedriver' 

driver = webdriver.Chrome(executable_path='C:\Users\Daniel\Desktop\chromedriver\chromedriver.exe')

url = 'http://www.cmegroup.com/trading/energy/crude-oil/light-sweet-crude_quotes_settlements_futures.html'
driver.get(url)

driver.find_element_by_xpath('//*[@id="cmeTradeDate"]/option[2]').click()

driver.implicitly_wait(10)

settle_price = driver.find_element_by_xpath('//*[@id="settlementsFuturesProductTable"]/tbody/tr[1]/td[6]').text

print settle_price

http://www.cmegroup.com/trading/energy/crude-oil/light-sweet-crude_quotes_settlements_futures.html

(Crude Oil Prices- drop down contains five days) Value needed from page is the settle price for May 16(xpath):

 //*[@id="settlementsFuturesProductTable"]/tbody/tr[1]/td[6]

Why is the text output for todays page and not the drop down element that the browser goes to?

Answer 1

Several things to introduce to solve it:

• use the Select class and the .options property to get and iterate over the dropdown options
• once an option value is selected, the table is getting updated. In order to catch when it finished updating, you would need to (well, there are different strategies, it's just one of them) explicitly wait for invisibility of the "Processing" spinner/loader element

Implementation:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.select import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)

url = 'http://www.cmegroup.com/trading/energy/crude-oil/light-sweet-crude_quotes_settlements_futures.html'
driver.get(url)

select = Select(driver.find_element_by_id("cmeTradeDate"))
for option in select.options:
    # selecting a value in the dropdown
    select.select_by_value(option.get_attribute("value"))

    # wait for the table to load
    wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR, ".cmeProgressPanel")))

    # get the desired price
    selected_price = driver.find_element_by_xpath('//*[@id="settlementsFuturesProductTable"]/tbody/tr[1]/td[6]')
    print(option.text, selected_price.text)

Prints:

(u'Friday, 15 Apr 2016 (Final)', u'40.36')
(u'Thursday, 14 Apr 2016 (Final)', u'41.50')
(u'Wednesday, 13 Apr 2016 (Final)', u'41.76')
(u'Tuesday, 12 Apr 2016 (Final)', u'42.17')
(u'Monday, 11 Apr 2016 (Final)', u'40.36')