Web-scraping for td data

Question

Can someone explain to me why my code is not picking up the PriorSettle td's? I am getting the months but for whatever reason the PrioSettle column is not showing up.

lc_result={}

url = "http://www.cmegroup.com/trading/agricultural/livestock/live-cattle.html"

driver = webdriver.Chrome() 
driver.set_window_size(2,2)
driver.get(url) #this will go the the actual url listed
print('     Live Cattle Futures'+localtime.center(50))
table = driver.find_element_by_id('quotesFuturesProductTable1')
for row in table.find_elements_by_tag_name('tr')[2:]:
    month=row.find_elements_by_tag_name('td')[0].text  
    priorsettle=row.find_elements_by_tag_name('td')[4].text

    print month, priorsettle
    lc_result[month]=[priorsettle]

driver.close()
print(str(date.today()))

Answer 1

You need to wait for the table to load. Simply adding a delay made it work for me:

driver.get(url)

time.sleep(3)

table = driver.find_element_by_id('quotesFuturesProductTable1')
...

Prints:

DEC 2014 168.025
FEB 2015 166.900
APR 2015 164.775
JUN 2015 154.800
AUG 2015 152.900
OCT 2015 154.100
DEC 2015 154.250
FEB 2016 153.850
APR 2016 0.000

FYI, implicit timeouts using time.sleep() is not a reliable and recommended way to wait for elements. Selenium has built-in Waits mechanism.