网页抓取 - Selenium - Python

Question

I want to extract all the fantasy teams that have been entered for past contests. To loop through the dates, I just change a small part of the URL as shown in my code below:

#Packages:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
import pandas as pd


# Driver
chromedriver =("C:/Users/Michel/Desktop/python/package/chromedriver_win32/chromedriver.exe")
driver = webdriver.Chrome(chromedriver)

# Dataframe that will be use later 
results = pd.DataFrame()
best_lineups=pd.DataFrame()
opti_lineups=pd.DataFrame()

#For loop over all DATES:

calendar=[]
calendar.append("2019-01-10")
calendar.append("2019-01-11")

for d in calendar:
    driver.get("https://rotogrinders.com/resultsdb/date/"+d+"/sport/4/")

Then, to access the different contests of that day, you need to click on the contest tab. I use the following code to locate and click on it.

 # Find "Contest" tab   
    contest= driver.find_element_by_xpath("//*[@id='root']/div/main/main/div[2]/div[3]/div/div/div[1]/div/div/div/div/div[3]")
    contest.click()

I simply inspect and copy the xpath of the tab. However, most of the times it is working, but sometimes I get an error message " Unable to locate element...". Moreover, it seems to work only for the first date in my calendar loop and always fails in the next iteration... I do not know why. I try to locate it differently, but I feel I am missing something such as:

contests=driver.find_element_by_xpath("//*[@role='tab']

Once, the contest tab is successfully clicked, all contests of that day are there and you can click on a link to access all the entries of that contest. I stored the contests in order to iterate throuhg all as follow:

    list_links = driver.find_elements_by_tag_name('a')
    hlink=[]
    for ii in list_links:
        hlink.append(ii.get_attribute("href"))
    sub="https://rotogrinders.com/resultsdb"
    con= "contest"
    contest_list=[]
    for text in hlink:
        if sub in text:
            if con in text:
                contest_list.append(text)
# Iterate through all the entries(user) of a contest and extract the information of the team entered by the user 

    for c in contest_list:
        driver.get(c)

Then, I want to extract all participants team entered in the contest and store it in a dataframe. I am able to do it successfully for the first page of the contest.

# Waits until tables are loaded and has text. Timeouts after 60 seconds
        while WebDriverWait(driver, 60).until(ec.presence_of_element_located((By.XPATH, './/tbody//tr//td//span//a[text() != ""]'))):

# while ????: 

# Get tables to get the user names
            tables = pd.read_html(driver.page_source)
            users_df  = tables[0][['Rank','User']]
            users_df['User'] = users_df['User'].str.replace(' Member', '')

# Initialize results dataframe and iterate through users

            for i, row in users_df.iterrows():

                rank = row['Rank']
                user = row['User']

    # Find the user name and click on the name
                user_link = driver.find_elements(By.XPATH, "//a[text()='%s']" %(user))[0]
                user_link.click()

    # Get the lineup table after clicking on the user name
                tables = pd.read_html(driver.page_source)
                lineup = tables[1]

    #print (user)
    #print (lineup)

    # Restructure to put into resutls dataframe
                lineup.loc[9, 'Name'] = lineup.iloc[9]['Salary']
                lineup.loc[10, 'Name'] = lineup.iloc[9]['Pts']

                temp_df = pd.DataFrame(lineup['Name'].values.reshape(-1, 11), 
                columns=lineup['Pos'].iloc[:9].tolist() + ['Total_$', 'Total_Pts'] )

                temp_df.insert(loc=0, column = 'User', value = user)
                temp_df.insert(loc=0, column = 'Rank', value = rank)
                temp_df["Date"]=d
                results = results.append(temp_df)
            #next_button = driver.find_elements_by_xpath("//button[@type='button']")
            #next_button[2].click()



            results = results.reset_index(drop=True)



driver.close()

However, there are other pages and to access it, you need to click on the small arrow next button at the bottom. Moreover, you can click indefinitely on that button; even if there are not more entries. Therefore, I would like to be able to loop through all pages with entries and stop when there are no more entries and change contest. I try to implement a while loop to do so, but my code did not work...

Answer 1

You must really make sure that page loads completely before you do anything on that page.

Moreover, it seems to work only for the first date in my calendar loop and always fails in the next iteration

Usually when selenium loads a browser page it tries to look for the element even if it is not loaded all the way. I suggest you to recheck the xpath of the element you are trying to click.

Also try to see when the page loads completely and use time.sleep(number of seconds) to make sure you hit the element or you can check for a particular element or a property of element that would let you know that the page has been loaded.

One more suggestion is that you can use driver.current_url to see which page are you targetting. I have had this issue while i was working on multiple tabs and I had to tell python/selenium to manually switch to that tab