Appending Dataframe Inside Loop

Question

I'm trying to get a list of links by looping through elements and then clicking through pagination. I'm not sure how to append each loop in the pandas dataframe after it goes through pagination shown below so I can call the dataframe outside the loop to list all the links.

It always overwrites and prints out the last line.

while True:

    links = [link.get_attribute('href') for link in driver.find_elements_by_class_name('view-detail-link')]

    for link in links:

        df_links = pd.DataFrame([[link]], columns=['link'])

    try:

        NextPage = driver.find_element_by_xpath('//a[@class="ui-pagination-next ui-goto-page"]')
        driver.execute_script("arguments[0].click();", NextPage)

        time.sleep(3)

    except NoSuchElementException:
        break

print(df_links.link[0])

Answer 1

You need to create your DataFrame outside the loop. Then each time you create a new DataFrame in the loop you append it to the main one:

df = pd.DataFrame()

while True:

    links = [link.get_attribute('href') for link in driver.find_elements_by_class_name('view-detail-link')]

    for link in links:

        df_links = pd.DataFrame([[link]], columns=['link'])
        df = df.append(df_links)

    try:

        NextPage = driver.find_element_by_xpath('//a[@class="ui-pagination-next ui-goto-page"]')
        driver.execute_script("arguments[0].click();", NextPage)

        time.sleep(3)

    except NoSuchElementException:
        break

print(df.link[0])