使用 Selenium 登录后使用 Pandas 和 Beautiful Soup 从表中收集数据

Question

I'm trying to scrape data from a paginated table. The table can only be accessed by logging in to a user account. I've decided to approach this using Selenium to log in. I then hope to be able to read this into a Pandas DataFrame. I plan on using BeautifulSoup as a go between.

Here is my code:

from selenium import webdriver
import time
import pandas as pd

from bs4 import BeautifulSoup

url = "https://www.example.com/userarea"

driver = webdriver.Chrome()
time.sleep(6)
driver.get(url)
time.sleep(6)

username = driver.find_element_by_id("user")
username.clear()
username.send_keys("xyz@email.com")

password = driver.find_element_by_id("password")
password.clear()
password.send_keys('password')

driver.find_element_by_xpath('//button[]').click()
driver.find_element_by_xpath('//button[text()="Log in"]').click()
time.sleep(6)

driver.find_element_by_xpath('//span[text()="Text"]').click()

driver.find_element_by_xpath('//span[text()="Text"]').click()

html = driver.page_source
soup = BeautifulSoup(html,'html.parser') 

try:
    tables = soup.find_all('th')
    print(tables) #Returns an empty list
    df = pd.read_html(str(tables))

    df.head()

except:
    driver.close()
driver.close()

Unfortunately, this is only printing an empty list. I've tried using lxml too but no joy. Using the inspection tools it does seem that there aren't any table tags, so I tried to find all <th> tags instead (which definitely are present). Again no joy. I've not yet tried to work through the individual pages. I only mention the pagination in case it offers a clue to the issue.

Any idea what I'm missing?

Answer 1

Thank you to those that offered suggestions. In the end furas' suggestion was best placed and it turned out the script was running too quickly. I paused Python for 6 seconds after clicking on the page with the table on. Seems to run on javascript and I can actually see the values pop into place now as the script works through the pagination.

import time

#Navigate to page, then let it load using:

time.sleep(6)