Update on Using Selenium To Scrape Java Heavy Websites in Python

Question

My first bit of code looked a little like this,

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://explorer.helium.com/accounts/13pm9juR7WPjAf7EVWgq5EQAaRTppu2EE7ReuEL9jpkHQMJCjn9")

earnings = driver.find_elements_by_class_name('text-base text-gray-600 mb-1 tracking-tight w-full break-all')

print(earnings)

driver.quit()

Now I have gotten to the point of adding wait times, but still returns as nothing with this code here.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(ChromeDriverManager().install())

driver = webdriver.Chrome()
driver.get("https://explorer.helium.com/accounts/13pm9juR7WPjAf7EVWgq5EQAaRTppu2EE7ReuEL9jpkHQMJCjn9")
try:
    element = WebDriverWait(driver, 60).until(EC.presence_of_element_located((By.XPATH, ".//*[@id='app']/article/div[2]/div/div[2]/div/div[2]/div[3]/div[1]/div[1]/div[3]")))
finally:
    earnings = driver.find_elements_by_xpath('.//*[@id="app"]/article/div[2]/div/div[2]/div/div[2]/div[3]/div[1]/div[1]/div[3]')
    print(earnings)
    print("loaded")
    driver.quit()

I am just trying to scrape the text with the dollar amount on it in this container Image of Container

Would love some further help on this problem I am having.

Answer 1

As already explained find_elements will return List of WebElements and access those you can use indexing - earnings[0] .

from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(ChromeDriverManager().install())

driver.get("https://explorer.helium.com/accounts/13pm9juR7WPjAf7EVWgq5EQAaRTppu2EE7ReuEL9jpkHQMJCjn9")
try:
    element = WebDriverWait(driver, 60).until(EC.presence_of_element_located((By.XPATH, ".//*[@id='app']/article/div[2]/div/div[2]/div/div[2]/div[3]/div[1]/div[1]/div[3]")))
finally:
    earnings = driver.find_elements_by_xpath('.//*[@id="app"]/article/div[2]/div/div[2]/div/div[2]/div[3]/div[1]/div[1]/div[3]')
    print(earnings[0].text)
    print("loaded")
driver.quit()

And find_element will return a WebElement and access the same:

try:
    element = WebDriverWait(driver, 60).until(EC.presence_of_element_located((By.XPATH, ".//*[@id='app']/article/div[2]/div/div[2]/div/div[2]/div[3]/div[1]/div[1]/div[3]")))
finally:
    earnings = driver.find_element_by_xpath('.//*[@id="app"]/article/div[2]/div/div[2]/div/div[2]/div[3]/div[1]/div[1]/div[3]')
    print(earnings.text)
    print("loaded")

For both the output was:

$14.08
loaded