从具有 Selenium python 的变量中提取 JavaScript 值时出错

Question

I'm attempting to get the extract the value of the variable remainingTimeString on this website in python using Selenium webdriver. I'm trying to use the driver.execute_script() function. Here is my code:

import selenium.webdriver

options = selenium.webdriver.FirefoxOptions()
options.add_argument("--headless")

driver = selenium.webdriver.Firefox(options=options)

driver.get('https://shopgoodwill.com/item/151632327')
print(driver.execute_script("return remainingTimeString"))

However, when I run it, I get:

selenium.common.exceptions.JavascriptException: Message: ReferenceError: remainingTimeString is not defined

What should I do? the variable is clearly in a script when I check the HTML source. Thanks!

Answer 1

That data is being pulled dynamically from an API, after page loads, so your options are - either use a WebDriverWait for that element (or an implicit wait for that matter), or use a less complex solution, like below (without selenium), where you inspect the Network Tab in Dev Tools, locate the API where data is being pulled from, and scrape that API directly:

import requests
import pandas as pd


headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
}

url = 'https://buyerapi.shopgoodwill.com/api/ItemDetail/GetItemDetailModelByItemId/151632327'

r = requests.get(url, headers=headers)
df = pd.json_normalize(r.json())
print(df['remainingTime'][0])

Result printed in terminal:

'17h 56m '

The code above is extracting remainingTime only from that dataframe. There are detailed product infos in that json, so you can get other data as well, if you need. Python requests documentation can be found at https://requests.readthedocs.io/en/latest/

Also pandas documentation: https://pandas.pydata.org/pandas-docs/stable/index.html