如何在Python中使用Selenium从具有隐藏元素的动态折叠表中提取数据

Question

I try to scrape these 20 classifications from https://patents.google.com/patent/JP2009517369A/en?oq=JP2009517369 , from which the first is displayed and the others are hidden in an expandable section.

I already tried to get the first visible one with

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='style-scope classification-tree' and not(@hidden)]/state-modifier[@class='code style-scope classification-tree']/a[@class='style-scope state-modifier']"))).get_attribute("innerHTML") 

However, it raises an exception and I don't know why. So I figured that scraping the whole table would be easier but most of the elements are folded.

Is there any approach on how to scrape dynamic hidden tables? Thank you for your help!

Answer 1

The First two options should print the value C07C311/51

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='style-scope classification-tree' and not(@hidden)]/state-modifier[@class='code style-scope classification-tree']/a[@class='style-scope state-modifier']"))).text)

OR

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='style-scope classification-tree' and not(@hidden)]/state-modifier[@class='code style-scope classification-tree']/a[@class='style-scope state-modifier']"))).get_attribute("innerHTML"))

However if you do not get the expected value try the last one.this should print any hidden content.

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='style-scope classification-tree' and not(@hidden)]/state-modifier[@class='code style-scope classification-tree']/a[@class='style-scope state-modifier']"))).get_attribute("textContent"))