如何通过Python使用ChromeDriver Chrome和Selenium在页面上打印类别链接？

Question

Using Python3 I am trying to get Chrome Webdriver and Selenium to identify the various 'Classifieds' categories on the webpage www.jtinsight.com and from there print out the category titles. So far using the code below the best i can do is get it to print out the first two - 'All categories' and 'Cars(Private)'. I have identified that the html for these two is different to the others and tried a number of different lines of code which i have listed in the code commented out, but cant identify the right tag/class/xpath etc. Any help would be appreciated.

from selenium import webdriver
from selenium.webdriver.common.by import By

# Creating the WebDriver object using the ChromeDriver
driver = webdriver.Chrome()

# Directing the driver to the defined url
driver.get("https://www.jtinsight.com/JTIRA/JTIRA.aspx#!/main")

# Locate the categories

# Each code line runs but only returns the first two categories
# categories = driver.find_elements_by_xpath('//div[@class="col-md-3 col-sm-4 col-xs-6"]')
# categories = driver.find_elements_by_xpath('//div[@class="mainCatEntry"]')
# categories = driver.find_elements_by_xpath('//div[@class="Description"]')

# Process ran but finished with exit code 0
# categories = driver.find_elements_by_xpath('//*[@class="col-md-3 col-sm-4 col-xs-6 ng-scope"]')
# categories = driver.find_elements_by_xpath('//div[@class="col-md-3 col-sm-4 col-xs-6 ng-scope"]')
# categories = driver.find_elements_by_partial_link_text('//href[@class="divLink"]')
# categories = driver.find_elements_by_tag_name('bindonce')
# categories = driver.find_elements_by_xpath('//div[@class="divLink"]')

# Error before finished running
# categories = driver.find_elements(By.CLASS_NAME, "col-md-3 col-sm-4 col-xs-6 ng-scope")
# categories = driver.find_elements(By.XPATH, '//div bindonce[@class="col-md-3 col-sm-4 col-xs-6 ng-scope"]')
# categories = driver.find_elements_by_class_name('//div bindonce[@class="col-md-3 col-sm-4 col-xs-6 ng-scope"]')

# Print out all categories on current page
num_page_items = len(categories)
print(num_page_items)
for i in range(num_page_items):
    print(categories[i].text)

# Clean up (close browser once task is completed.)
driver.close()

Answer 1

This is really a timing issue. If I added a "sleep(5)" before collecting categories, it found all 24. Interestingly, when I used a WebDriverWait instead, it would still only pull up 2 items. So, to force the driver to do a little more work I extended the xpath. the following worked for me:

categories = WebDriverWait(driver, 10).until(
EC.visibility_of_all_elements_located((By.XPATH, '//div[@class="mainCatEntry"]/div[@class="Description"]')))

Answer 2

To identify the various Classifieds categories on the webpage https://www.jtinsight.com/JTIRA/JTIRA.aspx#!/main and print the category titles, eg All categories , Cars (Private) , etc, you need to scroll down a bit and induce WebDriverWait for the visibility_of_all_elements_located() and you can use the following solution:

• Code Block:

   from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC options = webdriver.ChromeOptions() options.add_argument("start-maximized") options.add_argument('disable-infobars') driver=webdriver.Chrome(chrome_options=options, executable_path=r'C:\\Utility\\BrowserDrivers\\chromedriver.exe') driver.get("https://www.jtinsight.com/JTIRA/JTIRA.aspx#!/main") driver.execute_script("arguments[0].scrollIntoView(true);",WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.XPATH, "//span[@class='ng-scope' and text()='Classifieds']")))); print([elem.get_attribute("innerHTML") for elem in WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='mainCatEntry']//div[@class='Description']")))])