如何处理硒中的嵌套循环

Question

I am hoping someone can help me handle nested loop in selenium. I am trying to Scrape a website using selenium, it happens that i have to scrape multiple information with different links.

So i got all the links and looped through each, but in the process, the first link only displayed the items i needed, then the code breaks.

 def get_financial_info(self):

    chrome_options = Options()
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--window-size=1920x1080")
    driver = webdriver.Chrome(chrome_options=chrome_options, executable_path='/home/miracle/chromedriver')

    driver.get("https://www.financialjuice.com")

    try:
        WebDriverWait(driver, 60).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='trendWrap']")))
    except TimeoutException:
        driver.quit()

    category_url = driver.find_elements_by_xpath("//ul[@class='nav navbar-nav']/li[@class='text-uppercase']/a[@href]")
    for record in category_url:
        driver.get(record.get_attribute("href"))
        news = {}
        title_element = driver.find_elements_by_xpath("//p[@class='headline-title']")

        for news_record in title_element:
            news['title'] = news_record.text

            print news

Answer 1

Your category_url will be valid only on page where you've defined it and after first re-direction to another page it becomes stale...

You need to replace

category_url = driver.find_elements_by_xpath("//ul[@class='nav navbar-nav']/li[@class='text-uppercase']/a[@href]")

with

category_url = [a.get_attribute("href") for a in driver.find_elements_by_xpath("//ul[@class='nav navbar-nav']/li[@class='text-uppercase']/a")]

and then loop through the list of links as

for record in category_url:
    driver.get(record)