使用 selenium python 访问网页上的所有链接和子链接

Question

i have a webpage starts with 4 links and each link has 2 or more links and those links also has 1 or 2 links and so on, So how can i visit all links and nested links using selenium and python?

all links has same relative xpath

i have tried below code but it's not working

urls = {}

def visit_children(locator_path):
    children = get_children(locator_path)
    time.sleep(1)
    if children == 1:        
        click_func(locator_path)
        visit_children(locator_path)
    elif children > 1:
        print(children)
        time.sleep(2)
        url = driver.current_url
        print(url)
        urls[url] = children
        print(urls)
        for i in range(children):
            child_elements = driver.find_elements_by_xpath(locator_path)
            child_elements[i].click()
            time.sleep(2)
            visit_children(locator_path)                
    else:
         for link,no_elements in urls.items():
                if urls[driver.current_url] > 0:
                    driver.get(link)
                    time.sleep(1)
                    urls[driver.current_url] -= 1
                    print(urls)
                    time.sleep(2)

Answer 1

I think what you want to do is to implement a crawler, to do that you will need 2 data structures there, one to tell you what urls you have already visited and another where you just dump extract urls based in your criteria.

The crawler function just needs to pop the first URL from the of urls, check if it's a URL that you have already crawled and if not crawl it.

Something like this:

visited = {}
urls = ['initial_url']  

while len(urls) > 0:
  url = urls.pop()
  if visited[url] == 1:
    continue
  crawl(url)

Notice that set's have a O(1) complexity so you might want to use them to quickly confirm if you have already visited a url or not, while lists (FIFO) while be a awesome way to store the extracted urls.