Python Selenium - Get Google search HREF

Question

I have two examples of href values from my google search site:linkedin.com/in/ AND "Software Developer" AND "London":

<a href="https://uk.linkedin.com/in/roxana-andreea-popescu" data-ved="2ahUKEwjou5D9xeztAhUDoVwKHQStC5EQFjAAegQIARAC" ping="/url?sa=t&amp;source=web&amp;rct=j&amp;url=https://uk.linkedin.com/in/roxana-andreea-popescu&amp;ved=2ahUKEwjou5D9xeztAhUDoVwKHQStC5EQFjAAegQIARAC"><br><h3 class="LC20lb DKV0Md"><span>Roxana Andreea Popescu - Software Developer - Gumtree ...</span></h3><div class="TbwUpd NJjxre"><cite class="iUh30 Zu0yb qLRx3b tjvcx">uk.linkedin.com<span class="dyjrff qzEoUe"><span> › roxana-andreea-popescu</span></span></cite></div></a>

<a href="https://uk.linkedin.com/in/tunjijabitta" data-ved="2ahUKEwi-tsulxuztAhXViVwKHX0HAOMQFjABegQIBBAC" ping="/url?sa=t&amp;source=web&amp;rct=j&amp;url=https://uk.linkedin.com/in/tunjijabitta&amp;ved=2ahUKEwi-tsulxuztAhXViVwKHX0HAOMQFjABegQIBBAC"><br><h3 class="LC20lb DKV0Md"><span>Tunji Jabitta - London, Greater London, United Kingdom ...</span></h3><div class="TbwUpd NJjxre"><cite class="iUh30 Zu0yb qLRx3b tjvcx">uk.linkedin.com<span class="dyjrff qzEoUe"><span> › tunjijabitta</span></span></cite></div></a>

I am creating a LinkedIn scraper and I am having a problem when it comes to getting the href value (Which all differ) for each of the results so I can loop through them.

I tried

     linkedin_urls = driver.find_elements_by_xpath('//div[@class="yuRUbf"]//a')
links = [linkedin_url.get_attribute('href') for linkedin_url in linkedin_urls]
for linkedin_url in linkedin_urls:
    driver.get(links)
    sleep(5)
    sel = Selector(text=driver.page_source)

But I get the errror A invalid argument: 'url' must be a string'

Another alternative I have tried was

linkedin_urls = driver.find_elements_by_xpath('//div[@class="yuRUbf"]//a[@href]')
for linkedin_url in linkedin_urls:
    url = linkedin_url.get_attribute("href")

    driver.get(url)
    sleep(5)
    sel = Selector(text=driver.page_source)  

I managed to get the first link opened but it through an error url = linkedin_url.get_attribute("href") when trying to get the other link

Any help would be greatly appreciated, I have been stuck on this for quite a while.

Answer 1

Your driver is opening the link to the new page but it appears, is discarding the previous page. You may want to consider opening in a new tab or window, then switching to that tab/window, once complete, go back to previous page and continue.

Suggested execution:

1. Create a function to open link (or element) in a new tab – and to switch to that tab:

from selenium.webdriver.common.action_chains import ActionChains

# Define a function which opens your element in a new tab:
def open_in_new_tab(driver, element):
    """This is better than opening in a new link since it mimics 'human' behavior"""
    # What is the handle you're starting with
    base_handle = driver.current_window_handle    

    ActionChains(driver) \
        .move_to_element(element) \
        .key_down(Keys.COMMAND) \
        .click() \
        .key_up(Keys.COMMAND) \
        .perform()

    # There should be 2 tabs right now... 
    if len(driver.window_handles)!=2:
        raise ValueError(f'Length of {driver.window_handles} != 2... {len(driver.window_handles)=};')

    # get the new handle
    for x in driver.window_handles:
        if x!= base_handle:
                new_handle = x
    # Now switch to the new window
    driver.switch_to.window(new_handle)

2. Execute + Switch back to the main tab:

import time

# This returns a list of elements
linkedin_urls = driver.find_elements_by_xpath('//div[@class="yuRUbf"]//a[@href]')

# A bit redundant, but it's web scraping, so redundancy won't hurt you.
BASE_HANDLE = driver.current_window_handle # All caps so you can follow it more easily...

for element in linkedin_urls:
    # switch to the new tab:
    open_in_new_tab(driver, element)
    
    # give the page a moment to load:
    time.sleep(0.5)
        
    # Do something on this page
    print(driver.current_url

    # Once you're done, get back to the original tab
    # Go through all tabs (there should only be 2) and close each one unless
    # it's the "base_handle"

    for x in driver.window_handles:
        if x!= base_handle:
            driver.switch_to.window(x)
            driver.close()
    # Now switch to the new window
    assert BASE_HANDLE in driver.window_handles # a quick sanity check
    driver.switch_to.window(BASE_HANDLE) # this takes you back

# Finally, once you for-loop is complete, you can choose to continue with the driver or close + quit (like a human would)
driver.close()
driver.quit()