Python Selenium - 獲取 Google 搜索 HREF

Question

我有兩個來自我的谷歌搜索站點的 href 值示例：linkedin.com/in/ AND “Software Developer” AND “London”：

<a href="https://uk.linkedin.com/in/roxana-andreea-popescu" data-ved="2ahUKEwjou5D9xeztAhUDoVwKHQStC5EQFjAAegQIARAC" ping="/url?sa=t&amp;source=web&amp;rct=j&amp;url=https://uk.linkedin.com/in/roxana-andreea-popescu&amp;ved=2ahUKEwjou5D9xeztAhUDoVwKHQStC5EQFjAAegQIARAC"><br><h3 class="LC20lb DKV0Md"><span>Roxana Andreea Popescu - Software Developer - Gumtree ...</span></h3><div class="TbwUpd NJjxre"><cite class="iUh30 Zu0yb qLRx3b tjvcx">uk.linkedin.com<span class="dyjrff qzEoUe"><span> › roxana-andreea-popescu</span></span></cite></div></a>

<a href="https://uk.linkedin.com/in/tunjijabitta" data-ved="2ahUKEwi-tsulxuztAhXViVwKHX0HAOMQFjABegQIBBAC" ping="/url?sa=t&amp;source=web&amp;rct=j&amp;url=https://uk.linkedin.com/in/tunjijabitta&amp;ved=2ahUKEwi-tsulxuztAhXViVwKHX0HAOMQFjABegQIBBAC"><br><h3 class="LC20lb DKV0Md"><span>Tunji Jabitta - London, Greater London, United Kingdom ...</span></h3><div class="TbwUpd NJjxre"><cite class="iUh30 Zu0yb qLRx3b tjvcx">uk.linkedin.com<span class="dyjrff qzEoUe"><span> › tunjijabitta</span></span></cite></div></a>

我正在創建一個 LinkedIn 刮板，但在獲取每個結果的 href 值（它們都不同）時遇到了問題，因此我可以遍歷它們。

我試過了

     linkedin_urls = driver.find_elements_by_xpath('//div[@class="yuRUbf"]//a')
links = [linkedin_url.get_attribute('href') for linkedin_url in linkedin_urls]
for linkedin_url in linkedin_urls:
    driver.get(links)
    sleep(5)
    sel = Selector(text=driver.page_source)

但我得到錯誤A invalid argument: 'url' must be a string'

我嘗試過的另一種選擇是

linkedin_urls = driver.find_elements_by_xpath('//div[@class="yuRUbf"]//a[@href]')
for linkedin_url in linkedin_urls:
    url = linkedin_url.get_attribute("href")

    driver.get(url)
    sleep(5)
    sel = Selector(text=driver.page_source)  

我設法打開了第一個鏈接，但是在嘗試獲取另一個鏈接時出現錯誤url = linkedin_url.get_attribute("href")

任何幫助將不勝感激，我已經堅持了很長一段時間。

Answer 1

您的驅動程序正在打開指向新頁面的鏈接，但它出現了，正在丟棄上一頁。 您可能需要考慮在新選項卡或 window 中打開，然后切換到該選項卡/窗口，一旦完成，go 返回上一頁並繼續。

建議執行：

1. 創建 function 以在新選項卡中打開鏈接（或元素） - 並切換到該選項卡：

from selenium.webdriver.common.action_chains import ActionChains

# Define a function which opens your element in a new tab:
def open_in_new_tab(driver, element):
    """This is better than opening in a new link since it mimics 'human' behavior"""
    # What is the handle you're starting with
    base_handle = driver.current_window_handle    

    ActionChains(driver) \
        .move_to_element(element) \
        .key_down(Keys.COMMAND) \
        .click() \
        .key_up(Keys.COMMAND) \
        .perform()

    # There should be 2 tabs right now... 
    if len(driver.window_handles)!=2:
        raise ValueError(f'Length of {driver.window_handles} != 2... {len(driver.window_handles)=};')

    # get the new handle
    for x in driver.window_handles:
        if x!= base_handle:
                new_handle = x
    # Now switch to the new window
    driver.switch_to.window(new_handle)

2.執行+切換回主選項卡：

import time

# This returns a list of elements
linkedin_urls = driver.find_elements_by_xpath('//div[@class="yuRUbf"]//a[@href]')

# A bit redundant, but it's web scraping, so redundancy won't hurt you.
BASE_HANDLE = driver.current_window_handle # All caps so you can follow it more easily...

for element in linkedin_urls:
    # switch to the new tab:
    open_in_new_tab(driver, element)
    
    # give the page a moment to load:
    time.sleep(0.5)
        
    # Do something on this page
    print(driver.current_url

    # Once you're done, get back to the original tab
    # Go through all tabs (there should only be 2) and close each one unless
    # it's the "base_handle"

    for x in driver.window_handles:
        if x!= base_handle:
            driver.switch_to.window(x)
            driver.close()
    # Now switch to the new window
    assert BASE_HANDLE in driver.window_handles # a quick sanity check
    driver.switch_to.window(BASE_HANDLE) # this takes you back

# Finally, once you for-loop is complete, you can choose to continue with the driver or close + quit (like a human would)
driver.close()
driver.quit()