使用 Selenium 抓取 LinkedIn 个人资料

Question

I am trying to scrape linkedin profiles, however when I get profile URLs they are duplicated, because one url can be located in several classes or tags. Could you please suggest how to find only one copy of URL for each profile. Thanks.

options = Options()
options.add_argument("--start-maximized")


url = "https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin"
driver = webdriver.Chrome(r"path", options=options)

driver.get(url)
driver.find_element_by_id('username').send_keys('email')
driver.find_element_by_id('password').send_keys('pass', Keys.ENTER)
sleep(10)
driver.find_element_by_class_name('search-global-typeahead__input').send_keys('CEO', Keys.ENTER)
driver.implicitly_wait(10)
driver.find_element_by_xpath('//button[text()="Люди"]').click()

x = 0
linklist = []
driver.execute_script("window.scrollTo(0, 1300);")
driver.implicitly_wait(10)
links = driver.find_elements_by_xpath('//a[contains(@href, "/in/")]')

for i in links:
      sleep(2)
      link = i.get_attribute('href')
      linklist.append(link)
print(linklist)

Answer 1

If you have duplicated values in your linklist , you can get the unique values by converting them into a set .

linklist=list(set(linklist))

EDIT:

You are getting duplicate links because you are searching the enite website for links and as you mentioned, these are present in different elements. You can get unique links by first searching for the name title of each member.

options = Options()
options.add_argument("--start-maximized")


url = "https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin"
driver = webdriver.Chrome(r"path", options=options)

driver.get(url)
driver.find_element_by_id('username').send_keys('email')
driver.find_element_by_id('password').send_keys('pass', Keys.ENTER)
sleep(10)
driver.find_element_by_class_name('search-global-typeahead__input').send_keys('CEO', Keys.ENTER)
driver.implicitly_wait(10)
driver.find_element_by_xpath('//button[text()="Люди"]').click()
sleep(5) #Wait for enitre page to load
linkedin_members = driver.find_elements_by_xpath('//span[@class="entity-result__title"]')

You can then loop through the name titles and select the href within the element (note the . in .//a[@class="app-aware-link] . You could use a try/except statement to find all the hrefs of non-hidden profiles using .//a[contains(@href, "/in/")] , but if that element doesn't exist, it takes selenium a while to figure that out. It is faster to select all hrefs and filter the hidden profiles out afterwards.

linklist = [linkedin_member.find_element_by_xpath('.//a[@class="app-aware-link"]').get_attribute('href') for linkedin_member in linkedin_members if "/in/" in linkedin_member.find_element_by_xpath('.//a[@class="app-aware-link"]').get_attribute('href')]