简体   繁体   中英

Scraping Linkedin SalesNavigator with Selenium/BS on Python

I am trying to scrape the results of a Sales Navigator link using Selenium and BS on python. I first sign in to Linkedin and then use the driver.get(link) to access the SalesNav search URL in question. Unfortunately, the link redirects me to https://www.linkedin.com/sales/contract-chooser for no particular reason and loads indefinitely, if I simply copy paste the link on my browser, it works well. Here is my code:


# Sign in to LinkedIn account
driver = webdriver.Chrome("/Applications/chromedriver")
driver.get("https://linkedin.com/uas/login")
time.sleep(5)
  
username = driver.find_element_by_id("username")
username.send_keys("EMAIL")  # Enter Your Email Address
  
pword = driver.find_element_by_id("password")
pword.send_keys("PASSWORD")


driver.find_element_by_xpath("//button[@type='submit']").click()


driver.maximize_window()
time.sleep(10)

# Redirect to Page
link="https://www.linkedin.com/sales/search/people#logHistory=true&query=(recentSearchParam%3A(id%3A1560694714%2CdoLogHistory%3Atrue)%2Cfilters%3AList((type%3AREGION%2Cvalues%3AList((id%3A101282230%2Ctext%3AGermany%2CselectionType%3AINCLUDED)))%2C%2C%2C(type%3ACURRENT_COMPANY%2Cvalues%3AList((id%3A109842%2Ctext%3Aneckermann.de%20GmbH%2CselectionType%3AINCLUDED)%2C(id%3A12756300%2Ctext%3AFRIEDRICH%20VORWERK%20Unternehmensgruppe%2CselectionType%3AINCLUDED)%2C(id%3A9837432%2Ctext%3ABuffalo%20Boots%20GmbH%2CselectionType%3AINCLUDED)%2C(id%3A12481732%2Ctext%3AYazaki%20Systems%2CselectionType%3AINCLUDED)%2C(id%3A5860104%2Ctext%3ATurck%20Beierfeld%20GmbH%2CselectionType%3AINCLUDED)%2C(id%3A36436702%2Ctext%3ALUX%20Automation%20GmbH%2CselectionType%3AINCLUDED)%2C(id%3A41395024%2Ctext%3AGEDIA%20Gebr%C3%BCder%20DINGERKUS%20GmbH%2CselectionType%3AINCLUDED)%2C(id%3A41053846%2Ctext%3APAUL%20WURTH%20DEUTSCHLAND%20GMBH%2CselectionType%3AINCLUDED)%2C(id%3A3239211%2Ctext%3AArcandor%20AG%2CselectionType%3AINCLUDED)%2C(id%3A36565653%2Ctext%3ASport%20Voswinkel%20GmbH%20%26%20Co.%20KG%2CselectionType%3AINCLUDED)%2C(id%3A37277217%2Ctext%3Aaccredis%20Inkasso%20GmbH%20%26%20Co.%20KG%2CselectionType%3AINCLUDED)%2C(id%3A69449704%2Ctext%3ASch%C3%A4fer%20Informationstechnik%20GmbH%2CselectionType%3AINCLUDED)%2C(text%3AASSET%20Immobilienbeteiligungen%20GmbH%2CselectionType%3AINCLUDED)%2C(text%3AGothaer%20Systems%20GmbH%2CselectionType%3AINCLUDED)%2C(text%3AIFORM%20Spritzgu%C3%9Fformen%20f%C3%BCr%20die%20Kunststoffverarbeitung%20GmbH%2CselectionType%3AINCLUDED)))%2C(type%3ASENIORITY_LEVEL%2Cvalues%3AList((id%3A1%2Ctext%3AUnpaid%2CselectionType%3AEXCLUDED)%2C(id%3A2%2Ctext%3ATraining%2CselectionType%3AEXCLUDED)))%2C(type%3ATITLE%2Cvalues%3AList((text%3AAnwendung%2CselectionType%3AINCLUDED)%2C(text%3Aapplication%2CselectionType%3AINCLUDED)%2C(text%3Aarbeitsplatz%2CselectionType%3AINCLUDED)%2C(text%3Aautomation%2CselectionType%3AINCLUDED)%2C(text%3AAutomatisierung%2CselectionType%3AINCLUDED)%2C(text%3ABeachtung%2CselectionType%3AINCLUDED)%2C(text%3Acdo%2CselectionType%3AINCLUDED)%2C(text%3Acio%2CselectionType%3AINCLUDED)%2C(text%3Aciso%2CselectionType%3AINCLUDED)%2C(text%3Acloud%2CselectionType%3AINCLUDED)%2C(text%3Acollaboration%2CselectionType%3AINCLUDED)%2C(text%3Acompliance%2CselectionType%3AINCLUDED)%2C(text%3Acontainer%2CselectionType%3AINCLUDED)%2C(text%3Acontainer-orchestration%2CselectionType%3AINCLUDED)%2C(text%3Acontainer-orchestrierung%2CselectionType%3AINCLUDED)%2C(text%3Acontainerisieren%2CselectionType%3AINCLUDED)%2C(text%3Acontainerize%2CselectionType%3AINCLUDED)%2C(text%3Acto%2CselectionType%3AINCLUDED)%2C(text%3Adata%2CselectionType%3AINCLUDED)%2C(text%3Adaten%2CselectionType%3AINCLUDED)%2C(text%3Adatenschutz%2CselectionType%3AINCLUDED)%2C(text%3Adigital%2CselectionType%3AINCLUDED)%2C(text%3Adigital%2520transformation%2CselectionType%3AINCLUDED)%2C(text%3Adigitale%2520Transformation%2CselectionType%3AINCLUDED)%2C(text%3ADocker%2CselectionType%3AINCLUDED)%2C(text%3Afertigung%2CselectionType%3AINCLUDED)%2C(text%3AHerstellung%2CselectionType%3AINCLUDED)%2C(text%3Ahr%2CselectionType%3AINCLUDED)%2C(text%3Ahuman%2520resources%2CselectionType%3AINCLUDED)%2C(text%3AHumanressourcen%2CselectionType%3AINCLUDED)%2C(text%3AIndustrie%25204.0%2CselectionType%3AINCLUDED)%2C(text%3Aindustry%25204.0%2CselectionType%3AINCLUDED)%2C(text%3Ainformation%2CselectionType%3AINCLUDED)%2C(text%3AInformationen%2CselectionType%3AINCLUDED)%2C(text%3Ainnovation%2CselectionType%3AINCLUDED)%2C(text%3AInternet%2520der%2520Dinge%2CselectionType%3AINCLUDED)%2C(text%3Ainternet%2520of%2520things%2CselectionType%3AINCLUDED)%2C(text%3Aiot%2CselectionType%3AINCLUDED)%2C(text%3Ait%2CselectionType%3AINCLUDED)%2C(text%3AKubernetes%2CselectionType%3AINCLUDED)%2C(text%3ALeistung%2CselectionType%3AINCLUDED)%2C(text%3Amanufacturing%2CselectionType%3AINCLUDED)%2C(text%3AMenschen%2CselectionType%3AINCLUDED)%2C(text%3Anetwork%2CselectionType%3AINCLUDED)%2C(text%3Anetzwerk%2CselectionType%3AINCLUDED)%2C(text%3Apeople%2CselectionType%3AINCLUDED)%2C(text%3Aperformance%2CselectionType%3AINCLUDED)%2C(text%3APersonalwesen%2CselectionType%3AINCLUDED)%2C(text%3Aproduction%2CselectionType%3AINCLUDED)%2C(text%3Aproduktion%2CselectionType%3AINCLUDED)%2C(text%3Aprotection%2CselectionType%3AINCLUDED)%2C(text%3Aschutz%2CselectionType%3AINCLUDED)%2C(text%3Asecurity%2CselectionType%3AINCLUDED)%2C(text%3ASicherheit%2CselectionType%3AINCLUDED)%2C(text%3Asoftware%2CselectionType%3AINCLUDED)%2C(text%3Atechnologie%2CselectionType%3AINCLUDED)%2C(text%3Atechnology%2CselectionType%3AINCLUDED)%2C(text%3Atransformation%2CselectionType%3AINCLUDED)%2C(text%3AWolke%2CselectionType%3AINCLUDED)%2C(text%3Aworkplace%2CselectionType%3AINCLUDED)%2C(text%3Aprogram%2CselectionType%3AINCLUDED)%2C(text%3Aprokurist%2CselectionType%3AINCLUDED)%2C(text%3Asolution%2CselectionType%3AINCLUDED)%2C(text%3Acyber%2CselectionType%3AINCLUDED)%2C(text%3Aentwicklung%2CselectionType%3AINCLUDED)%2C(text%3Asupply%2520chain%2CselectionType%3AINCLUDED)%2C(text%3Ainfrastructure%2CselectionType%3AINCLUDED)%2C(text%3AInfrastruktur%2CselectionType%3AINCLUDED)%2C(text%3AVersorgungskette%2CselectionType%3AINCLUDED))%2CselectedSubFilter%3ACURRENT)%2C(type%3AFUNCTION%2Cvalues%3AList((id%3A13%2Ctext%3AInformation%2520Technology%2CselectionType%3AINCLUDED)%2C(id%3A12%2Ctext%3AHuman%2520Resources%2CselectionType%3AINCLUDED)%2C(id%3A21%2Ctext%3APurchasing%2CselectionType%3AINCLUDED)%2C(id%3A8%2Ctext%3AEngineering%2CselectionType%3AINCLUDED)%2C(id%3A19%2Ctext%3AProduct%2520Management%2CselectionType%3AINCLUDED)))))&rsLogId=1056057746&searchSessionId=HslSmbZJS06pJuDD8v5%2F0g%3D%3D&sessionId=7tfxUO0tQjSSh52UN4Uvww%3D%3D&viewAllFilters=true"


driver.get(link)

Selenium is used to click elements, input thngs in textboxes etc. If you want to scrape things you shouldn't use selenium but beautifulsoup instead.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM