检索搜索结果selenium python bs4

Question

I successfully put together a script to retrieve search results from sales navigator in Linkedin. The following is the script, using python, selenium, and bs4.

browser = webdriver.Firefox(executable_path=r'D:\geckodriver\geckodriver.exe')
url1 = "https://www.linkedin.com/sales/search/company?companySize=E&geoIncluded=emea%3A0%2Ceurope%3A0&industryIncluded=6&keywords=AI&page=1&searchSessionId=zreYu57eQo%2BSZiFskdWJqg%3D%3D"

browser.get(url1)
time.sleep(15)

parsed = browser.find_element_by_tag_name('html').get_attribute('innerHTML')
soup = BeautifulSoup(parsed, 'html.parser')

search_results = soup.select('dt.result-lockup__name a')
print(len(search_results))

time.sleep(5)
browser.quit()

Irrespective of the no.of results, the answer was always 10 (ie) only 10 results were returned. Upon further investigation into the source, I noticed the following :

That the first 10 results are represented at a different level and the rest are under a div tag with style class named as deferred area . Though the dt class name is the same for all the search results (result-lockup__name) , due to the change in levels, I am not able to access/retrieve it.

What would be the right way to retrieve all results in such a case?

EDIT 1

An example of how the tag levels are within li

And an example of the html script of the result that is not being retrieved

EDIT 2

The page source as requested

https://pastebin.com/D11YpHGQ

Answer 1

A lot of sites don't display all search results on page load rather only display them when needed, eg the visitor keeps scrolling indicating they want to view more.

We can use javascript to scroll to the bottom of the page for us window.scrollTo(0,document.body.scrollHeight) , (you may want to loop this if you expect hundreds of results) forcing all results on the page, after which we can grab the HTML.

Below should do the trick.

browser = webdriver.Firefox(executable_path=r'D:\geckodriver\geckodriver.exe')
url1 = "https://www.linkedin.com/sales/search/company?companySize=E&geoIncluded=emea%3A0%2Ceurope%3A0&industryIncluded=6&keywords=AI&page=1&searchSessionId=zreYu57eQo%2BSZiFskdWJqg%3D%3D"

browser.get(url1)
time.sleep(15)
browser.execute_script('window.scrollTo(0,document.body.scrollHeight)')
time.sleep(15)

parsed = browser.find_element_by_tag_name('html').get_attribute('innerHTML')
soup = BeautifulSoup(parsed, 'html.parser')

search_results = soup.select('dt.result-lockup__name a')
print(len(search_results))