bs4没有得到所有列表项

Question

Using the following url as an example, the code only gets 35 items instead of the 85 listed on the page. Is this a case of having to use selenium to load the view-source page? How could bs4 miss the rest of the li items?

r = requests.get(url=url)
soup = bs(r.text, 'html.parser')
jobkeys = []
jobs = soup.findAll("li", {"class": "cmp-JobListItem"})
for job in jobs:
    s = job.attrs.get('data-tn-entityid')
    jobkey = s[s.find(',')+1:s.rfind(',')]
    jobkeys.append(jobkey)

Edit:

Using selenium I was able to "see" what was going on when the page loaded. The URL automatically redirected to a prefiltered mobile site.

With this new site I could remove the filter, get the new url and obtain the correct number.

Thanks!

Answer 1

I have to be honest: I tried your code as you wrote it several times and I have got a list of 85 items: no more, nor less. So I don't know exactly how to answer the second question, but I can answer the first one: no, you don't have to use other other packages to achieve what you want: the problem is elsewhere. Just to be sure, I'm going to copypaste here the full code I just run:

from bs4 import BeautifulSoup as bs
import requests

r = requests.get(url='https://ca.indeed.com/cmp/Abb/jobs')
soup = bs(r.text, 'html.parser')
jobkeys = []
jobs = soup.findAll("li", {"class": "cmp-JobListItem"})
for job in jobs:
    s = job.attrs.get('data-tn-entityid')
    jobkey = s[s.find(',')+1:s.rfind(',')]
    jobkeys.append(jobkey)

print(len(jobkeys))

Output:

85