使用 BeautifulSoup 抓取求职网站

Question

I am trying to scrape all the complete job descriptions from this website but I got stuck: https://www.seek.co.nz/data-analyst-jobs/full-time?daterange=31&salaryrange=70000-999999&salarytype=annual

My logic is to find all the job links on one page first than loop through next pages.

My code looks like this:

from bs4 import BeautifulSoup
import requests

soup = BeautifulSoup(response.text, 'html.parser')
link_list = []

for a in soup.find_all('a', attrs={'data-automation': 'jobTitle'}, href=True):
    link_list.append('https://www.seek.co.nz/' + a['href'])
print(link_list)

The above code looks fine. I can print a list of job links and put them in a list, but the following code only printed out 2 paragraphs then it threw an error:

for link in link_list:
    response = requests.get(link, 'lxml')
    sp = BeautifulSoup(response.text, 'html.parser')
    table = sp.find_all('div',attrs={'data-automation': 'jobDescription'})
    for x in table:
        print(x.find('p').text)

---

AttributeError                            Traceback (most recent call last)
<ipython-input-41-8afe949a9497> in <module>()
      4     table = sp.find_all('div',attrs={'data-automation': 'jobDescription'})
      5     for x in table:
----> 6         print(x.find('p').text)

AttributeError: 'NoneType' object has no attribute 'text'

Can someone tell me why it didn't work and how to make it right? I am using Python 3 and bs4. Thank you!

Answer 1

Going through the webpage , it is clear that the first two posts that you'll get through soup.find_all('a', attrs={'data-automation': 'jobTitle'}, href=True) are advertisements or promoted posts, and consequently the webpages these two and the other job postings lead to are different. For the latter, perhaps soup.find('div', attrs={'class':'templatetext'}) will do the job.

Hope that helps!

Answer 2

you have to use find_all

for link in link_list:
    response = requests.get(link, 'lxml')
    sp = BeautifulSoup(response.text, 'html.parser')
    table = sp.find_all('div',attrs={'data-automation': 'jobDescription'})
    for x in table:
        all_p = x.find_all('p')
        for para in all_p:
            if para:
                print(para.text)