仅从网站获取 JSON 的一部分，我正在尝试使用 Python、BeautifulSoup、请求来抓取。 从 62 条回复中获得 20 条回复

Question

I am trying to scrape this site for job openings:

https://recruiting.ultipro.com/UNI1029UNION/JobBoard/74c2a308-3bf1-4fb1-8a83-f92fa61499d3/?q=&o=postedDateDesc&w=&wc=&we=&wpst=

I looked in dev tools and saw that the page makes an XHR request to this site to retrieve the job opening(s) information which is in the form of a JSON object:

https://recruiting.ultipro.com/UNI1029UNION/JobBoard/74c2a308-3bf1-4fb1-8a83-f92fa61499d3/JobBoardView/LoadSearchResults

So I'm like "Great: I can parse this in two seconds using a python program like this":

''' from bs4 import BeautifulSoup import json import requests

def crawl():
    union = requests.get('https://recruiting.ultipro.com/UNI1029UNION/JobBoard/74c2a308-3bf1-4fb1-8a83-f92fa61499d3/JobBoardView/LoadSearchResults').content
    soup = BeautifulSoup(union, 'html.parser')
    newDict = json.loads(str(soup))
    for job in newDict['opportunities']:
        print(job['Title'])

crawl() '''

Well it turns out that this page only returns 20 job openings out of 62. So I went back to the page and loaded the entirety of the page (clicked "view more opportunities")

And it said that it sent another XHR request to that same link, yet only 20 records are shown when I look.

How can I scrape all of the records from this page? And if someone could explain what is going on behind the scenes that would be great. I am a little new to web scraping so any insight is appreciated.

Answer 1

You don't need do a scraping, like you say the API that return all json is the link https://recruiting.ultipro.com/UNI1029UNION/JobBoard/74c2a308-3bf1-4fb1-8a83-f92fa61499d3/JobBoardView/LoadSearchResults but you need set in body request this parameters

import requests

headers = {
    'Content-Type': 'application/json'
}

data = '{\n  "opportunitySearch": {\n    "Top": 62,\n    "Skip": 0,\n    "QueryString": "",\n    "OrderBy": [\n      {\n        "Value": "postedDateDesc",\n        "PropertyName": "PostedDate",\n        "Ascending": false\n      }\n    ],\n    "Filters": [\n      {\n        "t": "TermsSearchFilterDto",\n        "fieldName": 4,\n        "extra": null,\n        "values": [\n          \n        ]\n      },\n      {\n        "t": "TermsSearchFilterDto",\n        "fieldName": 5,\n        "extra": null,\n        "values": [\n          \n        ]\n      },\n      {\n        "t": "TermsSearchFilterDto",\n        "fieldName": 6,\n        "extra": null,\n        "values": [\n          \n        ]\n      }\n    ]\n  },\n  "matchCriteria": {\n    "PreferredJobs": [\n      \n    ],\n    "Educations": [\n      \n    ],\n    "LicenseAndCertifications": [\n      \n    ],\n    "Skills": [\n      \n    ],\n    "hasNoLicenses": false,\n    "SkippedSkills": [\n      \n    ]\n  }\n}'

response = requests.post('https://recruiting.ultipro.com/UNI1029UNION/JobBoard/74c2a308-3bf1-4fb1-8a83-f92fa61499d3/JobBoardView/LoadSearchResults', headers=headers, data=data)
print(response.text)

And here using pandas (pip install pandas)

import requests
import pandas as pd
pd.set_option('display.width', 1000)

headers = {
    'Content-Type': 'application/json'
}

data = '{\n  "opportunitySearch": {\n    "Top": 62,\n    "Skip": 0,\n    "QueryString": "",\n    "OrderBy": [\n      {\n        "Value": "postedDateDesc",\n        "PropertyName": "PostedDate",\n        "Ascending": false\n      }\n    ],\n    "Filters": [\n      {\n        "t": "TermsSearchFilterDto",\n        "fieldName": 4,\n        "extra": null,\n        "values": [\n          \n        ]\n      },\n      {\n        "t": "TermsSearchFilterDto",\n        "fieldName": 5,\n        "extra": null,\n        "values": [\n          \n        ]\n      },\n      {\n        "t": "TermsSearchFilterDto",\n        "fieldName": 6,\n        "extra": null,\n        "values": [\n          \n        ]\n      }\n    ]\n  },\n  "matchCriteria": {\n    "PreferredJobs": [\n      \n    ],\n    "Educations": [\n      \n    ],\n    "LicenseAndCertifications": [\n      \n    ],\n    "Skills": [\n      \n    ],\n    "hasNoLicenses": false,\n    "SkippedSkills": [\n      \n    ]\n  }\n}'

response = requests.post('https://recruiting.ultipro.com/UNI1029UNION/JobBoard/74c2a308-3bf1-4fb1-8a83-f92fa61499d3/JobBoardView/LoadSearchResults', headers=headers, data=data)
data=response.json()
df=pd.DataFrame.from_dict(data['opportunities'])
df= df[['Id','Title','RequisitionNumber','JobCategoryName','PostedDate']]
print(df.head(5))

Where data has "TOP" 62 like a limited your results:

{
  "opportunitySearch": {
    "Top": 62,
    "Skip": 0,
    "QueryString": "",
    "OrderBy": [
      {
        "Value": "postedDateDesc",
        "PropertyName": "PostedDate",
        "Ascending": false
      }
    ],
    "Filters": [
      {
        "t": "TermsSearchFilterDto",
        "fieldName": 4,
        "extra": null,
        "values": [

        ]
      },
      {
        "t": "TermsSearchFilterDto",
        "fieldName": 5,
        "extra": null,
        "values": [

        ]
      },
      {
        "t": "TermsSearchFilterDto",
        "fieldName": 6,
        "extra": null,
        "values": [

        ]
      }
    ]
  },
  "matchCriteria": {
    "PreferredJobs": [

    ],
    "Educations": [

    ],
    "LicenseAndCertifications": [

    ],
    "Skills": [

    ],
    "hasNoLicenses": false,
    "SkippedSkills": [

    ]
  }
}