抓取网站缺失数据

Question

So I'm trying to scrape the open positions on this site and when I use any type of requests (currently trying request-html) it doesn't show everything that's in the HTML.

# Import libraries
import time
from bs4 import BeautifulSoup
from requests_html import HTMLSession

# Set the URL you want to webscrape from
url = 'https://germanamerican.csod.com/ux/ats/careersite/5/home?c=germanamerican'

session = HTMLSession()

# Connect to the URL
response = session.get(url)

response.html.render()

# Parse HTML and save to BeautifulSoup object¶
soup = BeautifulSoup(response.text, "html5lib")  

b = soup.findAll('a')

Not sure where to go. Originally thought the problem was due to javascript rendering but this is not working.

Answer 1

I don't think it's possible to scrape that website with Requests. I would suggest using Selenium or Scrapy.

Answer 2

The issue is that the initial GET doesn't get the data (which I assume is the job listings), and the js that does do that, uses a POST with a authorization token in the header. You need to get this token and then make the POST to get the data.

This token appears to be dynamic so we're going to get a little wonky getting it, but doable.

url0=r'https://germanamerican.csod.com/ux/ats/careersite/5/home?c=germanamerican'
url=r'https://germanamerican.csod.com/services/x/career-site/v1/search'

s=HTMLSession()
r=s.get(url0)
print(r.status_code)
r.html.render()

soup=bs(r.text,'html.parser')

scripts=soup.find_all('script')

for script in scripts:
    if 'csod.context=' in script.text: x=script

j=json.loads(x.text.replace('csod.context=','').replace(';',''))


payload={
    'careerSiteId': 5,
    'cities': [],
    'countryCodes': [],
    'cultureId': 1,
    'cultureName': "en-US",
    'customFieldCheckboxKeys': [],
    'customFieldDropdowns': [],
    'customFieldRadios': [],
    'pageNumber': 1,
    'pageSize': 25,
    'placeID': "",
    'postingsWithinDays': None,
    'radius': None,
    'searchText': "",
    'states': []
}

headers={
    'accept': 'application/json; q=1.0, text/*; q=0.8, */*; q=0.1',
    'accept-encoding': 'gzip, deflate, br',
    'accept-language': 'en-US,en;q=0.9',
    'authorization': 'Bearer '+j['token'],
    'cache-control': 'no-cache',
    'content-length': '272',
    'content-type': 'application/json',
    'csod-accept-language': 'en-US',
    'origin': 'https://germanamerican.csod.com',
    'referer': 'https://germanamerican.csod.com/ux/ats/careersite/5/home?c=germanamerican',
    'sec-fetch-mode': 'cors',
    'sec-fetch-site': 'same-origin',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36',
    'x-requested-with': 'XMLHttpRequest'
}

r=s.post(url,headers=headers,json=payload)
print(r.status_code)
print(r.json())

the r.json() thats printed out is a nice json format version of the table of job listings.

Answer 3

Welcome to SO!

Unfortunately, you won't be able to scrape that page with requests (nor requests_html or similar libraries) because you need a tool to handle dynamic pages - ie, javascript-based.

With python, I would strongly suggest selenium and its webdriver . Below a piece of code that prints the desired output - ie, all listed jobs (NB it requires selenium and Firefox webdriver to be installed and with the correct PATH to run)

# Import libraries
from bs4 import BeautifulSoup
from selenium import webdriver

# Set the URL you want to webscrape from
url = 'https://germanamerican.csod.com/ux/ats/careersite/5/home?c=germanamerican'

browser = webdriver.Firefox() # initialize the webdriver. I use FF, might be Chromium or else

browser.get(url) # go to the desired page. You might want to wait a bit in case of slow connection
page = browser.page_source # this is the page source, now full with the listings that have been uploaded
soup = BeautifulSoup(page, "lxml")
jobs = soup.findAll('a', {'data-tag' : 'displayJobTitle'})
for j in jobs:
    print(j.text)

browser.quit()