使用Python无法使用不变的网址抓取多个页面

Question

After searching for a hint I found my problem is closely related to this question and based on this answer I thought I was about to solve my problem, but I did not do it.

I need to extract all URLs from this site http://elempleo.com/cr/ofertas-empleo , I did the following:

page_no=1
payload = {"jobOfferId":0,
           "salaryInfo":[],
           "city":0,
           "publishDate":0,
           "area":40,
           "countryId":0,
           "departmentId":0,
           "companyId":0,
           "pageIndex":page_no,
           "pageSize":"20"},
           "sortExpression":"PublishDate_Desc"}

page = requests.get('http://elempleo.com/cr/ofertas-empleo/get', params=payload)
soup = BeautifulSoup(page.content, 'html.parser')

href_list=soup.select(".text-ellipsis")

for urls in href_list:
    print("http://elempleo.com"+urls.get("href"))

http://elempleo.com/cr/ofertas-trabajo/ap-representative/757190
http://elempleo.com/cr/ofertas-trabajo/ingeniero-de-procesos-sap/757189
http://elempleo.com/cr/ofertas-trabajo/sr-program-analyst-months/757188
http://elempleo.com/cr/ofertas-trabajo/executive-asistant/757187
http://elempleo.com/cr/ofertas-trabajo/asistente-comercial-bilingue/757186
http://elempleo.com/cr/ofertas-trabajo/accounting-assistant/757185
http://elempleo.com/cr/ofertas-trabajo/asistente-contable/757184
http://elempleo.com/cr/ofertas-trabajo/personal-para-cajas-alajuela-con-experiencia-en-farmacia/757183
http://elempleo.com/cr/ofertas-trabajo/oficial-de-seguridad/743703
http://elempleo.com/cr/ofertas-trabajo/tecnico-de-mantenimiento-en-extrusion/757182
http://elempleo.com/cr/ofertas-trabajo/gerente-servicio-al-cliente-y-ventas/757181
http://elempleo.com/cr/ofertas-trabajo/encargadoa-departamento-de-recursos-humanos-ingles-intermedio/757180
http://elempleo.com/cr/ofertas-trabajo/director-of-development/757177
http://elempleo.com/cr/ofertas-trabajo/generalista-de-recursos-humanos-ingles-intermedio/757178
http://elempleo.com/cr/ofertas-trabajo/accounts-payable-specialist-seasonal-contract/757176
http://elempleo.com/cr/ofertas-trabajo/electricista-industrial/757175
http://elempleo.com/cr/ofertas-trabajo/payroll-analyst-months-contract/757174
http://elempleo.com/cr/ofertas-trabajo/gerente-servicio-post-venta/757172
http://elempleo.com/cr/ofertas-trabajo/operario-de-proceso/757171
http://elempleo.com/cr/ofertas-trabajo/cajero-de-kiosco-ubicacion-area-metropolitana-fines-de-semana-disponibilidad-de-horarios/757170

As you can see, it shows 20 urls, which is OK, but if I chage page_no=2 , page_no=3 , ... page_no=100 and run the above code again it returns the same result as before; I need all urls from all pages in this website. Can anybody help me?

Also, I set "area":40 which corresponds to sistemas category in Área de trabajo field. It doesn't do nothing, because results are not filtered as sistemas category.

I used beautifulsoup in Python3 running on Ubuntu 18.04.

Answers using rvest package in R are also welcome!

Answer 1

if you try scrolling through the pages with web console open, you will notice that pagination is done through the findByFilter javascript query. Python requests cannot handle this kind of page modifications.

You have two choices here:

1. use selenium browser to get a javascript-enabled scraper
2. Try to mock the headers and request payload for http://elempleo.com/cr/api/joboffers/findbyfilter POST request and get the data straight from the api (which would also comfortably give you a json response that you can put straight to python dictionary).

Answer 2

To setup selenium visit this link

import time
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.common.exceptions import WebDriverException
url = "http://elempleo.com/cr/ofertas-empleo/"

Note: You need to download the suitable browser driver from this link and add it's path to system environment variable

# here I am using chrome webdriver
# setting up selenium
driver = webdriver.Chrome(executable_path=r"F:\Projects\sms_automation\chromedriver.exe")  # initialize webdriver instance
driver.get(url)  # open URL in browser
driver.find_element_by_id("ResultsByPage").send_keys('100')  # set items per page to 100
time.sleep(5)
soup = BeautifulSoup(driver.page_source, "html.parser")
url_set = ["http://elempleo.com"+i.get("href") for i in soup.select(".text-ellipsis")]
while True:
    try:
        driver.find_element_by_class_name("js-btn-next").click()  # go to next page
        time.sleep(3)
        soup = BeautifulSoup(driver.page_source, "html.parser")
        current_page_url = ["http://elempleo.com"+i.get("href") for i in soup.select(".text-ellipsis")]
        if url_set[-1] == current_page_url[-1]:
            break
        url_set += current_page_url
    except WebDriverException:
        time.sleep(5)

Result:

print(len(url_set))   # outputs 2641
print(url_set)  # outputs ['http://elempleo.comhttp://www.elempleo.com/cr/ofertas-trabajo/analista-de-sistemas-financieros/753845', 'http://elempleo.comhttp://www.elempleo.com/cr/ofertas-trabajo/balance-sheet-and-cash-flow-specialist/755211', 'http://elempleo.comhttp://www.elempleo.com/cr/ofertas-trabajo/coordinador-de-compensacion/757369', 'http://elempleo.comhttp://www.elempleo.com/cr/ofertas-trabajo/gerente-de-agronomia/757368', 'http://elempleo.comhttp://www.elempleo.com/cr/ofertas-trabajo/responsable-de-capacitacion-y-desempeno/757367', 'http://elempleo.comhttp://www.elempleo.com/cr/ofertas-trabajo/pmp-gestor-de-proyectos/757366', ....]