从 R 中的网络抓取中提取数据的效率

Question

This is no doubt very simple so apologies but I am new to webscraping and am trying to extract multiple datapoints in one call using rvest. Let's take for example the following code (NB I have not used the actual website which I have replaced in this code snippet with xxxxxx.com):

univsalaries <- lapply(paste0('https://xxxxxx.com/job/p', 1:20,'/key=%F9%80%76&final=1&jump=1&PGTID=0d3408-0000-24gf-ac2b-810&ClickID=2'),
                   function(url_base){
                     url_base %>% read_html() %>% 
                       html_nodes('.salary') %>% 
                       html_text()
                   })

Answer 1

Read the webpage once and then you can extract multiple values from the same page.

library(purrr)
library(rvest)

univsalaries <- map(paste0('https://xxxxxx.com/job/p', 1:20,'/key=%F9%80%76&final=1&jump=1&PGTID=0d3408-0000-24gf-ac2b-810&ClickID=2'),
                       function(url_base){
                         webpage <- url_base %>% read_html() 
                         data.frame(Salary = webpage %>% html_nodes('.salary') %>% html_text(), 
                                    Company = webpage %>% html_nodes('.company') %>% html_text())
                       })