在 R 中，在字符串的循环中编写循环的最佳方法是什么？

Question

Looking into a web crawler that will go through multiple indeed.com country URLs

I have the first part of the code that crawls through individual pages below:

library(tidyverse)
library(rvest)
library(xml2)
library(dplyr)
library(stringr)

listings<- data.frame(title=character(),
                      company=character(),
                      stringAsFactors = FALSE)

for(i in seq(0,500,10)){
   url_ds<-paste0('https://www.indeed.com/jobs?q=data+analyst&l=&radius=25&start=',i)
   var <-read_html(url_ds)

#job title
title<- var %>%
    html_nodes('#resultsCol .jobtitle') %>%
    html_text() %>%
    str_extract("(//w+,+)+")

#company
    company<- var %>%
    html_nodes('#resultsCol .company') %>%
    html_text() %>%
    str_extract("(//w+,+)+")

 listings<-rbind(listings, as.data.frame(cbind(company,
                                          title)))
 }

What I would like to do is also loop through an array of the different country urls at the beginning of the "url_ds" above using a url_basic_list below and add a column for the actual country. basically I would need to create a loop within a loop for a text string, what is the best way to do so?

url_basic_list<-
     c("http://www.indeed.com",
     "http://www.indeed.com.hk",
     "http://www.indeed.com.sg"
     )

country<-
     c("USA",
     "Hong Kong",
     "Singapore"
     )

Answer 1

Two suggestions:

• change your for loop to lapply ; this is mostly because iteratively adding rows to a data.frame starts out okay but gets slower and more memory-intensive with each pass through the loop. (For each rbind , it has to copy all of the contents in memory, so your memory needs are at least double the size of the frame.) By using lapply , it creates a list of data.frame s (read the link,), which is created and filled memory-efficiently (as much as R can do), and then we do a single rbind at the end on the whole dataset.

• functionize this, and call the country code ( cc ) as a function argument.

get_indeed <- function(cc = "") {
  dotcc <- if (cc == "us") "" else paste0(".", cc)

  listings_list <- lapply(seq(0, 500, by = 10), function(i) {
    url_ds <- sprintf('https://www.indeed.com%s/jobs?q=data+analyst&l=&radius=25&start=%i', dotcc, i)
    var <- read_html(url_ds)

    #job title
    title <- var %>%
      html_nodes('#resultsCol .jobtitle') %>%
      html_text() %>%
      str_extract("(//w+,+)+")

    #company
    company <- var %>%
      html_nodes('#resultsCol .company') %>%
      html_text() %>%
      str_extract("(//w+,+)+")

    data.frame(company, title)
  })
  listings <- do.call(rbind, listings_list)
  listings$cc <- if (nzchar(cc)) cc else ""
  listings
}

From here, to "loop" through a series of countries, one might do

all_countries <- lapply(c("us", "hk", "sg"), get_indeed)
all_countries <- do.call(rbind, all_countries)

From here, all of your $cc values will be the two-letter codes, which is fine. The bring in the full names, I suggest you have a simple data.frame to map one to the other:

countries <- data.frame(
  cc = c("us", "hk", "sg"),
  country = c("USA", "Hong Kong", "Singapore")
)
all_countries <- merge(all_countries, countries, by = "cc")

And your data will now have both $cc (two-letter) and $country (full words).