R: 如何对多个 URL 使用 map(read_html)？

Question

i'm trying to scrape title, date and content from multiple article URLs. I've had success using the following code below for a single string URL but I keep getting errors when trying to pass more than one string.

Working code:

article <- single_url  %>% purrr::map(read_html)
title <-
  article %>% map_chr(. %>% html_node("title") %>% html_text())
content <-
  article %>% map_chr(. %>% html_nodes("p") %>% html_text() %>% paste(., collapse = ""))
time <- 
  article %>% map_chr(. %>% html_nodes("time") %>% html_text() %>% paste(., collapse = ""))
article_table <- data.frame("Title" = title, "Content" = content, "New Time" = time)

I've had no success with trying to modify for passing each string that is in my vector variable 'Direct.Link' (see example data below)

Example Data:

test <- structure(list(Participant.Name = c("Participant 1", "Participant 1", 
                                    "Participant 2"), Direct.Link = c("https://chicago.suntimes.com/2020/3/26/21196297/jails-and-prisons-could-become-coronavirus-disaster", 
                                                                     "https://www.pressconnects.com/story/news/local/2017/07/28/cornell-study-sheds-light-students-incarcerated-parents/512160001/", 
                                                                     "https://www.newsobserver.com/news/local/article247133959.html"
                                    )), row.names = c(9L, 12L, 33L), class = "data.frame")

I've tried this to no avail.

Attempt at modifying code :

article <- test %>% mutate( art = purrr::map(read_html(Direct.Link)))
#Error: Problem with `mutate()` column `art`.
#i `art = purrr::map(read_html(Direct.Link))`.
#x argument ".f" is missing, with no default

Ultimately I'd like to get a dataset that looks like this:

Ideal Data:

test2 <- structure(list(Participant.Name = c("Participant 1", "Participant 1", 
                                            "Participant 2"), Direct.Link = c("https://chicago.suntimes.com/2020/3/26/21196297/jails-and-prisons-could-become-coronavirus-disaster", 
                                                                              "https://www.pressconnects.com/story/news/local/2017/07/28/cornell-study-sheds-light-students-incarcerated-parents/512160001/", 
                                                                              "https://www.newsobserver.com/news/local/article247133959.html"),
                        title= c("Title of News Article 1", "Date of News Article 2", "Date of News Article 3"), 
                        content = c("Content of News Article 1", "Content of News Article 2", "Content of News Article 3" ),
                        time = c("Date of News Article 1","Date of News Article 2","Date of News Article 3"
                        )), row.names = c(9L, 12L, 33L), class = "data.frame")

Thanks for any help offered!

Answer 1

It looks like map doesn't like the way you're providing the functions within it. map is really only designed to operate on a single function at a time, so you could either define a new function (eg getTitle ) that performs the three steps of getting the article, getting the node, and then getting the title, or you can break it up into multiple calls to mutate . The piping system you've got there doesn't really work neatly with purrr's intended use.

Here's an example of the split call to mutate. Note that each time I call map , I'm only providing a single function rather than a chain:

articles <- test %>% 
  slice(1:2) %>% #The third article was causing rvest to hang
  mutate(art = map(.x = Direct.Link, .f = read_html)) %>%
  mutate(title = map(art, html_node, "title")) %>% 
  mutate(title = map_chr(title, html_text)) %>%
  select(-art) # Drop the article external pointer so it can be printed neatly

And here's an example of the custom function:

getTitle <- function(art_url){
  art_url %>% read_html() %>% html_node("title") %>% html_text()
}
articles <- test %>% 
  slice(1:2) %>%
  mutate(title=map(Direct.Link, getTitle))

Both of the above return the output below:

This can then be expanded/repeated to extract the other nodes of interest. Note that the first approach is much more polite web behavior because you're only requesting the article once, rather than each time, but I thought it would still be helpful to illustrate how map works with it. I'll also give a shoutout to the polite package since you're webscraping, potentially repeatedly.