如何使用RSelenium和Rvest在R中循環遍歷多個網站並提取相同的信息？

Question

我正在嘗試開發一個可以通過組合RSelenium和rvest來解決問題的代碼，其中rvest單獨總是在刮掉許多網站列表時超時。

由於單獨使用rvest不起作用，RSelenium可以通過循環打開和關閉列表中的每個網站來解決問題，但我擔心如果網站列表很長，這種方法可能需要很長時間。

我嘗試結合我之前的代碼並使用RSelenium在多個網站中添加新的循環，但它看起來不像它正在工作。

library(xml2)
library(dplyr)
library(readr)
library(RSelenium)
webpages <- data.frame(name = c("amazon", "apple", "usps", "yahoo", "bbc", "ted", "surveymonkey", "forbes", "imdb", "hp"),
                       url = c("http://www.amazon.com",
                               "http://www.apple.com",
                               "http://www.usps.com",
                               "http://www.yahoo.com",
                               "http://www.bbc.com",
                               "http://www.ted.com",
                               "http://www.surveymonkey.com",
                               "http://www.forbes.com",
                               "http://www.imdb.com",
                               "http://www.hp.com"))

driver <- rsDriver(browser = c("chrome"))
remDr <- driver[["client"]]

i <- 1
while(i <= 4){
  url <- webpages$url[i]
  remDr$navigate(url)

  page_source <- remDr$getPageSource()

  URL <- read_html(page_source)

  results <- URL %>% html_nodes("head")

  records <- vector("list", length = length(results))

  for (i in seq_along(records)) {
  title <- xml_contents(results[i] %>% 
    html_nodes("title"))[1] %>% html_text(trim = TRUE)
  description <- results[i] %>% 
    html_nodes("meta[name=description]") %>% html_attr("content")
  keywords <- results[i] %>%
    html_nodes("meta[name=keywords]") %>% html_attr("content")
}

  i <- i + 1
  remDr$close()

  return(data.frame(name = x['name'],
                    url = x['url'],
                    title = ifelse(length(title) > 0, title, NA),
                    description = ifelse(length(description) > 0, desc, NA),
                    keywords = ifelse(length(keywords) > 0, kw, NA)))

}

我現在得到的錯誤是：

Error in UseMethod("read_xml") : 
  no applicable method for 'read_xml' applied to an object of class "list"

我想要的結果是這樣的：

url                            title                 description               keywords
http://www.apple.com           Apple             website description        keywords
http://www.amazon.com          Amazon            website description        keywords
http://www.usps.com            Usps              website description        keywords
http://www.yahoo.com           Yahoo             website description        keywords   
http://www.bbc.com             Bbc               website description        keywords
http://www.ted.com             Ted               website description        keywords
http://www.surveymonkey.com    Survey Monkey     website description        keywords
http://www.forbes.com          Forbes            website description        keywords
http://www.imdb.com            Imdb              website description        keywords
http://www.hp.com              Hp                website description        keywords

Answer 1

您只需要通過page_source[[1]]更改page_source ， page_source[[1]]變量命名（例如索引器，向量）和調用稍微小心一些。 我還建議您在使用這些循環時打印出一些消息。 此外，如果刪除remDr$close() ，則可以避免連接松動。 最后，您可以將結果存儲在循環中的變量中：

scrapped = list()

i <- 1
while(i <= 4){

  url <- webpages$url[i]

  print( paste("Accessing to:", url) )

  remDr$navigate(url)

  page_source <- remDr$getPageSource()

  URL <- read_html(page_source[[1]])

  results <- URL %>% html_nodes("head")

  records <- vector("list", length = length(results))

  for (ii in seq_along(records)) {

     title <- xml_contents(results[ii] %>%  html_nodes("title"))[1] %>%
      html_text(trim = TRUE)

     desc <- results[ii] %>% 
      html_nodes("meta[name=description]") %>% 
      html_attr("content")

    keywords <- results[ii] %>%
      html_nodes("meta[name=keywords]") %>% 
      html_attr("content")
  }

  #remDr$close()

  scrapped[[i]] =  data.frame(name = webpages[i,'name'],
                             url = webpages[i,'url'],
                             title = ifelse(length(title) > 0, title, NA),
                             description = ifelse(length(desc) > 0, desc, NA),
                             keywords = ifelse(length(keywords) > 0, keywords, NA))
  i = i + 1

}

產量

do.call('rbind', scrapped) 

#    name                   url                                                                               title
#1 amazon http://www.amazon.com Amazon.com: Online Shopping for Electronics, Apparel, Computers, Books, DVDs & more
#2  apple  http://www.apple.com                                                                               Apple
#3   usps   http://www.usps.com                                                                      Welcome | USPS
#4  yahoo  http://www.yahoo.com                                                                               Yahoo
                                                                                                                                                                                                                                                                                                   description
#1 Online shopping from the earth's biggest selection of books, magazines, music, DVDs, videos, electronics, computers, software, apparel & accessories, shoes, jewelry, tools & hardware, housewares, furniture, sporting goods, beauty & personal care, broadband & dsl, gourmet food & just about anything else.
#2                                                                                                                                                                                                                                                                                                             <NA>
#3                                                                                            Welcome to USPS.com. Find information on our most convenient and affordable shipping and mailing services. Use our quick tools to find locations, calculate prices, look up a ZIP Code, and get Track & Confirm info.
#4                                                                                                                                                                                       Las noticias, el correo electrónico y las búsquedas son tan solo el comienzo. Descubre algo nuevo todos los días en Yahoo.

#keywords
#1 Amazon, Amazon.com, Books, Online Shopping, Book Store, Magazine, Subscription, Music, CDs, DVDs, Videos, Electronics, Video Games, Computers, Cell Phones, Toys, Games, Apparel, Accessories, Shoes, Jewelry, Watches, Office Products, Sports & Outdoors, Sporting Goods, Baby Products, Health, Personal Care, Beauty, Home, Garden, Bed & Bath, Furniture, Tools, Hardware, Vacuums, Outdoor Living, Automotive Parts, Pet Supplies, Broadband, DSL
#2                                                                                                                                                                                                                                                                                                                                                                                                                                                    <NA>
#3                                                                                                                                                                             Quick Tools, Shipping Services, Mailing Services, Village Post Office, Ship Online, Flat Rate, Postal Store, Ship a Package, Send Mail, Manage Your Mail,  Business Solutions, Find Locations, Calculate a Price, Look Up a ZIP Code, Track Packages, Print a Label, Stamps
#4                                                                                                                                                                                                                                                                                                 yahoo, yahoo inicio, yahoo página de inicio, yahoo búsqueda, correo yahoo, yahoo messenger, yahoo juegos, noticias, finanzas, deportes, entretenimiento