使用 for 循环在 R 中抓取网页

Question

I would like to scrape the data from this link , and I have written the following code in R to do so. This, however, does not work and only returns the first page of the results. Apparently, the loop does not work. Does anybody know what's wrong with the loop?

library('rvest')

for (i in 1:40) {

     webpage <- read_html(paste0(("http://search.beaconforfreedom.org/search/censored_publications/result.html?author=&cauthor=&title=&country=7327&language=&censored_year=&censortype=&published_year=&censorreason=&sort=t&page=, i"))

     rank_data_html <- html_nodes(webpage,'tr+ tr td:nth-child(1)')

     rank_data <- html_text(rank_data_html)

     rank_data<-as.numeric(rank_data)

     title_data_html <- html_nodes(webpage,'.censo_list font')

     title_data <- html_text(title_data_html)

     author_data_html <- html_nodes(webpage,'.censo_list+ td font')
     author_data <- html_text(author_data_html)

     country_data_html <- html_nodes(webpage,'.censo_list~ td:nth-child(4) font')

     rcountry_data <- html_text(country_data_html)

     year_data_html <- html_nodes(webpage,'tr+ tr td:nth-child(5) font')

     year_data <- html_text(year_data_html)

     type_data_html <- html_nodes(webpage,'tr+ tr td:nth-child(6) font')

     type_data <- html_text(type_data_html)

}

censorship_df<-data.frame(Rank = rank_data, Title = title_data, Author = author_data, Country = rcountry_data, Type = type_data, Year = year_data)

write.table(censorship_df, file="sample.csv",sep=",",row.names=F)

Answer 1

Are you sure there's anything wrong with the loop? I would expect it to get the first page of results 40 times. Look at

webpage <- read_html(paste0(("http://search.beaconforfreedom.org/search/censored_publications/result.html?author=&cauthor=&title=&country=7327&language=&censored_year=&censortype=&published_year=&censorreason=&sort=t&page=, i"))

Shouldn't that be (difference in the last ten characters of the string; the quotation mark moves)

webpage <- read_html(paste0(("http://search.beaconforfreedom.org/search/censored_publications/result.html?author=&cauthor=&title=&country=7327&language=&censored_year=&censortype=&published_year=&censorreason=&sort=t&page=", i))

What paste0 does in R is it stitches together two strings without any separator. But you only have one string. So it tries to fetch results for page=, i . But you want it to fetch page=1 through page=40 . So put the quotation mark like page=", i so that it pastes the URL and i together.

I'm not an R programmer, but that simply leaps out at me.

Source for paste0 behavior.