是否可以抓取特定主题的所有谷歌学术搜索结果，是否合法？

Question

I have some Rexperience, but not with website coding, and think I was not able to select the correct CSS nodes to parse (I believe).

library(rvest)
library(xml2)
library(selectr)
library(stringr)
library(jsonlite)

url <-'https://scholar.google.com/scholar?hl=en&as_sdt=0%2C38&q=apex+predator+conservation&btnG=&oq=apex+predator+c'
webpage <- read_html(url)

title_html <- html_nodes(webpage, 'a#rh06x-YUUvEJ')
title <- html_text(title_html)
head(title)

Ultimately, if I could scrape and divide all scholar results into a csv file with headers like 'Title', 'Author', 'Year', 'Journal', that would be great. Any help would be much appreciated! Thanks

Answer 1

Concerning your code, you almost had it - you did not select the proper element. I believe you selected by id where I found html_nodes works best when selecting by class . The classes you are looking for are gs_rt and gs_a .

With regex you can then process the data to the desired format by extracting authors and years.

url_name <- 'https://scholar.google.com/scholar?hl=en&as_sdt=0%2C38&q=apex+predator+conservation&btnG=&oq=apex+predator+c'
wp <- xml2::read_html(url_name)
# Extract raw data
titles <- rvest::html_text(rvest::html_nodes(wp, '.gs_rt'))
authors_years <- rvest::html_text(rvest::html_nodes(wp, '.gs_a'))
# Process data
authors <- gsub('^(.*?)\\W+-\\W+.*', '\\1', authors_years, perl = TRUE)
years <- gsub('^.*(\\d{4}).*', '\\1', authors_years, perl = TRUE)
# Make data frame
df <- data.frame(titles = titles, authors = authors, years = years, stringsAsFactors = FALSE)

Answer 2

Thanx Nico that was really helpful. however, it would just scrape data from the first page result, and the maximum results per page (as per google scholar settings) is 20. Is there any way to do scrape data from all page results?? Thank you so much!