导出 google 学者结果中的链接

Question

I want to export data from google scholar. In particular, I want to export a list of articles that cite a particular paper. If I click the Cited By link I can get this page. One way I can export these data is to add all of them to my library. Then you can export in 4 different formats (BibTex, Refman, Endnote, CSV). However, none of these export formats include the HTML link (URL) to each paper.

The other strategy would be to scrape the data, but I don't want to do that as I know that this can be very tricky with google scholar's captchas.

Is there a way to export the results of a google scholar search that includes the URLs of each paper?

Answer 1

For the page you're on you mean? From the console (F12) do:

copy($$('li > a').map(a => a.href))

Now they're in your clipboard.

Answer 2

To extract Cited by data, you'll need an ID of a Google Scholar organic search result that the Cited by link belongs to. You can find the ID inside data-cid html attribute.

You can then query the next link to retrieve the data: https://scholar.google.com/scholar?q=info: this_is_where_you_put_the_cite_id:scholar.google.com/&output=cite

There is also a third party solution like SerpApi to do this for you. It's a paid API with a free trial.

Example python code (available in other libraries also):

from serpapi import GoogleSearch

params = {
  "engine": "google_scholar_cite",
  "q": "FDc6HiktlqEJ",
  "api_key": "secret_api_key",
}

search = GoogleSearch(params)
results = search.get_dict()

Example JSON output:

"citations": [
  {
    "title": "MLA",
    "snippet": "Schwertmann, U. T. R. M., and Reginald M. Taylor. \"Iron oxides.\" Minerals in soil environments 1 (1989): 379-438."
  },
  {
    "title": "APA",
    "snippet": "Schwertmann, U. T. R. M., & Taylor, R. M. (1989). Iron oxides. Minerals in soil environments, 1, 379-438."
  },
  {
    "title": "Chicago",
    "snippet": "Schwertmann, U. T. R. M., and Reginald M. Taylor. \"Iron oxides.\" Minerals in soil environments 1 (1989): 379-438."
  },
  {
    "title": "Harvard",
    "snippet": "Schwertmann, U.T.R.M. and Taylor, R.M., 1989. Iron oxides. Minerals in soil environments, 1, pp.379-438."
  },
  {
    "title": "Vancouver",
    "snippet": "Schwertmann UT, Taylor RM. Iron oxides. Minerals in soil environments. 1989 Jan 1;1:379-438."
  }
],
"links": [
  {
    "name": "BibTeX",
    "link": "https://scholar.googleusercontent.com/scholar.bib?q=info:FDc6HiktlqEJ:scholar.google.com/&output=citation&scisdr=CgXpniNQGAA:AAGBfm0AAAAAYMu3WkYJI4po_pgcUVKgwwFp1dl5uNYk&scisig=AAGBfm0AAAAAYMu3WlZR_joxo-i8FTZ1CphjzmW_d447&scisf=4&ct=citation&cd=-1&hl=en"
  },
  {
    "name": "EndNote",
    "link": "https://scholar.googleusercontent.com/scholar.enw?q=info:FDc6HiktlqEJ:scholar.google.com/&output=citation&scisdr=CgXpniNQGAA:AAGBfm0AAAAAYMu3WkYJI4po_pgcUVKgwwFp1dl5uNYk&scisig=AAGBfm0AAAAAYMu3WlZR_joxo-i8FTZ1CphjzmW_d447&scisf=3&ct=citation&cd=-1&hl=en"
  },
  {
    "name": "RefMan",
    "link": "https://scholar.googleusercontent.com/scholar.ris?q=info:FDc6HiktlqEJ:scholar.google.com/&output=citation&scisdr=CgXpniNQGAA:AAGBfm0AAAAAYMu3WkYJI4po_pgcUVKgwwFp1dl5uNYk&scisig=AAGBfm0AAAAAYMu3WlZR_joxo-i8FTZ1CphjzmW_d447&scisf=2&ct=citation&cd=-1&hl=en"
  },
  {
    "name": "RefWorks",
    "link": "https://scholar.googleusercontent.com/scholar.rfw?q=info:FDc6HiktlqEJ:scholar.google.com/&output=citation&scisdr=CgXpniNQGAA:AAGBfm0AAAAAYMu3WkYJI4po_pgcUVKgwwFp1dl5uNYk&scisig=AAGBfm0AAAAAYMu3WlZR_joxo-i8FTZ1CphjzmW_d447&scisf=1&ct=citation&cd=-1&hl=en"
  }
]

Check out the documentation for more details.

Disclaimer: I work at SerpApi.