使用 rvest 抓取网站：“当前页面似乎不是 html。”

Question

I try to access this website: https://www.apa.org/pubs/journals/browse?query=Title:*&type=journal

However, I get the error message: Current page doesn't appear to be html.

I thus cannot proceed to scrape the website with html_nodes etc.

This is my code:

apa_url <- "https://www.apa.org/pubs/journals/browse?query=Title:*&type=journal"

apa_page <- rvest::html_session(apa_url,
                                httr::user_agent("Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.20 (KHTML, like Gecko) Chrome/11.0.672.2 Safari/534.20"))

If you know how to fix it, I would be grateful for your help!

Answer 1

You haven't shared what you want to scrape but you don't need to create a session.

For example, to get titles of the journal in the first page you can do:

library(rvest)
apa_url <- "https://www.apa.org/pubs/journals/browse?query=Title:*&type=journal"

apa_url %>%
  read_html() %>%
  html_nodes('section.sresults li a') %>%
  html_text()

# [1] "American Journal of Orthopsychiatry - APA Publishing | APA"               
# [2] "American Psychologist Journal - APA Publishing | APA"                     
# [3] "Archives of Scientific Psychology"                                     
# [4] "Asian American Journal of Psychology"                                     
# [5] "Behavior Analysis: Research and Practice"                          
# [6] "Behavioral Development"       
#...
#...