rvest HTML表格抓取技术返回空列表

Question

I have had success with rvest when scraping data from html tables, however, for this particular website, http://www.sanzarrugby.com/superrugby/competition-stats/2016-team-ranking/ , when I run the code

url <- "http://www.sanzarrugby.com/superrugby/competition-stats/2016-team-ranking/"
rankings <- url %>%
read_html %>%
html_nodes("table") %>%
html_table()

All that is returned is an empty list. What might be wrong?

Answer 1

The "problem" with this site is that it dynamically loads a javascript file which it then executes via a callback mechanism to create the JS data which it then builds the tables/vis from.

One way to get the data is [R]Selenium, but that's problematic for many folks.

Another way is to use the Developer Tools of your browser to see the JS request, run "Copy as cURL" (right-click, usually) and then use some R-fu to get what you need. Since this is going to be returning javascript, we'll need to do some mangling before ultimately converting the JSON.

library(jsonlite)
library(curlconverter)
library(httr)

# this is the `Copy as cURL` result, but you can leave it in your clipboard 
# and not do this in production. Read the `curlconverter` help for more info

CURL <- "curl 'http://omo.akamai.opta.net/competition.php?feed_type=ru3&competition=205&season_id=2016&user=USERNAME&psw=PASSWORD&jsoncallback=RU3_205_2016' -H 'DNT: 1' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36 Vivaldi/1.1.453.54' -H 'Accept: */*' -H 'Referer: http://www.sanzarrugby.com/superrugby/competition-stats/2016-team-ranking/' -H 'Connection: keep-alive' -H 'If-Modified-Since: Wed, 11 May 2016 14:47:09 GMT' -H 'Cache-Control: max-age=0' --compressed"

req <- make_req(straighten(CURL))[[1]]
req

# that makes:

# httr::VERB(verb = "GET", url = "http://omo.akamai.opta.net/competition.php?feed_type=ru3&competition=205&season_id=2016&user=USERNAME&psw=PASSWORD&jsoncallback=RU3_205_2016", 
#     httr::add_headers(DNT = "1", `Accept-Encoding` = "gzip, deflate, sdch", 
#         `Accept-Language` = "en-US,en;q=0.8", `User-Agent` = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36 Vivaldi/1.1.453.54", 
#         Accept = "*/*", Referer = "http://www.sanzarrugby.com/superrugby/competition-stats/2016-team-ranking/", 
#         Connection = "keep-alive", `If-Modified-Since` = "Wed, 11 May 2016 14:47:09 GMT", 
#         `Cache-Control` = "max-age=0"))

# which we can transform into the following after experimenting

URL <- "http://omo.akamai.opta.net/competition.php?feed_type=ru3&competition=205&season_id=2016&user=USERNAME&psw=PASSWORD&jsoncallback=RU3_205_2016"

pg <- GET(URL,
          add_headers(
            `User-Agent` = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36 Vivaldi/1.1.453.54", 
            Referer = "http://www.sanzarrugby.com/superrugby/competition-stats/2016-team-ranking/"))

# now all we need to do is remove the callback

dat_from_json <- fromJSON(gsub(")$", "", gsub("^RU3_205_2016\\(", "", content(pg, as="text"))), flatten=FALSE)


# we can also try removing the JSON callback, but it will return XML instead of JSON,
# which is fine since we can parse that easily

URL <- "http://omo.akamai.opta.net/competition.php?feed_type=ru3&competition=205&season_id=2016&user=USERNAME&psw=PASSWORD"

pg <- GET(URL,
          add_headers(
            `User-Agent` = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36 Vivaldi/1.1.453.54", 
            Referer = "http://www.sanzarrugby.com/superrugby/competition-stats/2016-team-ranking/"))

xml_doc <- content(pg, as="parsed", encoding="UTF-8")

# but then you have to transform the XML, which I'll leave as an exercise to the OP :-)