使用getURL在R中进行Web爬取

Question

Hi I am trying to read data of the World's powerfl brands from the link " http://www.forbes.com/powerful-brands/list/3/#tab:rank " into a data fame using R

I am a beginner so I tried using the following code to retrieve the data

  library(XML)
  library(RCurl)
  # Read and parse HTML file
  forbe = 'http://www.forbes.com/powerful-brands/list/#tab:rank'

  data <- getURL('http://www.forbes.com/powerful-brands/list/#tab:rank')
  data
  htmldata <- readHTMLTable(data)
  htmldata 

Could anyone please help me in retrieving data from the webpage mentioned

Answer 1

They use XHR requests to populate the page via javascript. Use browser Developer Tools to see the Network requests

and grab the JSON directly:

brands <- jsonlite::fromJSON("http://www.forbes.com/ajax/list/data?year=2015&uri=powerful-brands&type=organization")
str(brands)

## 'data.frame':    100 obs. of  10 variables:
##  $ position          : int  12 44 83 87 13 22 1 39 16 72 ...
##  $ rank              : int  12 44 83 87 13 22 1 39 16 72 ...
##  $ name              : chr  "AT&T" "Accenture" "Adidas" "Allianz" ...
##  $ uri               : chr  "att" "accenture" "adidas" "allianz" ...
##  $ imageUri          : chr  "att" "accenture" "adidas" "allianz" ...
##  $ industry          : chr  "Telecom" "Business Services" "Apparel" "Financial Services" ...
##  $ revenue           : num  132400 32800 14900 131600 87500 ...
##  $ oneYearValueChange: int  17 14 -14 -6 32 13 17 1 -5 -1 ...
##  $ brandValue        : num  29100 12000 6800 6600 28100 ...
##  $ advertising       : num  3272 88 NA NA 3300 ...

Answer 2

Why don't you try something like this . Basically, doing something like:

download.file(forbe, htmldata, auto, quiet = FALSE, cacheOK = TRUE)

And the read data should be in the htmldata array variable.