用户评论的数据提取

Question

I am trying to learn R out of my personal self learning interest. Neither a coder nor an analyst. I wanted to extract user reviews from Trip Advisor. In single page we have 10 reviews, but using below codes i am getting unwanted reviews/lines as well. I am not sure if i am using the correct html node. Moreover, i want to extract full review of a user but its ending giving me the partial reviews of a user. Can you please help me in extracting the full user reviews of count 10? Thank you so much for your help.

  dat <- readLines("http://www.tripadvisor.in/Hotel_Review-g60763-d93450-Reviews-Grand_Hyatt_New_York-New_York_City_New_York.html", warn=FALSE)
  raw2 <- htmlTreeParse(dat, useInternalNodes = TRUE)
  ##User Review
  plain.text <- xpathSApply(raw2, "//div[@class='col2of2']//p[@class='partial_entry']", xmlValue)
  UR <-gsub("\\\n","",plain.text)
  Result <- unlist(UR)
  Result

Answer 1

This is much more an exercise in web-scraping than R programming.

In R, I prefer the httr package to grab the http response and extract the content as parsed html. Using readLines(...) is just about the worst way to do this. So the code below will extract the review summaries.

library(httr)
library(XML)
url <- "http://www.tripadvisor.in/Hotel_Review-g60763-d93450-Reviews-Grand_Hyatt_New_York-New_York_City_New_York.html"
response <- GET(url)
doc      <- content(response,type="text/html")
smry     <- xpathSApply(doc,'//div[@class="entry"]/p[@class="partial_entry"]',xmlValue)
length(smry)
# [1] 10
smry[1]
# [1] "\nThats all that matters really...I wonder if anyone would chose this hotel for any other factor at all...located right next to Grand central station in midtown and within walking distance of many tourist attractions, top restaurants and corp offices. Stayed 3 nights here on a business trip, I chose this hotel over others purely based on its location. Price is...\n\n\nMore  \n\n"

Getting the full reviews is more complicated, because it involves clicking on the "More" button. So you need to determine which http requests are fired when you click the "More" link on a reference. You can do this using the Network Monitor tab in Firefox's developer tools (or many other tools, I'm sure). It turns out that this is a link of the form:

http://www.tripadvisor.com/ExpandedUserReviews-g{xxx}-d{yyy}?querystring

where {xxx} and {yyy} are unique to the hotel and are the same as in the original url, and querystring is fully identified in the Network Monitor tool. So we form a new http request with that url and the appropriate query string and parse the result, as below.

cls   <- doc['//div[@class="entry"]//span[contains(@class,"moreLink")]/@class']
xr.refno <- sapply(cls,function(x)sub(".*\\str(\\d+)\\s.*","\\1",x))
code     <- sub(".*Hotel_Review(\\-g\\d+\\-d\\d+)\\-Reviews.*","\\1",url)
xr.url   <- paste0("http://www.tripadvisor.com/ExpandedUserReviews",code)
xr.response <- GET(xr.url,query=list(target=xr.refno[1],
                                     context=1,
                                     reviews=paste(xr.refno,collapse=","),
                                     servlet="Hotel_Review",
                                     expand=1))
xr.doc   <- content(xr.response,type="text/html")
xr.full  <- xpathSApply(xr.doc,'//div[@class="entry"]/p',xmlValue)
length(xr.full)
# [1] 6
xr.full[1]
# [1] "\nThats all that matters really...I wonder if anyone would chose this hotel for any other factor at all...located right next to Grand central station in midtown and within walking distance of many tourist attractions, top restaurants and corp offices. Stayed 3 nights here on a business trip, I chose this hotel over others purely based on its location. Price is about average in NYC I think. Asked for a room with a good view and was given a 2 BR on the 30th floor. After checking in I realized there may not be the kind of view that I expected at all from any room in this hotel - due to it being surrounded by high rises in all directions. However, no other complaints as such - except may that the bathroom was a bit too cramped. That I guess is the norm in NYC. I would stay here again if it was a business visit based on the location. Faster than avg wifi (free) was a good plus.\n"

There is one more nuance/problem. Notice that there are only 6 "Expanded Reviews". This is because short reviews, which fit in the "Partial Review" format, do not have a "More" button. So you'd need to figure out which of the partial reviews are in fact full. Since you say you're learning R, I'll leave that to you...