I am attempting to scrape REI's Reviews (Hammocks) using Rselarium and Rvest. I want to hit the button at the bottom x amount of times so I can scrape all the reviews. I'm a little lost. Here's what I have so far. If you know too, how to preview in the finder what you're doing (not screen print) that would be cool. Thanks Stack Community.
replicate(100,
{
remDr$navigate("https://www.amazon.com/Eagles-Nest-Outfitters-DoubleNest-Portable/product-reviews/B00K30GXK8/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviewshttps://www.amazon.com/Eagles-Nest-Outfitters-DoubleNest-Portable/product-reviews/B00K30GXK8/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews")
webElem <- remDr$findElement("css", "body")
webElem$sendKeysToElement(list(key = "end"))
morereviews <- remDr$findElement(using = 'css selector', ".a-last a")
morereviews$clickElement
Sys.sleep(4)
reviews <- xml2::read_html(remDr$getPageSource()[[1]])%>%
rvest::html_nodes(".review-text")%>%
dplyr::data_frame(reviews = .)
})
Try this:
# Click the Load More button
replicate(100,
{
# scroll down
webElem <- remDr$findElement("css", "body")
webElem$sendKeysToElement(list(key = "end"))
# find button
morereviews <- remDr$findElement(using = 'css selector', "#BVRRContainer div.bv-content-pagination-container button")
# click button
morereviews$clickElement()
# wait
Sys.sleep(4)
})
# Scrap the reviews
reviews <- xml2::read_html(remDr$getPageSource()[[1]])%>%
rvest::html_nodes("#BVRRContainer div.bv-content-summary-body-text") %>%
rvest::html_text() %>%
dplyr::data_frame(reviews = .)
reviews
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.