Scrapy：用同样的规则重新爬行

Question

i am using scrapy to extract reviews from tripAdvisor.

my start_urls are the hotels. for example:

http://www.tripadvisor.com/Hotel_Review-g60763-d80075-Reviews-Amsterdam_Court_Hotel-New_York_City_New_York.html#REVIEWS.

from this pages i am crawling to the reviews page, using this rules:

rules = (
    Rule(SgmlLinkExtractor(allow=("ShowUserReviews-g.*",), restrict_xpaths=('//*[@id="REVIEWS"]/div[4]/div/div[2]/div/div/div[1]/a',), unique=True), callback='parse_item', follow= True),

    Rule(SgmlLinkExtractor(allow=("ShowUserReviews-g.*",),restrict_xpaths=('//*[@id="REVIEWS"]/div[contains(@class,"deckTools btm")]',),unique=True),callback='parse_item',follow=True),
)

an example for reviews page:

http://www.tripadvisor.com/ShowUserReviews-g187514-d228523-r275442835-Hotel_Petit_Palace_Arturo_Soria-Madrid.html#REVIEWS

at the end of every reviews page there are links for the next reviews pages for this hotel, numbered like : 1, 2, 3, 4 ..... i can use the same rule i think, the next addresses are similar.

follow this printscreen:

http://s16.postimg.org/w68m82ouc/Screenshot_from_2015_07_02_12_36_03.jpg

my questions:

1. how the rule crawling works? the scraper can re-crawling to the next reviews pages with the same rule? i need something else?

2. how can i avoid crawling to reviews pages i saw before? for example, crawl from page no.3 to page no.1 and 2..?

thanks

Answer 1

Filter for the "Next" site in the Rule for the next sites. This avoids visiting already visited review-sites.

Rule(SgmlLinkExtractor(allow=("ShowUserReviews-g.*",), restrict_xpaths=('//*[@id="REVIEWS"]/div[4]/div/div[2]/div/div/div[1]/a[text() = "Next"]',), unique=True), callback='parse_item', follow= True)