网址在scrapy中重定向

Question

I am new to scrapy and trying to scrap hotels name for booking.com, just playing. The response URL is different from request URL. I also want to get Hotels names from all pages.

    class BookingSpider(CrawlSpider):
    name = 'booking.com_mumbai'
    allowed_domains = ['booking.com']
    start_urls = [
        'https://www.booking.com/searchresults.en-gb.html?aid=304142&label=gen173nr-1DCAEoggJCAlhYSDNiBW5vcmVmaGyIAQGYAS7CAQN4MTHIAQzYAQPoAQGSAgF5qAID&sid=73f533eb666233525bc516654c914549&checkin_month=4&checkin_monthday=26&checkin_year=2017&checkout_month=4&checkout_monthday=30&checkout_year=2017&class_interval=1&dest_id=20014181&dest_type=city&dtdisc=0&group_adults=2&group_children=0&inac=0&index_postcard=0&label_click=undef&mih=0&no_rooms=1&postcard=0&raw_dest_type=city&room1=A%2CA&sb_price_type=total&search_selected=1&src=index&src_elem=sb&ss=Los%20Angeles%2C%20California%2C%20USA%2C%20North%20America%2C%20CA&ss_all=0&ss_raw=Los%20Angeles%2C%20CA%3B&ssb=empty&sshis=0&rows=40&offset=40'
    ]

    rules = (
        Rule(LinkExtractor(allow='rows?.\w+', unique=True), follow=True, callback="parse"),
    )

    def parse_hotel_item(self):
        pass

    def parse(self, response):
        hotels = response.xpath('//*[@id="hotellist_inner"]/div')
        for hotel in hotels:
            print hotel.xpath("//h3/a/span[2]/text()").extract()
        print("Done")

logs

2017-04-15 23:40:48 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: scrap_booking_dot_com)
2017-04-15 23:40:48 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'scrap_booking_dot_com.spiders', 'SPIDER_MODULES': ['scrap_booking_dot_com.spiders'], 'ROBOTSTXT_OBEY': True, 'BOT_NAME': 'scrap_booking_dot_com'}
2017-04-15 23:40:48 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2017-04-15 23:40:48 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-04-15 23:40:48 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-04-15 23:40:48 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-04-15 23:40:48 [scrapy.core.engine] INFO: Spider opened
2017-04-15 23:40:48 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-04-15 23:40:48 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-04-15 23:40:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.booking.com/robots.txt> (referer: None)
2017-04-15 23:40:49 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.booking.com/searchresults.en-gb.html?aid=304142&label=gen173nr-1DCAEoggJCAlhYSDNiBW5vcmVmaGyIAQGYAS7CAQN4MTHIAQzYAQPoAQGSAgF5qAID&sid=73f533eb666233525bc516654c914549&checkin_month=4&checkin_monthday=26&checkin_year=2017&checkout_month=4&checkout_monthday=30&checkout_year=2017&class_interval=1&dest_id=20014181&dest_type=city&dtdisc=0&group_adults=2&group_children=0&inac=0&index_postcard=0&label_click=undef&mih=0&no_rooms=1&postcard=0&raw_dest_type=city&room1=A%2CA&sb_price_type=total&search_selected=1&src=index&src_elem=sb&ss=Los%20Angeles%2C%20California%2C%20USA%2C%20North%20America%2C%20CA&ss_all=0&ss_raw=Los%20Angeles%2C%20CA;&ssb=empty&sshis=0&rows=40&offset=40> from <GET https://www.booking.com/searchresults.en-gb.html?aid=304142&label=gen173nr-1DCAEoggJCAlhYSDNiBW5vcmVmaGyIAQGYAS7CAQN4MTHIAQzYAQPoAQGSAgF5qAID&sid=73f533eb666233525bc516654c914549&checkin_month=4&checkin_monthday=26&checkin_year=2017&checkout_month=4&checkout_monthday=30&checkout_year=2017&class_interval=1&dest_id=20014181&dest_type=city&dtdisc=0&group_adults=2&group_children=0&inac=0&index_postcard=0&label_click=undef&mih=0&no_rooms=1&postcard=0&raw_dest_type=city&room1=A%2CA&sb_price_type=total&search_selected=1&src=index&src_elem=sb&ss=Los%20Angeles%2C%20California%2C%20USA%2C%20North%20America%2C%20CA&ss_all=0&ss_raw=Los%20Angeles%2C%20CA%3B&ssb=empty&sshis=0&rows=40&offset=40>
2017-04-15 23:40:49 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.booking.com/searchresults.en-gb.html?dest_id=20014181;dest_type=city;offset=30;ss=Los%2520Angeles%252C%2520California%252C%2520USA%252C%2520North%2520America%252C%2520CA> from <GET https://www.booking.com/searchresults.en-gb.html?aid=304142&label=gen173nr-1DCAEoggJCAlhYSDNiBW5vcmVmaGyIAQGYAS7CAQN4MTHIAQzYAQPoAQGSAgF5qAID&sid=73f533eb666233525bc516654c914549&checkin_month=4&checkin_monthday=26&checkin_year=2017&checkout_month=4&checkout_monthday=30&checkout_year=2017&class_interval=1&dest_id=20014181&dest_type=city&dtdisc=0&group_adults=2&group_children=0&inac=0&index_postcard=0&label_click=undef&mih=0&no_rooms=1&postcard=0&raw_dest_type=city&room1=A%2CA&sb_price_type=total&search_selected=1&src=index&src_elem=sb&ss=Los%20Angeles%2C%20California%2C%20USA%2C%20North%20America%2C%20CA&ss_all=0&ss_raw=Los%20Angeles%2C%20CA;&ssb=empty&sshis=0&rows=40&offset=40>
2017-04-15 23:40:50 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.booking.com/searchresults.en-gb.html?dest_id=20014181;dest_type=city;offset=30;ss=Los%20Angeles%2C%20California%2C%20USA%2C%20North%20America%2C%20CA> from <GET https://www.booking.com/searchresults.en-gb.html?dest_id=20014181;dest_type=city;offset=30;ss=Los%2520Angeles%252C%2520California%252C%2520USA%252C%2520North%2520America%252C%2520CA>
2017-04-15 23:40:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.booking.com/searchresults.en-gb.html?dest_id=20014181;dest_type=city;offset=30;ss=Los%20Angeles%2C%20California%2C%20USA%2C%20North%20America%2C%20CA> (referer: None)

I am getting different results.

Answer 1

According to your logs and source code your scraper is starting at

https://www.booking.com/searchresults.en-gb.html?dest_id=20014181;dest_type=city;offset=30;ss=Los%20Angeles%2C%20California%2C%20USA%2C%20North%20America%2C%20CA

and then is redirected multiple times so it ends up scraping

https://www.booking.com/searchresults.en-gb.html?aid=304142&label=gen173nr-1DCAEoggJCAlhYSDNiBW5vcmVmaGyIAQGYAS7CAQN4MTHIAQzYAQPoAQGSAgF5qAID&sid=73f533eb666233525bc516654c914549&checkin_month=4&checkin_monthday=26&checkin_year=2017&checkout_month=4&checkout_monthday=30&checkout_year=2017&class_interval=1&dest_id=20014181&dest_type=city&dtdisc=0&group_adults=2&group_children=0&inac=0&index_postcard=0&label_click=undef&mih=0&no_rooms=1&postcard=0&raw_dest_type=city&room1=A%2CA&sb_price_type=total&search_selected=1&src=index&src_elem=sb&ss=Los%20Angeles%2C%20California%2C%20USA%2C%20North%20America%2C%20CA&ss_all=0&ss_raw=Los%20Angeles%2C%20CA%3B&ssb=empty&sshis=0&rows=40&offset=40

And there is nothing wrong with this.

Why?

That's the way booking.com handles your start url. It's nothing special happening to your scraper, the same happens when trying to access the start url in a browser.

Just paste your start url into a browser and you will see the browser's URL line change to an almost identical long URL.

So the response url is fine.

The real problem comes from the xpath "//h3/a/span[2]/text()" in the for loop.

There are two problems with it:

1. as the xpath starts with '/' it will work on the root object while you most probably want it to work on each of the hotel elements that you are iterating over.

The fix is to add a '.' to the front of the xpath (similar to using a '.' when navigating folders on a file system).

Example: ".//h3/a/span[2]/text()"

2. Nonetheless I couldn't make your xpath work. Crafted the following xpath instead which works fine for me: ".//span[@class='sr-hotel__name']/text()"

(a good xpath should rely preferable on ids and class names, instead of a particular nesting of elements, as the former are less at risk to change)