python scrapy忽略start_url

Question

i am trying to scrap a website but the start_url i am using is not crawled. I put in the line print(response) to see what was happening and the output i get is (it seems that the start_url is ignored in favour of https://www.purplebricks.com/search?ref=header ):

c:\Users\andrew\Documents\Big Data Project\Data Collectors\PurpleBricks\Purplebricks>scrapy crawl purplebricks
2017-02-05 22:18:31 [scrapy] INFO: Scrapy 1.0.5 started (bot: Purplebricks)
2017-02-05 22:18:31 [scrapy] INFO: Optional features available: ssl, http11
2017-02-05 22:18:31 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'Purplebricks.spiders', 'SPIDER_MODULES': ['Purplebricks.spiders'], 'BOT_NAME': 'Purplebricks'}
2017-02-05 22:18:31 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2017-02-05 22:18:31 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2017-02-05 22:18:31 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2017-02-05 22:18:31 [scrapy] INFO: Enabled item pipelines:
2017-02-05 22:18:31 [scrapy] INFO: Spider opened
2017-02-05 22:18:31 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-02-05 22:18:31 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-02-05 22:18:32 [scrapy] DEBUG: Crawled (200) <GET https://www.purplebricks.com/search?ref=header#?q&location=se3&page=1&latitude=51.4688273310245&longitude=0.0176656312203414&searchType=ForSale&sortBy=1> (referer: None)
<200 https://www.purplebricks.com/search?ref=header>
2017-02-05 22:18:32 [scrapy] INFO: Closing spider (finished)
2017-02-05 22:18:32 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 235,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 13413,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 2, 5, 22, 18, 32, 326000),
 'log_count/DEBUG': 2,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2017, 2, 5, 22, 18, 31, 913000)}
2017-02-05 22:18:32 [scrapy] INFO: Spider closed (finished)

the code i am using is below. I can sort of get it to work with selenium though i need to force the driver to pull up the webpage - even with selenium it will not open the start_url

import scrapy

from Purplebricks.items import PurplebricksItem

class purplebricksSpider(scrapy.Spider):
    name = "purplebricks"
    allowed_domains = ["purplebricks.com"]
    start_urls = ["https://www.purplebricks.com/search?ref=header#?q&location=se3&page=1&latitude=51.4688273310245&longitude=0.0176656312203414&searchType=ForSale&sortBy=1",
                  ]

    def parse(self, response):
        print(response)
        for sel in response.xpath('//*[@class="row properties"]'):
            item = PurplebricksItem()
            prices = sel.xpath('//*/p/span[@data-bind="formatCurrency: property.marketPrice, roundDecimalsTo: 0"]/text()').extract()
            prices = [price.strip() for price in prices]
            property_ids = sel.xpath('//*[@class="title"]/div[@class="primary"]/a/@href').re(r'(?<=property-for-sale\/)(.*?)(?=\/#)')
            property_ids = [property_id.strip() for property_id in property_ids]
            addresses = sel.xpath('//*[@class="title"]/div[@class="primary"]/a/p[@class="type"]/text()').extract()
            addresses = [address.strip() for address in addresses]
            descriptions = sel.xpath('//*[@class="title"]/div[@class="primary"]/a/p[@class="address"]/text()').extract()
            descriptions = [description.strip() for description in descriptions]

            result = zip(prices, property_ids, addresses, descriptions)
            for price, property_id, address, description in result:
                item['price'] = price
                item['property_id'] = property_id
                item['description'] = description
                item['address'] = address
                yield item

any views as to what is occurring here?

Answer 1

Remove # from url. It has special meaning. Normally with chars on right side it describes place on current page but scrapy doesn't check what is on rigth size of # and strip all.