在Scrapy中传递xPath转换功能不适用于特殊字符

Question

I'm building a Scrapy spider which takes the xpath query as an input parameter.

The specific page I'm trying to scrape has line feeds, new lines and other characters within the price text field and I'm using the translate() function to remove them.

The selector works fine with the translate if explicitly included in the code but the translate doesn't work if passed as a parameter.

Here is my spider:

import scrapy
from spotlite.items import SpotliteItem


class GenericSpider(scrapy.Spider):
   name = "generic"
   xpath_string = ""

   def __init__(self, start_url, allowed_domains, xpath_string, *args, **kwargs):
       super(GenericSpider, self).__init__(*args, **kwargs)
       self.start_urls = ['%s' % start_url]
       self.allowed_domains = ['%s' % allowed_domains]
       self.xpath_string = xpath_string

    def parse(self, response):
       self.logger.info('URL is %s', response.url)
       self.logger.info('xPath is %s', self.xpath_string)
       item = SpotliteItem()
       item['url'] = response.url
       item['price'] = response.xpath(self.xpath_string).extract()
       return item

And I use the following to call the spider.

scrapy crawl generic -a start_url=https://www.danmurphys.com.au/product/DM_4034/penfolds-kalimna-bin-28-shiraz -a allowed_domains=danmurphys.com.au -a "xpath_string=translate((//span[@class='price'])[1]/text(),',$\r\n\t','')"

The issue seems to be passing specical characters in the argument ie \\r\\n\\t.

The '$'character is correctly removed but the \\r\\n\\t characters are not as per the output below.

{'price': [u'\r\n\t\t\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\t\t\t\t27.50\r\n\t\t\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\t\t'],
 'url': 'https://www.danmurphys.com.au/product/DM_4034/penfolds-kalimna-bin-28-shiraz.jsp;jsessionid=B0211294F13A980CA41261379CD83541.ncdlmorasp1301?bmUID=loERXI6'}

Any assistance or advice will be appreciated!

Thanks,

Michael

Answer 1

Try using the normalize-space() XPath function in your selector:

scrapy crawl generic -a start_url=<URL> -a \
    allowed_domains=danmurphys.com.au \
    -a "xpath_string=normalize-space(//span[@class='price'][1]/text())"

In your parse method, you can use the extract_first() to get the price as a single string object, instead of a list:

item['price'] = response.xpath(self.xpath_string).extract_first()

You could also use the re_first() method to remove the $ sign from the string:

item['price'] = response.xpath(self.xpath_string).re_first("\$(.+)")