使用scrapy时，在提取值时忽略空白值

Question

from scrapy.spider import BaseSpider
from scrapy.selector import Selector
from sample3.items import taamaaItem

class taamaaSpider(BaseSpider):
   name = "taamaa"
   allowed_domains = ["taamaa.com"]
   start_urls = [
       "http://www.taamaa.com/store-directory/"]

   def parse(self, response):
       sel = Selector(response)
       sites = sel.xpath('//div/div[@class="section clearfix col-md-12"]')
       items = []
       list1 = []
       list2 = []
       for site in sites:
           list1 = sites[0].xpath('//div[@class="pull-left col-md-3 merchant"]/div[@class="name"]/a/text()').extract()
           list2 = sites[0].xpath('//div[@class="pull-left col-md-3 merchant"]/div[@class="url"]/a/text()').extract()
       for index in range(len(list2)):
           td = taamaaItem()
           td['name'] = list1[index] 
           td['link'] = list2[index] 
           items.append(td)
       return items

While extracting data it leaves the blank value and fetches the next value of link, thus incorrecting my data alignment.

Example if A = a , B = , C = c, D = d, E = e

it fetches the output A = a , B = c , C = d , D = e , E = a

and I want the output to be like this

A = a , B = , C = c, D = d, E = e

how can I achieve this.

Answer 1

I see 2 strange things:

• you are using absolute XPath expressions in your loop
• and you applying them to sites[0] in your loop for each iteration

For your problem grouping 2 lists with some empty text elements, you can use the same structure with a loop on sites but extracting name and link in each iteration, so you don't need intermediate lists

from scrapy.spider import BaseSpider
from scrapy.selector import Selector
from sample3.items import taamaaItem

class taamaaSpider(BaseSpider):
   name = "taamaa"
   allowed_domains = ["taamaa.com"]
   start_urls = [
       "http://www.taamaa.com/store-directory/"]

   def parse(self, response):
       sel = Selector(response)
       sites = sel.xpath('//div/div[@class="section clearfix col-md-12"]')
       items = []
       for site in sites:
           td = taamaaItem()           
           td['name'] = site.xpath("""
                .//div[@class="pull-left col-md-3 merchant"]
                    /div[@class="name"]/a/text()""").extract()
           td['link'] = site.xpath("""
                .//div[@class="pull-left col-md-3 merchant"]
                    /div[@class="url"]/a/text()""").extract()
           items.append(td)
       return items

See how I use relative XPath expression ( .//div...... )