How to get back to the previous spider in Scrapy

Question

I'm trying to do one thing. I have a spider written using Scrapy. It has two parse function like the following

def parse(self, response):

   ...
   for var in dict
      ...

      scrapy.Request(link + var, callback=self.parse_info)


 def parse_info(self, response)

    ...
    do something

What I would like to do is getting data in parse_info and after finished come back to the parse method to continue the iteration. Does exist a way to do this in scrapy?

Answer 1

You will use 'Yield' in your parse call and 'return' in your parse_info call

def parse(self, response):

...
for var in dict
  ...

  yield Request(item['page_url'], callback=self.parse_info)


def parse_info(self, response)

  ...
  do something
  return something