Scrapy: How to get return values from Scrapy.Request?

Question

I'm currently trying to figure out how to get a return value from a scrapy request. The first snippet shows the way I'm doing the request. In the parse function (2nd snippet) I'm extracting some values which I would like to return with two lists. My question is: How can I get these two values after the yield statement in the first snippet?

        while i < pages:
        # default link is combined with the specific page number
        url = link+str(i)
        yield scrapy.Request(url=url, callback=self.parse)
        i = i+1

2nd snippet:

    def parse(self, response):

    # here the code is filling x_array and y_array with values

    return x_array, y_array

Thank you so much for your help!!

Answer 1

Instead of just returning values, Requests from Scrapy can fill up Items (a dictionary-like structure), which you can treat further in Item Pipelines. In your case, it suffices to add this to your item.py file:

from scrapy.item import Item, Field

class TestItem(Item):
    x_array = Field()
    y_array = Field()

In your spider, you fill up the item and yield it like this:

from ..items import TestItem
def parse(self, response):
    x_array = ['test1', 'test2']
    y_array = ['test3']
    item = TestItem()
    item['x_array'] = x_array
    item['y_array'] = y_array
    yield item