如何在Scrapy中使用全局变量解析响应？

Question

I modified my start_requets method to be:

def start_requests(self):
    array = list(open("file", 'r'))
    for i in array:
        yield self.make_requests_from_url("http://example.org/test.php?id=" + i)

How can I access the value of i in the parse method?

I tried setting a global variable idd and adding

global idd
idd = i

in the start_requests method and then

def parse(self, response):
    item = DataItem()
    item['id'] = idd

But all of items' id fields were filled with idd's last value.

How do I fix this?

Answer 1

One (and probably the easiest) option would be to pass it inside meta :

yield scrapy.Request("http://example.org/test.php?id=" + i, 
                     meta={"index": i},
                     dont_filter=True)

Then, read it in parse() :

def parse(self, response):
    index = response.meta["index"]