如何将发布参数推送到scrapy-redis

Question

I have a post request like

def start_requests(self):
    yield FormRequest(url,formdata={'id': "parameter from redis"})

Can I use redis-cli lpush to save post parameter and that my crawler run it?

Answer 1

By default the scrapy-redis queue working only with url as messages. One message = one url. But you can modify this behavior. For example you can use some object for your messages/requests:

    class ScheduledRequest:
        def __init__(self, url, method, body)
            self.url = url
            self.method = method
            self.body = body

Pass it to queue as json encoded dic:

    redis.lpush(
        queue_key,
        json.dumps(
            ScheduledRequest(
               url='http://google.com',
               method='POST',
               body='some body data ...'
            ).__dict__
        )
    )

And rewrite the make_request_from_data and schedule_next_requests methods:

class MySpiderBase(RedisCrawlSpider, scrapy.Spider):

    def __init__(self, *args, **kwargs):
        super(MySpiderBase, self).__init__(*args, **kwargs)

    def make_request_from_data(self, data):
        scheduled = ScheduledRequest(
            **json.loads(
                bytes_to_str(data, self.redis_encoding)
            )
        )
        # here you can use and FormRequest
        return scrapy.Request(url=scheduled.url, method=scheduled.method, body=scheduled.body)

   def schedule_next_requests(self):
       for request in self.next_requests():
           self.crawler.engine.crawl(request, spider=self)

   def parse(self, response):
       pass