Passing extra arguments to scrapy.Request()

Question

Actually I want to store all the data(text,hrefs,images) related to specific website to a single folder.In order to do that I need to pass the path for that folder to all different parsing function.So I want to pass this path as extra kwargs in scrapy.Request() like this:

yield scrapy.Request(url=url,dont_filter=True, callback=self.parse,errback = self.errback_function,kwargs={'path': '/path/to_folder'})

But it gives the error TypeError: __init__() got an unexpected keyword argument 'kwargs'

How can I pass that path to next function?

Answer 1

For anyone who may need it......

You can pass extra arguments by using meta arguments like this...

   yield scrapy.Request(url=url,dont_filter=True, 
callback=self.parse,errback = self.errback_function,  meta={'filepath': filepath})

UPDATE:

Request.cb_kwargs was introduced in version 1.7. Prior to that, using Request.meta was recommended for passing information around callbacks. After 1.7, Request.cb_kwargs became the preferred way for handling user information, leaving Request.meta for communication with components like middlewares and extensions.

So for version >= 1.7 following would work :

 request = scrapy.Request('http://www.example.com/index.html', callback=self.parse_page2, cb_kwargs=dict(main_url=response.url))

you can refer to this documentation: https://doc.scrapy.org/en/latest/topics/request-response.html#passing-additional-data-to-callback-functions

Answer 2

It is an old topic, but for anyone who needs it, to pass an extra parameter you must use cb_kwargs , then call the parameter in the parse method.

You can refer to this part of the documentation.