Scrapy: Pass variable from Middleware to Spider itself

Question

I am trying to capture the raw request payload and request headers for the purpose of tracking inside my database. I know about the response.request.headers but that is the returned request headers.

Would it be possible to create a middleware that captures the request.headers and payload (body) and send that to the spider as a meta tag or anything like that?

Answer 1

I found a way to do it (granted without the middleware):

• Store the scrapy.Request() into a variable eg req
• Store req.headers.to_unicode_dict() into self.req_headers
• Store req.body into self.req_body
• Execute yield req to send the request