How to get response from scrapy.Request without callback?

Question

I want to send a request and wait for a response from the server in order to perform action-dependent actions. I write the following

resp = yield scrapy.Request(*kwargs)

and got None in resp. In documentation I find that need to use call_back function, but this function call after processing next commands. How to wait response from server?

Answer 1

I found the inline_requests module which has inline_requests decorator.

It solved my problem.

Answer 2

This isn't really how scrapy should be used, as waiting for a response is the same as using a callback. If you need to keep processing previous responses in conjunction with the new one, you can always pass and keep passing the response on the meta argument .

Now, to make this sometimes more readable you can also use scrapy-inline-requests which makes exactly the same as explained before under the hood, as it doesn't stop scrapy but makes the following request in order (same as doing a request after another with callbacks).

If using scrapy-inline-requests please be careful on making the methods to only be generators and also sending new requests or items when a new inline request is being processed.

Answer 3

it's not an answer to this question, but is alternative how to get response object and parse it using xpath. Here I use requests, bs4 and lxml libraries.

import requests
from bs4 import BeautifulSoup
from lxml import etree

url = 'your_url'
soup = BeautifulSoup(requests.get(url).text, 'html.parser')
dom = etree.HTML(str(soup))
target_data = dom.xpath("//div......target path......")