Python request.post() not working when converted to Python Scrapy Request

Question

I have simple POST request code.

headers = {
    dictionary of headers
}

params = (
    ('param1', '0'),
    ('param2', '5668294380'),
    ('param3', '8347915011'),
)

response = requests.post('https://website.com', headers=headers, params=params, data=__data)

This works perfectly as standalone Python program.

But I want to do this in Python Scrapy

Request(url='https://website.com',callback=self.callback_fun, headers=headers, body=__data, method="POST")

It gives me response that URL cannot handle POST request

I tried

FormRequest(url='https://website.com',callback=self.callback_fun, headers=headers, body=__data)

It gives me same response.

I tried

Request(url='https://website.com?' + urllib.urlencode(self.params),callback=self.callback_fun, headers=headers, body=__data, method="POST")

But it gives me 400 Bad Request

Whats wrong with Scrapy? I mean pure Python Script works but in Scrapy does not work.

I think main issue is how to send params=params using Scrapy. Scrapy only allows to send Request Payload via body parameter

Answer 1

class scrapy.http.FormRequest(url[, formdata, ...])

Parameters: formdata (dict or iterable of tuples) – is a dictionary (or iterable of (key, value) tuples) containing HTML Form data which will be url-encoded and assigned to the body of the request.

in HTTP, if you want to post data, the data is set in the request body and encoded. you can encode the dict you self or use Scrapy FormRequest :

class FormRequest(Request):

def __init__(self, *args, **kwargs):
    formdata = kwargs.pop('formdata', None)
    if formdata and kwargs.get('method') is None:
        kwargs['method'] = 'POST'

    super(FormRequest, self).__init__(*args, **kwargs)

    if formdata:
        items = formdata.items() if isinstance(formdata, dict) else formdata
        # encode dict here
        querystr = _urlencode(items, self.encoding)
        if self.method == 'POST':
            # set message header
            self.headers.setdefault(b'Content-Type', b'application/x-www-form-urlencoded')
            # set message body
            self._set_body(querystr)
        else:
            self._set_url(self.url + ('&' if '?' in self.url else '?') + querystr)

----------------------------update--------------

in requests code:

response = requests.post('https://website.com', headers=headers, params=params, data=__data)

it first adds the parameter to the URL the post data to the modified URL. you should change you URL. you can get the URL by:

print(response.url)