[英]Python request.post() not working when converted to Python Scrapy Request
I have simple POST request code. 我有简单的POST请求代码。
headers = {
dictionary of headers
}
params = (
('param1', '0'),
('param2', '5668294380'),
('param3', '8347915011'),
)
response = requests.post('https://website.com', headers=headers, params=params, data=__data)
This works perfectly as standalone Python program. 这完全可以作为独立的Python程序使用。
But I want to do this in Python Scrapy 但是我想在Python Scrapy中做到这一点
Request(url='https://website.com',callback=self.callback_fun, headers=headers, body=__data, method="POST")
It gives me response that URL cannot handle POST request 它给我的回应是URL无法处理POST请求
I tried 我试过了
FormRequest(url='https://website.com',callback=self.callback_fun, headers=headers, body=__data)
It gives me same response. 它给了我同样的回应。
I tried 我试过了
Request(url='https://website.com?' + urllib.urlencode(self.params),callback=self.callback_fun, headers=headers, body=__data, method="POST")
But it gives me 400 Bad Request
但这给了我400 Bad Request
Whats wrong with Scrapy? Scrapy怎么了? I mean pure Python Script works but in Scrapy does not work. 我的意思是纯Python脚本有效,但在Scrapy中不起作用。
I think main issue is how to send params=params
using Scrapy. 我认为主要问题是如何使用Scrapy发送params=params
。 Scrapy only allows to send Request Payload via body
parameter Scrapy仅允许通过body
参数发送请求有效负载
class scrapy.http.FormRequest(url[, formdata, ...])
Parameters: formdata (dict or iterable of tuples) – is a dictionary (or iterable of (key, value) tuples) containing HTML Form data which will be url-encoded and assigned to the body of the request. 参数:formdata(元组的dict或可迭代)–是一个字典(或元组(键,值)可迭代),其中包含HTML表单数据,该数据将被url编码并分配给请求的主体。
in HTTP, if you want to post data, the data is set in the request body and encoded. 在HTTP中,如果要发布数据,则将数据设置在请求正文中并进行编码。 you can encode the dict you self or use Scrapy FormRequest
: 您可以编码自己的字典,也可以使用Scrapy FormRequest
:
class FormRequest(Request):
def __init__(self, *args, **kwargs):
formdata = kwargs.pop('formdata', None)
if formdata and kwargs.get('method') is None:
kwargs['method'] = 'POST'
super(FormRequest, self).__init__(*args, **kwargs)
if formdata:
items = formdata.items() if isinstance(formdata, dict) else formdata
# encode dict here
querystr = _urlencode(items, self.encoding)
if self.method == 'POST':
# set message header
self.headers.setdefault(b'Content-Type', b'application/x-www-form-urlencoded')
# set message body
self._set_body(querystr)
else:
self._set_url(self.url + ('&' if '?' in self.url else '?') + querystr)
----------------------------update-------------- ----------------------------更新--------------
in requests code: 在请求代码中:
response = requests.post('https://website.com', headers=headers, params=params, data=__data)
it first adds the parameter to the URL the post data to the modified URL. 它首先将参数添加到URL,将发布数据添加到修改后的URL。 you should change you URL. 您应该更改您的URL。 you can get the URL by: 您可以通过以下方式获取网址:
print(response.url)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.