I want to crawl a website which supports only post data. I want to send the query params in post data in all the requests. How to achieve this?
POST requests can be made using scrapy's Request or FormRequest classes.
Also, consider using start_requests()
method instead of start_urls
property.
Example:
from scrapy.http import FormRequest
class myspiderSpider(Spider):
name = "myspider"
allowed_domains = ["www.example.com"]
def start_requests(self):
return [ FormRequest("http://www.example.com/login",
formdata={'someparam': 'foo', 'otherparam': 'bar'},
callback=self.parse) ]
Hope that helps.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.