如何在scrapy蜘蛛的start_urls中发送帖子数据

Question

I want to crawl a website which supports only post data. I want to send the query params in post data in all the requests. How to achieve this?

Answer 1

POST requests can be made using scrapy's Request or FormRequest classes.

Also, consider using start_requests() method instead of start_urls property.

Example:

from scrapy.http import FormRequest

class myspiderSpider(Spider):
    name = "myspider"
    allowed_domains = ["www.example.com"]

    def start_requests(self):
        return [ FormRequest("http://www.example.com/login",
                     formdata={'someparam': 'foo', 'otherparam': 'bar'},
                     callback=self.parse) ]

Hope that helps.