从脚本运行 scrapy 蜘蛛

Question

I would like to run my scrapy sprider from python script. I can call my spider with the following code,

subprocess.check_output(['scrapy crawl mySpider'])

Untill all is well. But before that, I instantiate the class of my spider by initializing the start_urls, then the call to scrapy crawl doesn't work since it doesn't find the variable start_urls.

from flask import Flask, jsonify, request
import scrapy
import subprocess

class ClassSpider(scrapy.Spider):
    name        = 'mySpider'
    #start_urls = []
    #pages      = 0
    news        = []

    def __init__(self, url, nbrPage):
        self.pages      = nbrPage
        self.start_urls = url

    def parse(self):
        ...

    def run(self):
        subprocess.check_output(['scrapy crawl mySpider'])
        return self.news

app = Flask(__name__)
data = []

@app.route('/', methods=['POST'])
def getNews():
    mySpiderClass = ClassSpider(request.json['url'], 2)

    data.append(mySpider.run())
    return jsonify({'data': data})

if __name__ == "__main__":
    app.run(debug=True)

The error I get is: TypeError: init missing 1 required positional argument: 'start_url' and 'pages'

any help please?

Answer 1

Another way to start your spider from a script (and provide arguments):

from scrapy.crawler import CrawlerProcess
from path.to.your.spider import ClassSpider
from scrapy.utils.project import get_project_settings

process = CrawlerProcess(get_project_settings())
process.crawl(
    ClassSpider,
    start_urls, # you need to define it somewhere
    number_of_pages, # you need to define it somewhere
)
process.start()

Answer 2

The reason you are getting this error message is that you start the crawling process with the command scrapy crawl mySpider which creates a new instance of ClassSpider . It does so without passing url and nbrPage .
It could work if you replaced subprocess.check_output(['scrapy crawl mySpider']) with subprocess.check_output([f'scrapy crawl mySpider -a url={self.start_urls} nbrPage={self.pages}']) . Also you should make sure that start_urls is a list.
However, then you would still create two separate instances of the same spider, so I would suggest that you should implement run as a function taking url and nbrPage as arguments.
There are also other methods of using Scrapy and Flask in the same script. For that purpose check this question .