How do I restart the scrapyd daemon?

Question

I've installed the scrapyd daemon on an EC2 server exactly as described in the documentation . Now I've changed some of the configuration variables in /etc/scrapyd/conf.d/000-default .

How do I get scrapyd to recognize those changes? I assume it involves restarting the daemon, but I can't find any good guidance on how to do so.

One complicating factor: I have a bunch of crawls queued up, and I'd rather not lose them. I think scrapy knows how to quit and resume them gracefully, but this feature isn't well-documented. Any guidance?

Answer 1

This turned out to be pretty simple.

Kill the process like this:

kill -INT $(cat /var/run/scrapyd.pid)

Then restart it like this:

/usr/bin/python /usr/local/bin/twistd -ny /usr/share/scrapyd/scrapyd.tac -u scrapy -g nogroup --pidfile /var/run/scrapyd.pid -l /var/log/scrapyd/scrapyd.log &

As far as I can tell, both commands need to be run as root.

Answer 2

If you have installed scrapyd via apt on an Ubuntu or Debian machine, the following should work:

sudo service scrapyd reload

Whether it will persist your scheduled spider runs I do not know know, however.

Answer 3

Debian/Ubuntu：

sudo /etc/init.d/scrapyd restart