Run Anaconda-Scrapy as Cron job

Question

I am trying to run a scrapy (installed with anaconda2) crawler as a cron job. No luck so far.

Here is cron command:

* * * * * sh /home/username/anaconda2/folder/getdata.sh

The cron runs every 1 min ( I checked the log file, it seem to be ok) And here is the code inside the sh file:

#!/home/username/anaconda2/bin/python
cd /home/username/anaconda2/crawler_folder
PATH=$PATH:/home/username/anaconda2/bin
export PATH
scrapy crawl spider

The sh code executes as expected in the terminal but not as a cron job. Can you please help

Answer 1

This worked for me as the .sh file and the cron job was unchanged.

#!/home/username/anaconda2/bin/bash
PATH=/home/username/anaconda2/bin:home/username/anaconda2/sbin
cd /home/username/anaconda2/crawler_folder && scrapy crawl spider