Running script using python3 instead of default python 2.7

Question

I have written a python program to scrape certain websites... I am using scrapy which is a python framework.

I have setup the following cron job to run the scraping program:

# m h  dom mon dow   command
PATH=/usr/local/bin
1 6 * * * cd /home/ubuntu/scrapers/my_scraper && /home/ubuntu/.local/bin/scrapy crawl my_spider >> /tmp/scraper.log 2>&1

The above program is using python 2.7. I want it to be run using python3...

I have install python3 and pip3 and have added the following alias to.bashrc:

# .bashrc
alias python=python3
alias pip=pip3

But when the cronttab is executed, I am getting the following error:

  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/ubuntu/scrapers/my_scraper/my_scraper/spiders/spider.py", line 3, in <module>
    from python_json_config import ConfigBuilder

The error message shows that my program is being run by python 2.7

This program used to work fine with Python 2, but I have added python_json_config import and I am not able to install this package using python 2.. that's why I want to run the program using python3

Background

I don't need to update the Puthon version, I just need to run the scrapy script using python 3. I followed this question and the accepted answer here changes the.bashrc, that's why I did that.

I installed python using apt-get and installed scrapy using pip... I had not added any alias in.bashrc at the time of installation.

My Ubuntu server is an Amazon EC2 and I connect to to it using Putty.

Answer 1

Ubuntu has a special line that stops .bashrc from running in non-interactive shells.

You can change this behavior easily , and add a line to run the .bashrc before you run the script:

1 6 * * * source /home/ubuntu/.bashrc && cd /home/ubuntu/scrapers/my_scraper && /home/ubuntu/.local/bin/scrapy crawl my_spider >> /tmp/scraper.log 2>&1

Another option is to modify the cronjob line to execute the script directly using Python 3:

1 6 * * * cd /home/ubuntu/scrapers/my_scraper &&  python3 /home/ubuntu/.local/bin/scrapy crawl my_spider >> /tmp/scraper.log 2>&1