Execute Python scripts with Selenium via Crontab

Question

I have several python scripts that use selenium webdriver on a Debian server. If I run them manually from the terminal (usually as root) everything is ok but every time I tried to run them via crontab I have an exception like this:

WebDriverException: Message: Can't load the profile. Profile Dir: /tmp/tmpQ4vStP If you specified a log_file in the FirefoxBinary constructor, check it for details.

Try for example this script:

from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from pyvirtualdisplay import Display
from selenium import webdriver
import datetime
import logging

FIREFOX_PATH = '/usr/bin/firefox'

if __name__ == '__main__':
    cur_date = datetime.datetime.now().strftime('%Y-%m-%d')
    logging.basicConfig(filename="./logs/download_{0}.log".format(cur_date),
                        filemode='w',
                        level=logging.DEBUG,
                        format='%(asctime)s - %(levelname)s - %(message)s')
    try:
        display = Display(visible=0, size=(800, 600))
        display.start()
        print 'start'
        logging.info('start')
        binary = FirefoxBinary(FIREFOX_PATH,
                               log_file='/home/egor/dev/test/logs/firefox_binary_log.log')
        driver = webdriver.Firefox()
        driver.get("http://google.com")
        logging.info('title: ' + driver.title)
        driver.quit()
        display.stop()
    except:
        logging.exception('')
    logging.info('finish')
    print 'finish'

The crontab command for it:

0 13 * * * cd "/home/egor/dev/test" && python test.py

The log file for this script looks like this:

2016-09-27 16:30:01,742 - DEBUG - param: "['Xvfb', '-help']" 
2016-09-27 16:30:01,743 - DEBUG - command: ['Xvfb', '-help']
2016-09-27 16:30:01,743 - DEBUG - joined command: Xvfb -help
2016-09-27 16:30:01,745 - DEBUG - process was started (pid=23042)
2016-09-27 16:30:01,747 - DEBUG - process has ended
2016-09-27 16:30:01,748 - DEBUG - return code=0
2016-09-27 16:30:01,748 - DEBUG - stdout=
2016-09-27 16:30:01,751 - DEBUG - param: "['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1724']" 
2016-09-27 16:30:01,751 - DEBUG - command: ['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1724']
2016-09-27 16:30:01,751 - DEBUG - joined command: Xvfb -br -nolisten tcp -screen 0 800x600x24 :1724
2016-09-27 16:30:01,753 - DEBUG - param: "['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1725']" 
2016-09-27 16:30:01,753 - DEBUG - command: ['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1725']
2016-09-27 16:30:01,753 - DEBUG - joined command: Xvfb -br -nolisten tcp -screen 0 800x600x24 :1725
2016-09-27 16:30:01,755 - DEBUG - process was started (pid=23043)
2016-09-27 16:30:01,755 - DEBUG - DISPLAY=:1725
2016-09-27 16:30:01,855 - INFO - start
2016-09-27 16:30:31,965 - ERROR - 
Traceback (most recent call last):
  File "test.py", line 31, in <module>
    driver = webdriver.Firefox()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 103, in __init__
    self.binary, timeout)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/extension_connection.py", line 51, in __init__
    self.binary.launch_browser(self.profile, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 68, in launch_browser
    self._wait_until_connectable(timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 106, in _wait_until_connectable
    % (self.profile.path))
WebDriverException: Message: Can't load the profile. Profile Dir: /tmp/tmpQ4vStP If you specified a log_file in the FirefoxBinary constructor, check it for details.

2016-09-27 16:30:31,966 - INFO - finish

What I have tried:

1. Ensure that the script file is owned by root
2. Use export DISPLAY=:0; or export DISPLAY=:99; in crontab command
3. set the HOME variable in the crontab to the path of the home directory of the user that the cronjob was being run as

I'm really stuck with this problem.

I have python 2.7.10, selenium 2.53.6 with Xvbf and Firefox 47.0.1 on Debian 7.7

Answer 1

The problem concerns environment variables: cron is started by the system and knows nothing about user environments.

So, the solution to a problem is to make cron run a shell script that sets the needed environment variables first and then runs the script. In my case I needed to set the PATH varibable like this: PATH=/root/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Also, it may be useful to set HOME or DISPLAY varibles in some cases.

Answer 2

Try a hardcoded Firefox binary https://seleniumhq.github.io/selenium/docs/api/py/webdriver_firefox/selenium.webdriver.firefox.firefox_binary.html

selenium.webdriver.firefox.firefox_binary.FirefoxBinary("/your/binary/location/firefox")
driver = webdriver.Firefox(firefox_binary=binary)