cx_freeze executable won't play nice with Geckodriver

Question

I'm working on a GUI desktop app to give users control over a web scraper. I have two executables, one runs a Selenium scraper (Py27) and one runs the GUI which starts and stops the scraper using subprocess.Popen (Python 38). Unfortunately, using the same Python version in both apps is not possible for me right now. Although we don't seem to be hitting environment problems because of the Python version discrepancy.

I'm on Windows, using Selenium 3.141, Geckodriver 0.26, cx_Freeze 5, and Firefox 77.

I am able to run both exes manually without problem, but when I try to run the scraper exe through the GUI exe, it gets caught in an infinite loop of creating a tmp directory with User.js inside, opening a geckodriver command prompt, and then closing and starting over. It ends up continually generating more and more of these tmp directories until I kill the GUI window. The traceback tells me a few things -- Geckodriver is hitting a permissions error, and the app is having trouble establishing the home directory.

I've tested starting a scrape process using Popen, so that's not the issue.

When I run it manually and dump the Firefox options, and compare it to the Firefox options when I run it through the GUI, there is some difference. This is from the FF profile on a manual (successful) run:

'userPrefs': 'c:\\users\\atadmin\\appdata\\local\\temp\\tmpqctvna\\user.js',
'profile_dir': 'c:\\users\\atadmin\\appdata\\local\\temp\\tmpqctvna',
'extensionsDir': 'c:\\users\\atadmin\\appdata\\local\\temp\\tmpqctvna\\extensions',

and this is the profile from an unsuccessful GUI run:

'userPrefs': 'c:\\users\\atadmin\\documents\\aeleads-master\\aeleads-portal-testing\\build\\exe.win-amd64-3.8\\tmpu9ktpl\\user.js',
'profile_dir': 'c:\\users\\atadmin\\documents\\aeleads-master\\aeleads-portal-testing\\build\\exe.win-amd64-3.8\\tmpu9ktpl',
'extensionsDir': 'c:\\users\\atadmin\\documents\\aeleads-master\\aeleads-portal-testing\\build\\exe.win-amd64-3.8\\tmpu9ktpl\\extensions',

And here is the latest traceback:

 Can't determine home directory

 utils.tss_logging: Logging initailizing with None/None


 Error: Traceback (most recent call last):

   File "li_scraper.py", line 63, in run_scraper

   File "C:\Users\atadmin\Documents\aeleads-master\aeleads-portal-testing\scrape\scraper.py", line 394, in collection

     self.login()

   File "C:\Users\atadmin\Documents\aeleads-master\aeleads-portal-testing\scrape\scraper.py", line 85, in login

     driver = self._get_driver(use_proxy=self.use_proxy)

   File "C:\Users\atadmin\Documents\aeleads-master\aeleads-portal-testing\scrape\scraper.py", line 622, in _get_driver

     use_proxy=use_proxy, reserve_proxy=False)

   File "C:\Users\atadmin\Documents\aeleads-master\aeleads-portal-testing\lib\webdrivers.py", line 164, in open_webdriver

     capabilities=capabilities, log_path="C:\\Users\\atadmin\\Documents\\aeleads-master\\geckodriver.log")

   File "C:\Python27\Py27\lib\site-packages\selenium\webdriver\firefox\webdriver.py", line 174, in __init__

     keep_alive=True)

   File "C:\Python27\Py27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 157, in __init__

     self.start_session(capabilities, browser_profile)

   File "C:\Python27\Py27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 252, in start_session

     response = self.execute(Command.NEW_SESSION, parameters)

   File "C:\Python27\Py27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute

     self.error_handler.check_response(response)

   File "C:\Python27\Py27\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response

     raise exception_class(message, screen, stacktrace)

 WebDriverException: Message: permission denied

It seems like cx_Freeze creates its own build environment, maybe causing me to lose the user, based off the permissions error. I'm fairly new to Python.

Answer 1

Figured it out. When passing the environment into Popen, I was passing a stripped out environment that didn't include the TEMP path. I passed in my full environment and then changed the variables that needed to be changed, and now it is working.