Pyinstaller not allowing multiprocessing with MacOS

Question

I have a python file that I would like to package as an executable for MacOS 11.6.

The python file (called Service.py) relies on one other json file and runs perfectly fine when running with python. My file uses argparse as the arguments can differ depending on what is needed.

Example of how the file is called with python:

python3 Service.py -v Zephyr_Scale_Cloud https://myurl.cloud/ philippa@email.com password1 group3

The file is run in exactly the same way when it is an executable:

./Service.py -v Zephyr_Scale_Cloud https://myurl.cloud/ philippa@email.com password1 group3

I can package the file using PyInstaller and the executable runs.

Command used to package the file:

pyinstaller --paths=/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/ Service.py

However, when I get to the point that requires multiprocessing, the arguments get lost. My second argument (here noted as https://myurl.cloud ) is a URL that I require.

The error I see is:

[MAIN] Starting new process RUNID9157
url before constructing the client recognised as pipe_handle=15
usage: Service [-h] test_management_tool url
Service: error: the following arguments are required: url
Traceback (most recent call last):
  File "urllib3/connection.py", line 174, in _new_conn
  File "urllib3/util/connection.py", line 72, in create_connection
  File "socket.py", line 954, in getaddrinfo

I have done some logging and the url does get correctly read. But as soon as the process started and picking up what it needs to, the url is changed to 'pipe_handle=x', in the code above it is pipe_handle=15. I need the url to retrieve an authentication token, but it just stops being read as the correct value and is changed to this pipe_handle value. I have no idea why.

Has anyone else seen this?!

I am using Python 3.9, PyInstaller 4.4 and ArgParse. I have also added

if __name__ == "__main__":
    if sys.platform.startswith('win'):
        # On Windows - multiprocessing is different to Unix and Mac.
        multiprocessing.freeze_support() 

to my if name = "main" section as I saw this on other posts but it doesn't help.

Can someone please assist?

Answer 1

Sending commands via sys.argv is complicated by the fact that multiprocessing's "spawn" start method uses that to pass the file descriptors for the initial communication pipes between the parent and child.

I'm projecting here a little because you did not share the code of how/where you call argparse, and how/where you call multiprocessing

If you are parsing args outside of if __name__ == "__main__": , the args may get parsed (re-parsed on child import __main__ ) before sys.argv gets automatically cleaned up by multiprocessing.spawn.prepare() in the child. You should be able to fix this by moving the argparse stuff inside your target function. It also may be easier to parse the args in the parent, and simply send the parsed results as an argument to the target function. See this answer of mine for further discussion on sys.argv with multiprocessing .