Executing Django management commands that spins off multiple processes and threads in windows and linux

Question

I am relatively new to multi-threading and multi-processing. I just encountered another learn-block when i just realized that windows and linux handles multi-processing very differently. I do not know th technicalities, but I do know that it is different.

I am using a django to execute my application: python manage.py random_script , within random_script, I am importing multiprocessing and spinning of different processes. i get the following error:

  File "<string>", line 1, in <module>
  File "C:\FAST\Python\3.6.4\lib\multiprocessing\spawn.py", line 99, in spawn_main
    new_handle = reduction.steal_handle(parent_pid, pipe_handle)
  File "C:\FAST\Python\3.6.4\lib\multiprocessing\reduction.py", line 82, in steal_handle
    _winapi.PROCESS_DUP_HANDLE, False, source_pid)
OSError: [WinError 87] The parameter is incorrect

I tried adding this at the top because my development server is windows but my production server is linux:

if 'win' in sys.platform:
    print('Window')
    multiprocessing.set_start_method('spawn')
else:
    print('Linux')
    multiprocessing.set_start_method('fork')

But to no success. When i continued to look through google, it suggest writing the portion of the process spawning under the if __name__ == '__main__': line. That would be fine if I am executing my scripts normally (ie python random_script.py ), but I am not. I have ran out of ideas and no longer know how to proceed.

++ EDITED ++

manage.py

#!/usr/bin/env python
import os
import sys
import argparse

DEFAULT_SETTINGS_MODULE = "api.test_settings"

if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", DEFAULT_SETTINGS_MODULE)
    try:
        from django.core.management import execute_from_command_line
    except ImportError:
        # The above import may fail for some other reason. Ensure that the
        # issue is really that Django is missing to avoid masking other
        # exceptions on Python 2.
        try:
            import django
        except ImportError:
            raise ImportError(
                "Couldn't import Django. Are you sure it's installed and "
                "available on your PYTHONPATH environment variable? Did you "
                "forget to activate a virtual environment?"
            )
        raise
    execute_from_command_line(sys.argv)

random_script.py:

class Command(BaseCommand):
    def __init__(self):
        super().__init__()

    def handle(self, *args, **kwargs):
        <...>
        self.main()

    def main(self):
        <...>

Above is my manage.py and my random_script.py.

Thanks for the guidance

Answer 1

Every app has main module which inits / starts it.

For Django manually run management commands this is manage.py and you can set desired method in there:

# manage.py
...

if __name__ == "__main__":
    import multiprocessing
    if 'win' in sys.platform:
        multiprocessing.set_start_method('spawn')
    else:
        multiprocessing.set_start_method('fork')

    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings")
...

And sample of custom management command:

# random_script.py


def calculation(x):
    import time
    time.sleep(1)
    return x


class Command(BaseCommand):

    def handle(self, *args, **options):
        calc_args = [1, 2, 3, 4, 5]
        with multiprocessing.Pool(processes=3) as pool:
            results = pool.map(calculation, calc_args)
        self.stdout.write(
            self.style.SUCCESS('Success: %s' % results)
        )