Running multiple scripts from a single Script in python

Question

I have two scripts Server.py and ServerGUI.py . I want them to run independently and in parallel. Say I make another script main.py . How can I run Server.py and ServerGUI.py from main.py ?

Can you suggest me the code for main.py ?

Answer 1

To run 2 or more scripts from within a python script, you can use the subprocess package with nohup. This will run each script in the background allowing you to run them in parallel from the same source script. Also, as an option, this example will save the standard output from each script in a different file

import os
from subprocess import call
from subprocess import Popen


# subprocess.call(['python', 'exampleScripts.py', somescript_arg1, somescript_val1,...]).

Popen(['nohup', 'python', 'exampleScripts.py'],
                 stdout=open('null1', 'w'),
                 stderr=open('logfile.log', 'a'),
                 start_new_session=True )
                 
Popen(['nohup', 'python', 'exampleScripts.py'],
                 stdout=open('null2', 'w'),
                 stderr=open('logfile.log', 'a'),
                 start_new_session=True )

Popen(['nohup', 'python', 'exampleScripts.py'],
                 stdout=open('null3', 'w'),
                 stderr=open('logfile.log', 'a'),
                 start_new_session=True )      

Output: the start and end times in each script overlap showing that the 2nd one started before the first one ended

(ds_tensorflow) C:\DataScience\SampleNotebooks\Threading>python RunScripts.py

(ds_tensorflow) C:\DataScience\SampleNotebooks\Threading>cat null*
2020-07-13 15:46:21.251606
List processing complete.
2020-07-13 15:46:29.130219
2020-07-13 15:46:22.501599
List processing complete.
2020-07-13 15:46:31.227954
2020-07-13 15:46:23.758498
List processing complete.
2020-07-13 15:46:32.431079

You can also use the same idea with functions, if you want to keep the code in once place. This example will run two different functions two times, in parallel.

Function example:

...
import threading
...

def some_function()
    # code

def other_function()
    # code

if __name__ == "__main__":    
    jobs = []
    #same function run multiple times
    threads = 2
    for i in range(0, threads):
        out_list = list()
        thread1 = threading.Thread(target=some_function(size, i, out_list))
        jobs.append(thread1)
        thread2 = threading.Thread(target=other_function(size, i, out_list))
        jobs.append(thread2)
        
    # Start the threads (i.e. calculate the random number lists)
    for j in jobs:
        j.start()

    # Ensure all of the threads have finished
    for j in jobs:
        j.join()

    # continue processing

Answer 2

You can use threading or multiprocessing 'for running the python scripts in parallel.

Threading:https://www.tutorialspoint.com/python/python_multithreading.htm

Multiprocessing: https://www.tutorialspoint.com/multiprocessing-in-python#:~:text=The%20multiprocessing%20package%20supports%20spawning,is%20similar%20to%20threading%20module .

Hope this helps you

Answer 3

You could do something along these lines (for the sake of example, I have assumed your scripts accept two arguments, arg1 and arg2 . This needs to be modified according to your specific needs.):

1. If you have "main" functions in server.py and servergui.py:

import threading
from server import server_main
from servergui import server_gui_main

thread_list = []
thread_list.append(
    threading.Thread(target=server_main, args=(arg1, arg2))
)
thread_list.append(
    threading.Thread(target=server_gui_main, args=(arg1, arg2))
)

for thread in thread_list:
    thread.start()

for thread in thread_list:
    thread.join()

The above will start the two main functions in separate threads.

If you want separate parallel processes use multithreading :

import multiprocessing

process_list = []
process_list.append(
    multiprocessing.Process(target=server_main, args=(arg1, arg2))
)
process_list.append(
    multiprocessing.Process(target=server_gui_main, args=(arg1, arg2))
)

for process in process_list:
    process.start()

for process in process_list:
    process.join()

As you see, the differences in the API of multiprocessing and threading are small. However, your performance may suffer from threading if you are performing CPU-bound tasks. This is because the Python GIL forces Python to run only one thread at any given moment. Thus, if you have CPU-intensive tasks you should use multiprocessing as this creates separate processes which do indeed run in parallel.

2. If you want to start server.py and servergui.py as if you start them from the command line:

import subprocess

subprocess.run(
    ['python', 'server.py', 'arg1', 'arg2'],
    shell=True
)

subprocess.run(
    ['python', 'servergui.py', 'arg1', 'arg2'],
    shell=True
)