Python：同时运行多个功能

Question

I am using Pycharm for running my Python code. I have a function that combines multiple excel files in a folder and write to a csv file. I have 5 folders which I want to create 5 csv files for. Right now, I am running this sequentially, ie, one folder after a folder. This takes quite a long time. I have another option:running the same code, but using 5 different Pycharm projects. This works. But I am wondering if there is a way to run this function 5 times concurrently in 1 single project? My pseudo code is:

combine("folder1", "csvfile1")
combine("folder2", "csvfile2")
combine("folder3", "csvfile3")
combine("folder4", "csvfile4")
combine("folder5", "csvfile5")

Answer 1

Try using multiprocessing to map the function combine to separate cores and run it asynchronously. Here is an example -

#!pip install multiprocessing
import multiprocessing as mp

fo = ["folder1","folder2","folder3","folder4"]
fi = ["csvfile1","csvfile2","csvfile3","csvfile4"]

def combine(a,b):
    #YOUR CODE HERE
    print("Completed: ",a,'->',b)

pool = mp.Pool(processes=4) #Set this to the max number of cores you have
results = [pool.apply_async(combine, args=(x)) for x in zip(fo,fi)]

Completed:  folder2 -> csvfile2
Completed:  folder1 -> csvfile1
Completed:  folder3 -> csvfile3
Completed:  folder4 -> csvfile4

Each iteration of function is run asynchronously and in parallel for better utilization fo your resources.

---

As Furas pointed out, you can now use starmap (and its async version) since multiprocessing now supports it ( added in version 3.3 ). This helps mapping a tuple of params to a function directly instead of the iterating and applying over zip .

results = pool.starmap_async(combine, zip(fo,fi)) #async version
results = pool.starmap(combine, zip(fo,fi)) #sync version

If you have a return as part of your function and you want to retrieve those values, for the synchronous version you can simply get it from result but for the asynchronous version, you will need a result.get()

Answer 2

Akshay Sehgal already explain you how to use it. I add only few information.

You can write it shorter using map() (for single argument) or starmap() (for many arguments)

 results = pool.starmap_async(combine, zip(fo,fi))

if you use async version then you may need .get() to wait for all results

def combine(a,b):
    return b

results = pool.starmap_async(combine, zip(fo,fi))
print(results.get())

if you will use non-async version then you don't need `.get()

def combine(a,b):
    return b

results = pool.starmap(combine, zip(fo,fi))
print(results)

Processes may print() at the same time and it may mix messages from different processes so it it good to create one string before printing

print(f"Completed: {a} -> {b}")

---

import multiprocessing as mp

fo = ["folder1","folder2","folder3","folder4"]
fi = ["csvfile1","csvfile2","csvfile3","csvfile4"]

def combine(a,b):
    #YOUR CODE HERE
    print(f"Completed: {a} -> {b}") # it is good to create one string to display all as one string without strings from other processes
    return b

pool = mp.Pool(processes=4)

# async version

results = pool.starmap_async(combine, zip(fo,fi))
# it needs `.get()` because it is `async`
print(results.get())

# non-async version

results = pool.starmap(combine, zip(fo,fi))
# it doesn't needs `.get()` because it is not `async`
print(results)