简体   繁体   中英

Can I run multiprocess of subprocesses in python?

I want to process data faster with the JAVA base machine learning app: MOA, so I split my data into many small batches, so the input =[[a,b,c],[d,e,f],...] . for each input I want to process them parallel. Can I use multiprocess's Pool for that?

def batch_process(input):
    ...some pandas processing
    // p = Pool(1000)
    // map to subprocess, can I do that??
    subprocess.run(...)
...
import threading
def batch_process(input):
    ...some pandas processing
    // p = Pool(1000)
    // map to subprocess, can I do that??    
    t1 = threading.Thread(target=subprocess, args=(10,))
    # starting thread 1
     t1.start()
     ...

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM