I want to process data faster with the JAVA base machine learning app: MOA, so I split my data into many small batches, so the input =[[a,b,c],[d,e,f],...]
. for each input I want to process them parallel. Can I use multiprocess's Pool
for that?
def batch_process(input):
...some pandas processing
// p = Pool(1000)
// map to subprocess, can I do that??
subprocess.run(...)
...
import threading
def batch_process(input):
...some pandas processing
// p = Pool(1000)
// map to subprocess, can I do that??
t1 = threading.Thread(target=subprocess, args=(10,))
# starting thread 1
t1.start()
...
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.