I am having a hard time getting my code to properly run in parallel. What I need is the following:
Because of the nature of the codes and of the data that I need to store, after trying processes pools, I've decided my best way to go is to use a Queue. I have tried two different codes. For this example, disk_codes
are just numbers, but in my actual code they are individual instances of another code to solve equations of motion in a disk.
Code 1:
import multiprocessing
try:
import queue
except:
import Queue as queue
def evolve_single_disk(queue, dt):
print "Empty queue? ", queue.empty()
code = queue.get()
print "Evolving disk {0} in {1}".format(code, multiprocessing.current_process().name)
queue.task_done()
if __name__ == '__main__':
ncores = 4
code_queue = queue.Queue()
disk_codes = range(ncores)
for disk in disk_codes:
code_queue.put(disk)
dt = 1
t_end = 10
t = 0
# Evolve!
while t < t_end:
print "t=", t
processes = multiprocessing.Process(target=evolve_single_disk, args=(code_queue, dt, ))
processes.start()
processes.join()
disk_codes = code_queue.get()
print "disk codes: ", disk_codes
t += dt
This results in:
t= 0
Empty queue? False
Evolving disk 0 in Process-1
disk codes: 0
t= 1
Empty queue? False
Evolving disk 1 in Process-2
disk codes: 1
t= 2
Empty queue? False
Evolving disk 2 in Process-3
disk codes: 2
t= 3
Empty queue? False
Evolving disk 3 in Process-4
disk codes: 3
t= 4
Empty queue? True
So, in every time step, one of the disks is "evolved". This is not what I want, since I want all four disks to be evolved in parallel in the same time step.
Then I tried this:
Code 2:
import multiprocessing
try:
import queue
except:
import Queue as queue
def evolve_single_disk(queue, dt):
print "Empty queue? ", queue.empty()
code = queue.get()
print "Evolving disk {0} in {1}".format(code, multiprocessing.current_process().name)
queue.task_done()
if __name__ == '__main__':
ncores = 4
code_queue = queue.Queue()
disk_codes = range(ncores)
for disk in disk_codes:
code_queue.put(disk)
dt = 1
t_end = 10
t = 0
# Evolve!
while t < t_end:
print ""
print "t=", t
processes = [multiprocessing.Process(target=evolve_single_disk, args=(code_queue, dt, )) for x in range(ncores)]
for p in processes:
p.start()
p.join()
disk_codes = [code_queue.get() for p in processes]
print "disk codes: ", disk_codes
t += dt
Which results in:
t= 0
Empty queue? False
Evolving disk 0 in Process-1
Empty queue? False
Evolving disk 0 in Process-2
Empty queue? False
Evolving disk 0 in Process-3
Empty queue? False
Evolving disk 0 in Process-4
disk codes: [0, 1, 2, 3]
t= 1
Empty queue? True
...and then the code just hangs. So here I am starting 4 processes in every time step, but each process is receiving the exact same disk.
How can I do this properly, so that on each time step there are 4 processes and each one of them evolves 1 single disk? I have read the documentation and many tutorials and SO questions/answers, but I am still very confused.
Edit:
I tried using multiprocessing
queues, but then I get a TypeError when I try to put disk codes into the queue. Sadly the disk codes are also not pickleable. Traceback when using multiprocessing
queue with disk codes:
t= 0
<multiprocessing.queues.Queue object at 0x7fee4a2d3950>
Traceback (most recent call last):
File "/home/fran/anaconda2/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
send(obj)
TypeError: expected string or Unicode object, NoneType found
Traceback (most recent call last):
File "/home/fran/anaconda2/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
send(obj)
TypeError: expected string or Unicode object, NoneType found
Traceback (most recent call last):
File "/home/fran/anaconda2/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
send(obj)
TypeError: expected string or Unicode object, NoneType found
Traceback (most recent call last):
File "/home/fran/anaconda2/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
send(obj)
TypeError: expected string or Unicode object, NoneType found
Empty queue? True
Move this inside the while loop:
for disk in disk_codes:
code_queue.put(disk)
Here is the complete code:
import multiprocessing
def evolve_single_disk(queue, result, dt):
print "Empty queue? ", queue.empty()
code = queue.get()
print "Evolving disk {0} in {1}".format(code, multiprocessing.current_process().name)
result.put(code)
if __name__ == '__main__':
ncores = 4
code_queue = multiprocessing.Queue()
result_queue = multiprocessing.Queue()
disk_codes = range(ncores)
dt = 1
t_end = 10
t = 0
# Evolve!
while t < t_end:
for disk in disk_codes:
code_queue.put(disk)
print ""
print "t=", t
process_list = list()
for x in range(ncores):
process = multiprocessing.Process(target=evolve_single_disk, args=(code_queue, result_queue, dt))
process_list.append(process)
for p in process_list:
p.start()
p.join()
disk_codes = [result_queue.get() for p in process_list]
print "disk codes: ", disk_codes
t += dt
Output
t= 0
Empty queue? False
Evolving disk 0 in Process-1
Empty queue? False
Evolving disk 1 in Process-2
Empty queue? False
Evolving disk 2 in Process-3
Empty queue? False
Evolving disk 3 in Process-4
disk codes: [0, 1, 2, 3]
t= 1
Empty queue? False
Evolving disk 0 in Process-5
Empty queue? False
Evolving disk 1 in Process-6
Empty queue? False
Evolving disk 2 in Process-7
Empty queue? False
Evolving disk 3 in Process-8
disk codes: [0, 1, 2, 3]
...
t= 9
Empty queue? False
Evolving disk 0 in Process-37
Empty queue? False
Evolving disk 1 in Process-38
Empty queue? False
Evolving disk 2 in Process-39
Empty queue? False
Evolving disk 3 in Process-40
disk codes: [0, 1, 2, 3]
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.