Delayed start of tasks with Python multiprocessing

Question

I've been studying Python multiprocessing capabilities recently and encountered an issue with the following code

import syslog
from multiprocessing import Pool
def launcher(i):
    time.sleep(i)
    syslog.openlog( 'test', 0, syslog.LOG_LOCAL4 )
    syslog.syslog( '{} {}'.format(i,datetime.now()))

if __name__ == '__main__':
    pool=Pool(8)
    pool.map(launcher,range(1,3000))
    pool.close()
    pool.join()

The idea behind it is simple: I need to get a stead flow of messages in my syslog (one message per second), but I want to spawn it across 8 worker-processes with multiprocessing Pool.

In my syslog (which is local /var/log/syslog on my Ubuntu) I've got the following

Sep 17 17:17:57 test: 1 2015-09-17 17:17:57.225699
Sep 17 17:17:58 test: 2 2015-09-17 17:17:58.226957
Sep 17 17:18:00 test: 3 2015-09-17 17:18:00.229196
Sep 17 17:18:03 test: 4 2015-09-17 17:18:03.232390
Sep 17 17:18:07 test: 5 2015-09-17 17:18:07.236587
Sep 17 17:18:12 test: 6 2015-09-17 17:18:12.241737
Sep 17 17:18:18 test: 7 2015-09-17 17:18:18.247926
Sep 17 17:18:25 test: 8 2015-09-17 17:18:25.255169
Sep 17 17:18:29 test: 9 2015-09-17 17:18:29.258229
Sep 17 17:18:33 test: 10 2015-09-17 17:18:33.263454
Sep 17 17:18:42 test: 64 2015-09-17 17:18:42.272675
Sep 17 17:18:52 test: 33 2015-09-17 17:18:52.283012
Sep 17 17:19:01 test: 11 2015-09-17 17:19:01.290070
Sep 17 17:19:02 test: 12 2015-09-17 17:19:02.259826

Firstly, the flow is not uniform and, secondly, out of order.

What can be the cause if that?

Why linux process scheduler work like that with Python multiprocessing?

Is there any way to solve my task with multiprocessing at all?

Answer 1

Even if the OS would do real-time scheduling of just your program, you wouldn't get uniform messages after each second:

1. take 8 kitchen timers, and a stack of post-its.
2. Number 2999 post-its 1 to 2999.
3. take a timer and a post-it and set the time to the number on the post-it and set it aside.
4. repeat 3 until you run out of timers
5. (if you are blazingly fast, sub-second speed) you have 8 timers counting down from [1, 2, 3 ,4, 5, 6, 7, 8]
6. wait a second
7. now your first timer should go off, repeat step 3.
8. you now have 8 timers running

The sequence will be

[9, 1, 2, 3, 4, 5, 6, 7]
[8, 10, 1, 2, 3, 4, 5, 6]
[7, 9, 11, 1, 2, 3, 4, 5]
[6, 8, 10, 12, 1, 2, 3, 4]
[5, 7, 9, 11, 13, 1, 2, 3]
[4, 6, 8, 10, 12, 14, 1, 2]
[3, 5, 7, 9, 11, 13, 15, 1]
[2, 4, 6, 8, 10, 12, 14, 16]
#Notice that for the next timer to go off, you have to wait 2 seconds, not 1!
[17, 2, 4, 6, 8, 10, 12, 14]
[15, 18, 2, 4, 6, 8, 10, 12]
...
[3, 6, 9, 12, 15, 18, 21, 24]
#3 seconds to wait, not 1!
[25, 3, 6, 9, 12, 15, 18, 21]
...
[4, 8, 12, 16, 20, 24, 28, 32]

EDIT:

My guess to why you experience waits of 1, 2 ,3 so fast is probably because of your 8 workers simultaneously trying to do IO to the same syslog. I think there is some blocking going there.