Python, using threading with multiprocessing

Question

Can someone explain why threading don't work in multiprocessing.Process.

I've attached some example to explain my problem.

I have a process that executed every second and write to file. When I run it from shell, it works as expected.

stat_collect.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from threading import Timer
from os import path
from datetime import datetime

STAT_DATETIME_FMT = '%Y-%m-%d %H:%M:%S'


def collect_statistics():
    my_file = 'test.file'
    if not path.exists(my_file):
        with open(my_file, 'w') as fp:
            fp.write(datetime.now().strftime(STAT_DATETIME_FMT) + '\n')
    else:
        with open(my_file, 'a') as fp:
            fp.write(datetime.now().strftime(STAT_DATETIME_FMT) + '\n')

    Timer(1, collect_statistics).start()


if __name__ == '__main__':
    collect_statistics()

When I try to run it from other script (to work in background):

#!/usr/bin/env python

from multiprocessing import Process
from stat_collect import collect_statistics  # logger sc

if __name__ == '__main__':
    # This don't work
    p = Process(target=collect_statistics)
    p.start()

    while True:
        pass

Method collect_statistics executed only once, but if I use Thread(target=collect_statistics).start() it works as if I run it from shell. Why this is happen?

Answer 1

Here is what is going on:

1. You start your process
2. collect_statistics runs
3. Timer is started
4. now the function called in the process( collect_statistics ) is finished, so the process quit, killing the timer in the same time.

Here is how to fix it :

stat_collect.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from threading import Timer
from os import path
from datetime import datetime
import time

STAT_DATETIME_FMT = '%Y-%m-%d %H:%M:%S'


def collect_statistics():
    while True:
        my_file = 'test.file'
        if not path.exists(my_file):
            with open(my_file, 'w') as fp:
                fp.write(datetime.now().strftime(STAT_DATETIME_FMT) + '\n')
        else:
            with open(my_file, 'a') as fp:
                fp.write(datetime.now().strftime(STAT_DATETIME_FMT) + '\n')

        time.sleep(1)


if __name__ == '__main__':
    collect_statistics()

And for the calling script :

#!/usr/bin/env python

from multiprocessing import Process
from stat_collect import collect_statistics  # logger sc

if __name__ == '__main__':
    # This don't work
    p = Process(target=collect_statistics)
    p.start()
    p.join() # wait until process is over, e.g forever

p.join() is just here to replace you infinite while loop, which is taking a lot of ressource for nothing.