Python - how to read txt file every xxx time and add to multiprocess if new text added

Question

So basically I am trying to do something that I am not sure if it is possible to do nor not.

My idea is that I am using a multiprocessing which means it runs its separated process on the background for each "task" done. Similar to:

names.txt

Oskar Baldwin
Khalil Whittle
Jevon Burn
Paddy Wilkinson
Jocelyn Weiss
Ishaq Glenn
Zahraa Macfarlane
Marianna Roy
Humera Schultz
Luther Pugh

---

import json, time, sys, os, timeit, random, multiprocessing.dummy, re
import threading


def main(names):
    print(names)
    time.sleep(5)


if __name__ == '__main__':

    try:
        jobs = []
        for names in [line.rstrip('\n') for line in open('names.txt')]:
            p = multiprocessing.dummy.Process(target=main, args=(names,))
            jobs.append(p)
            p.start()

    except KeyboardInterrupt:
        print('Keyboard - Interrupted)
        sys.exit()

So the output of this will be that each process will take care of each of names for themselves so process 1 will use the first line in txt file and the 2nd process will use the second name and so on.

My question is how can I move on and be able to check the names.txt xxx time and check if there is a new name added then add it to a new process and if there is no new names then just continue the process and sleep again until another xxx time while the other process is running in the background ?

Answer 1

Quick and dirty way of doing it. I didn't research whether there's a way to query an item in the jobs list for its "args" attribute.

Beware that it does not correctly handle the case where a duplicate name is added after that process has finished executing. You'll have to come up with a way to remove finished processes from the used_names list.

jobs = []
used_names = []
for i in range(0, 10): # repeat as often as necessary
    try:
        for names in [line.rstrip('\n') for line in open('names.txt')]:
            if names not in used_names: # dont spawn new process if one already exists
                p = multiprocessing.dummy.Process(target=main, args=(names,))
                jobs.append(p)
                used_names.append(p)
                p.start()
        time.sleep(0.1) # arbitrary sleep

    except KeyboardInterrupt:
        print('Keyboard - Interrupted')
        sys.exit(0)