multi threaded processes in python using queue to write to file checking if work had been done

Question

from multiprocessing.dummy import Pool as ThreadPool
import multiprocessing as mp



def func(a):

    pthData = "C:/temp/temp.txt"
    with open(pthData, 'r') as file:
        done = file.read().splitlines()

    if a in done:
        return 'done'

    q.put(a)
    return a

def listener(q):

    pthData = "C:/temp/temp.txt"
    m = q.get()
    with open(pthData, 'a') as the_file:
        the_file.write( m + '\n')
        #he_file.write(str(m) + '\n')


a =  ['a', 'b', 'c', 'd', 'a', 'b']


# Make the Pool of workers
pool = ThreadPool(4)

#must use Manager queue here, or will not work
manager = mp.Manager()
q = manager.Queue()    

#put listener to work first
watcher = pool.apply_async(listener, (q,))

pool.starmap(func, a, q)
## TypeError: '<=' not supported between instances of 'AutoProxy[Queue]' and 'int'

pool.starmap(func, a)
## Runs but only writes 'a' to temp file

pool.starmap(func, (a, q))
## func() takes 1 positional argument but 6 were given

pool.apply_async(func, (a, q))
## freezes on pool.join

# Close the pool and wait for the work to finish
pool.close()
pool.join()

Why is the apply_async freezing on the pool.join()? I tried putting it into a if name == ' main ' but it had the same result.

How do I properly call func passing 1 argument (a) and the queue (q)?

Answer 1

How do I properly call func passing 1 argument (a) and the queue (q)?

This at-least does not freeze :

• Ensure temp.txt exists before execution.
• Add a q parameter to func .

      def func(a,q):
          print(f'func({a})')
          ...

• Use apply_async in a list comprehension.

    if __name__ == '__main__':

        # Make the Pool of workers
        with ThreadPool(4) as pool:
            q = queue.Queue()
            #put listener to work first
            watcher = pool.apply_async(listener, (q,))
            results = [pool.apply_async(func, (item, q)) for item in a]
            # just check stuff
            for result in results:
                result.wait()
                print(result, result.successful(),result.get())
            pool.close()
            pool.join()

---

• You will need to work out some other problems like listener running once then stopping.
• Many other ways to do this, I used apply_async because it was one of the options in your question.
• I like using concurrent.futures myself.
• You may benefit from reading through the search results using variations of python threading producer consumer site:stackoverflow.com