concurrent.futures ThreadPoolExecutor fails to wait?

Question

I am trying to get a script performance improved, using ThreadPoolExecutor from concurrent.futures. I am launching some external python scripts via Popen and encapsulating them as a future objects, but these objects enter the callback function as finished, but I can see them running on my machine (they run for quite some minutes). The code looks like this:

with futures.ThreadPoolExecutor(max_workers=4) as executor: 
        p1 = executor.submit(subprocess.Popen([myotherPythonscript1], stdout = subprocess.PIPE))
        p1.add_done_callback(process_result)
        p2 = executor.submit(subprocess.Popen([myotherPythonscript2], stdout = subprocess.PIPE))
        p2.add_done_callback(process_result)

def process_result( future ):
        logger.info( "Seeding process finished...")

I also tried different approaches with running() and wait() future functions, but with the same results. Future objects are marked as already done, but in fact they are still running. Am I missing something?

Thanks,

Answer 1

you can't just pass the result of Popen to your executor, you have to pass a callable.

>>> from concurrent.futures import *
>>> from subprocess import *
>>> executor = ThreadPoolExecutor(4)
>>> future = executor.submit(Popen(['echo', 'test'], stdout=PIPE))
>>> future.exception()
TypeError("'Popen' object is not callable",)

this on the other hand works:

from concurrent.futures import *
from subprocess import *

def make_call():
    process = Popen(['echo', 'test'], stdout=PIPE)
    return process.communicate()[0]

executor = ThreadPoolExecutor(4)
future = executor.submit(make_call)
print(future.result())