Multiprocessing List Comprehension/Filter

Question

I'm trying to multiprocesses a list-comprehension:

l = [i for i in range(10) if fun(i)]

If I was keeping all the values in the list, I could simply call pool.map , but this loop might not keep all values.

How can I go about implementing this?

Answer 1

Use itertools.compress to filter the results of pool.map.

Here is the serial implementation using filter:

ages = [5, 12, 17, 18, 24, 32]

def fun(x):
  if x < 18:
    return False
  else:
    return True

adults = list(filter(fun, ages))

adults
  
Out[29]: [18, 24, 32]

The parallel implementation uses map and compress:

from itertools import compress
import multiprocessing
with multiprocessing.Pool(processes=2) as pool:
    adult_ages = list(compress(ages, pool.map(fun, ages)))
adult_ages

Out[27]: [18, 24, 32]