Map each element of array with its own method using multiprocessing

Question

I have an array of class objects called "rules".

self.rules looks like this

[<creative_task.rules.rules.Rule object at 0x7fde7a33f518>, <creative_task.rules.rules.Rule object at 0x7fde7a33f830>, <creative_task.rules.rules.Rule object at 0x7fde7a33f888>, <creative_task.rules.rules.Rule object at 0x7fde7a33f8e0>]

This Rule class has method self.check that validates data that stored in it.

There are a lot of rules to be validated: requested value is MORE, LESS, EQUALS, BETWEEN, POSITIVE, NEGATIVE respecting to the value from database. Each such rule on self.check function call makes a call to Clickhouse database to validate itself.

Basically, when I need to validate array of rules I call check_all function that validates all the Rules like this rules=[r.check(asset, context) for r in self.rules] :

    def check_all(self, asset: dict, context: dict = None) -> RuleGroupResult:
        context = {} if context is None else context
        return RuleGroupResult(
            identifier=self.identifier,
            rules=[r.check(asset, context) for r in self.rules],
            groups=[g.check(asset, context) for g in self.groups],
            logical_operator=self.logical_operator,
        )

These rules are not depending on each other.

Now I want to make this kind of validation to be a parallel process. So I need to call Rule.check method for each Rule in array, but this calls should occur simultaneously

Answer 1

This can be done relatively easily with Pool.map , along with operator.methodcaller , since your arguments are all the same, and the method has the same name on each object:

import multiprocessing as mp  # Top of file
from operator import methodcaller

def check_all(self, asset: dict, context: dict = None) -> RuleGroupResult:
    context = {} if context is None else context
    with mp.Pool() as pool:  # Create pool
        return RuleGroupResult(
            identifier=self.identifier,
            rules=pool.map(methodcaller('check', asset, context), self.rules),  # Map in parallel
            groups=pool.map(methodcaller('check', asset, context), self.groups),
            logical_operator=self.logical_operator,
        )