is np.array == num comparison very slow? Can multiprocessing be used to accelerate it?

Question

Is == comparison between a large np.array with a single number very low in python? I used line_profiler to locate the bottleneck in my code. The bottleneck is just a simple comparison between a 1d np.array with a constant number. It accounts for 80% of the total runtime. Did I do anything wrong causing it to be so slow? Is there any way to accelerate it?

I tried to use multiprocessing, however, in the test code (snippet 2), using multiprocessing is slower than running in sequence and using map directly. Could anyone explain this phenomenon?

Any comments or suggestions are sincerely appreciated.

Snippet 1:

Line # Hits Time Per Hit %Time Line Contents

38 12635 305767927.0 24200.1 80.0 res = map(logicalEqual,assembly)

def logicalEqual(x):
         return F[:,-1] == x

assembly = [1,2,3,4,5,7,8,9,...,25]

F is an int typed (281900, 6) np.array

Snippet 2:

import numpy as np
from multiprocessing import Pool
import time

y=np.random.randint(2, 20, size=10000000)

def logicalEqual(x):
    return y == x

p=Pool()
start = time.time()
res0=p.map(logicalEqual, [1,2,3,4,5,7,8,9,10,11,12,13,14,15])
# p.close()
# p.join()
runtime = time.time()-start
print(f'runtime using multiprocessing.Pool is {runtime}')

res1 = []
start = time.time()
for x in [1,2,3,4,5,7,8,9,10,11,12,13,14,15]:
    res1.append(logicalEqual(x))
runtime = time.time()-start
print(f'sequential runtime is {runtime}')


start = time.time()
res2=list(map(logicalEqual,[1,2,3,4,5,7,8,9,10,11,12,13,14,15]))
runtime = time.time()-start
print(f'runtime is {runtime}')

runtime using multiprocessing.Pool is 0.3612203598022461
sequential runtime is 0.17401981353759766
runtime is  0.19697237014770508

Answer 1

Array comparison is fast, since it is done in C code, not Python.

x = np.random.rand(1000000)
y = 4.5
test = 0.55
%timeit x == test
386 µs ± 4.68 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit y == test
33.2 ns ± 0.121 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

So, comparing one Python float to another takes 33*10^-9 s, while comparing 1E6 numpy floats takes only 386 µs / 33 ns ~= 11700 times longer, despite comparing 1000000 more values. The same is true for ints (377 µs vs 34 ns). But as Dunes mentioned in a comment, comparing a lot of values takes a lot of cycles. Nothing you can do about that.