在各种情况下循环迭代的最快方法

Question

After extensive profiling of our source code base, we found out that all performance issues are caused while we are looping over some huge lists.

The code passages that are causing the issues could be identified as follows:

#ISSUE 1
myList = [i for j, i in enumerate(myList) if j not in anotherList]

#ISSUE 2
TargetIndex = next((myList.index(n) for n in myList if n > someBoundary), len(myList))

#ISSUE 3
def myFunction():
    for i in myList:
        if abs(i) > someLimit:
            return 0
    return 1

#ISSUE 4
for n,i in enumerate(myList):
    if abs(i) < someLimit:
        myList[n] = 0

I am quite sure that some numpy experts could write down four one-liners that would lead to a great performance boost of our application. But perhaps there may even be a better way for those looping operations than numpy which I am not aware of.

Any suggestions on the topic are highly appreciated.

Answer 1

First issue: make a lookup in a set instead of a list

anotherSet = set(anotherList)
myList = [i for j, i in enumerate(myList) if j not in anotherSet]

Second issue: why computing index of n when you already iterating on the list? Use enumerate

TargetIndex = next((i for i,n in enumerate(myList) if n > someBoundary), len(myList))

For issues 3 & 4, there's not much you can do but precompute the list of absolute values so you don't perform it twice on the same list.

abs_vals = [abs(n) for n in myList]

so for instance the 4th snippet becomes:

for index,av in enumerate(abs_vals):
    if av < someLimit:
        myList[index] = 0

Answer 2

As a warning, you'll have to change a lot more than these if you want to keep your data as numpy arrays, but this is how you fix the issues you have.

import numpy as np

myArr=np.array(myList)

#1
myArr = myArr[np.in1d(np.arange(myArr.size), anotherList, invert = True)]

#2
TargetIndex = next(np.nonzero(myArr > someBoundary)[0].flat, myArr.size)

#3
def myFunction():
    return (np.abs(myArr) <= someLimit).astype(int)

#4
np.where(np.abs(myArr) < someLimit, 0, myArr)