迭代速度比for循环快

Question

I have a for loop for a new project, but this form of the code is too slow. I am trying to find the fastest way to resolve it. Maybe as a vector?

I have tried def approach, but it did not execute properly.

%%time
for x in df2.index:   
    if x > 0: 
        if (
            (df2.loc[x,'DEF RANK'] == df2.loc[x,'OFF RANK']) 
            & (df2.loc[x,'W']=='nan')
            & (pd.isnull(df2.loc[(x-1),'Event2']) == False)
            & ((df2.loc[(x-1),'Event2'] == 'nan') == False)
        ):
            df2.loc[x,'W'] = df2.loc[(x-1),'W']
        else: # if the above isn't true - pass
            pass
    else: 
        pass

Wall time: 6.76 ms

Answer 1

In Python, the bitwise & operator does not short circuit. Meaning that all of your comparisons are happening every time, regardless of what the statements before them evaluated to. Try this for a demonstration:

bool(print('a')) & bool(print('b')) & bool(print('c'))

Outputs:

a
b
c

Compare that to the and logical operator, which does short circuit the chain of comparisons:

bool(print('a')) and bool(print('b')) and bool(print('c'))

Outputs:

a

Try subbing out your & s with and s to limit the number of comparisons being done.

Once you've done that, you can try fiddling with which comparisons should come first. You'll want to order them by either/both which ones are most likely to evaluate to False and which ones are most performant.

Answer 2

The first thing you want to learn when dealing with pandas dataframe is to view the data as a whole, and try to deal with it as a whole. So let's dig in your code and see how we can improve it.

for x in df2.index:   
    # the next if means you essentially want to look at 
    # df.index > 0 only
    if x > 0: 
        # this if clause chains several 'and' conditions:
        if (
            # this is df2['DEF RANK'].eq(df2['OFF RANK'])
            (df2.loc[x,'DEF RANK'] == df2.loc[x,'OFF RANK']) 

            # this is df2['W'].eq('nan')
            & (df2.loc[x,'W']=='nan')

            # this is df2.loc[df.index - 1, 'Event2'].notnull()
            & (pd.isnull(df2.loc[(x-1),'Event2']) == False)

            # this is df2.loc[df.index - 1, 'Event2'].ne('nan')
            & ((df2.loc[(x-1),'Event2'] == 'nan') == False)
        ):
            # here you copy some position to other position
            df2.loc[x,'W'] = df2.loc[(x-1),'W']

        # if you don't do anything after else, why not delete it?
        else: # if the above isn't true - pass
            pass
    else: 
        pass

So with all the comments, how do we write codes that runs faster. Plainly from your code:

idx_gt0 = (df.index > 0)
rank_filters = df2['DEF RANK'].eq(df2['OFF RANK'])
w_isnan = df2['W'].eq('nan')

# the next two conditions are more challenging:
# we start with looking at the series  df.loc[df.index-1, 'Event2']
df2['event2_shifted'] = df2.loc[df2.index-1, 'Event2'].values

event2_notnull = df2['event2_shifted'].notnull()
event2_notnan = df2['event2_shifted'].ne('nan')

# now we can merge all filters:
filters = (idx_gt0 & rank_filters 
           & w_isnan & event2_notnull & event2_notnan
          ) 

 # last assign:
 df2.loc[filters, 'W'] = df2.loc[df2.index - 1, 'W']

Of course, this is literally translated from your code. But as you said, your code does not work properly. So it would help if you give a sample input data and its expected output.

Answer 3

Unfortunately, there is no good faster way.

Here is one recommendation: change all the & s in your code to and s.

It may not help that much, though.

You need to use a different programming language, like C, C++, or Java, as compiled code can run faster than interpreted code. If you are willing to get really frustrated, you can try assembly language, but I'm not sure that it is worth it for just a for loop.