我该如何优化这个 python 循环？

Question

I am running this code on a large csv file (1.5 million rows). Is there a way to optimise?

df is a pandas dataframe. I take a row and want to know what happens 1st in the 1000 folowing rows:

I find my value + 0.0004 or i find my value - 0.0004

result = []
for row in range(len(df)-1000):
    start = df.get_value(row,'A')
    win = start + 0.0004
    lose = start - 0.0004
    for n in range(1000):
        ref = df.get_value(row + n,'B')
        if ref > win:
            result.append(1)
            break
        elif ref <= lose:
            result.append(-1)
            break
        elif n==999 :
            result.append(0)

the dataframe is like:

         timestamp           A         B
0   20190401 00:00:00.127  1.12230  1.12236
1   20190401 00:00:00.395  1.12230  1.12237
2   20190401 00:00:00.533  1.12229  1.12234
3   20190401 00:00:00.631  1.12228  1.12233
4   20190401 00:00:01.019  1.12230  1.12234
5   20190401 00:00:01.169  1.12231  1.12236 

the result is: result[0,0,1,0,0,1,-1,1,…]

this is working but takes a long time to process on such large files.

Answer 1

To generate values for the "first outlier", define the following function:

def firstOutlier(row, dltRow = 4, dltVal = 0.1):
    ''' Find the value for the first "outlier". Parameters:
    row    - the current row
    dltRow - number of rows to check, starting from the current
    dltVal - delta in value of "B", compared to "A" in the current row
    '''
    rowInd = row.name                        # Index of the current row
    df2 = df.iloc[rowInd : rowInd + dltRow]  # "dltRow" rows from the current
    outliers = df2[abs(df2.B - row.A) >= dlt]
    if outliers.index.size == 0:  # No outliers within the range of rows
        return 0
    return int(np.sign(outliers.iloc[0].B - row.A))

Then apply it to each row:

df.apply(firstOutlier, axis=1)

This function relies on the fact that the DataFrame has the index consisting of consecutive numbers, starting from 0, so that having ind - the index of any row we can access it calling df.iloc[ind] and a slice of n rows, starting from this row, calling df.iloc[ind: ind + n] .

For my test, I set the default values of parameters to:

• dltRow = 4 - look at 4 rows, starting from the current one,
• dltVal = 0.1 - look for rows with B column "distant by" 0.1 or more from A in the current row.

My test DataFrame was:

      A     B
0  1.00  1.00
1  0.99  1.00
2  1.00  0.80
3  1.00  1.05
4  1.00  1.20
5  1.00  1.00
6  1.00  0.80
7  1.00  1.00
8  1.00  1.00

The result (for my data and default values of parameters) was:

0   -1
1   -1
2   -1
3    1
4    1
5   -1
6   -1
7    0
8    0
dtype: int64

For your needs, change default values of params to 1000 and 0.0004 respectively.

Answer 2

The idea is to loop through A and B while maintaining a sorted list of A values. Then, for each B , find the highest A that loses and the lowest A that wins. Since it's a sorted list it's O(log(n)) to search. Only those A 's that have index in the last 1000 are used for setting the result vector. After that the A 's that are no longer waiting for a B are removed from this sorted list to keep it small.

import numpy as np
import bisect
import time

N = 10
M = 3
#N=int(1e6)
#M=int(1e3)
thresh = 0.4

A = np.random.rand(N)
B = np.random.rand(N)
result = np.zeros(N)

l = []

t_start = time.time()

for i in range(N):
    a = (A[i],i)
    bisect.insort(l,a)
    b = B[i]
    firstLoseInd = bisect.bisect_left(l,(b+thresh,-1))
    lastWinInd = bisect.bisect_right(l,(b-thresh,-1))
    for j in range(lastWinInd):
        curInd = l[j][1]
        if curInd > i-M:
            result[curInd] = 1
    for j in range(firstLoseInd,len(l)):
        curInd = l[j][1]
        if curInd > i-M:
            result[curInd] = -1
    del l[firstLoseInd:]
    del l[:lastWinInd]

t_done = time.time()

print(A)
print(B)
print(result)
print(t_done - t_start)

This is a sample output:

[ 0.22643589  0.96092354  0.30098532  0.15569044  0.88474775  0.25458535
  0.78248271  0.07530432  0.3460113   0.0785128 ]
[ 0.83610433  0.33384085  0.51055061  0.54209458  0.13556121  0.61257179
  0.51273686  0.54850825  0.24302884  0.68037965]
[ 1. -1.  0.  1. -1.  0. -1.  1.  0.  1.]

For N = int(1e6) and M = int(1e3) it took about 3.4 seconds on my computer.