如果没有逐行迭代数据帧，这需要很长时间，我如何检查许多行是否都满足条件？

Question

I want to do the following, but obviously I realise that this kind of iterative method is very slow with large DataFrames, what other solutions are there to this problem?:

for i in range(len(df)):
    for n in range(1001):
        if df["Close"][(i+n)] > df["MA"][i+n]:
            df["Strategy 1"][i] = "Buy"

What I would expect the code above to do is:

Sub in n from 0 to 1,000 into line 3, with an i of 0 , and then if the condition in line 3 held for each n in the range of 0 to 1,000 then it would go on and carry out the operation in line 4.

After this it would take i of 1 and then sub in n from 0 to 1,000 into line 3, and if the condition held for all n in that range then it would carry out line 4.

After this it would take i of 2 and then sub in n from 0 to 1,000 into line 3, and if the condition held for all n in that range then it would carry out line 4.

After this it would take i of 3 and then sub in n from 0 to 1,000 into line 3, and if the condition held for all n in that range then it would carry out line 4.

... ...

After this it would take i of len(df) and then sub in n from 0 to 1,000 into line 3, and if the condition held for all n in that range then it would carry out line 4.

Regardless of if the code presented above does what i'd expect or not, is there a much faster way to compute this for very large multi Gigabyte DataFrames?

Answer 1

Using the .apply function would be faster. For a general example...

import pandas as pd

# only required to create the test dataframe in this example
import numpy as np

# create a dataframe for testing using the numpy import above
df = pd.DataFrame(np.random.randint(100,size=(10, )),columns=['A'])

# create a new column based on column 'A' but moving the column 'across and up'
df['NextRow'] = df['A'].shift(-1)

# create a function to do something, anything, and return that thing
def doMyThingINeedToDo(num, numNext):
#     'num' is going to be the value of whatever is in column 'A' per row 
#     as the .apply function runs below and 'numNext' is plus one.
    if num >= 50 and numNext >= 75:
        return 'Yes'
    else:
        return '...No...'

# create a new column called 'NewColumnName' based on the existing column 'A' and apply the
# function above, whatever it does, to the frame per row.
df['NewColumnName'] = df.apply(lambda row : doMyThingINeedToDo(row['A'], row['NextRow']), axis = 1)

# output the frame and notice the new column
print(df)

Outputs:

    A  NextRow NewColumnName
0  67     84.0           Yes
1  84     33.0      ...No...
2  33     59.0      ...No...
3  59     85.0           Yes
4  85     39.0      ...No...
5  39     81.0      ...No...
6  81     76.0           Yes
7  76     83.0           Yes
8  83     60.0      ...No...
9  60      NaN      ...No...

The main point is that you can separate what exactly you want to do per row and contain it in a function (that can be tweaked and updated as required) and just call that function for all rows on a frame when required.

Answer 2

You can accomplish what you are attempting with only your close data. Calculating the MA and 1000 conditions on the fly via vectorization. Maybe try this:

import numpy as np

ma_window = 1000 
n = 1000 

df['Strategy 1'] = \
    np.where( \
        (df['close'] > df['close'].rolling(window=ma_window).mean()).rolling(window=n).mean() == 1, \
             'buy','')
         

Play around with this and see if it might work for you.

Answer 3

First, let me state how I understand your rule. As near as I can tell you are trying to get a value of "Buy" in the "Strategy 1" column of the df only if there are 1000 consecutive cases where MA was greater than the Close preceding that time. I think you can get that done simply by using a rolling sum on the comparison:

import pandas as pd
import numpy as np

# build some repeatable sample data
np.random.seed(1)
df = pd.DataFrame({'close': np.cumsum(np.random.randn(10000))})
df['MA'] = df['close'].rolling(1000).mean()

# Apply strategy
npoints = 1000

df['Strategy 1'] = float('nan')
buypoints = (df['MA'] > df['close']).rolling(npoints).sum() == npoints
df.loc[buypoints, "Strategy 1"] = "Buy"

# just for visualisation show where the Buys would be
df['Buypoints'] = buypoints*10
df.plot()

This comes out like this (with the same seed it should look the same on your machine too)

Answer 4

Iteration is a last resort with Pandas.

The solution you are looking for is coming from numpy:

import numpy as np
df["Strategy 1"] = np.where(df["Close"] > df["MA"], "Buy", df["Strategy 1"])