提高代码性能并摆脱循环

Question

I have written a code that works by using standard for loop over a dataframe. I want to check if performance can be improved by making the code more pythonic by using groupby, apply, lamda, etc.

The code is designed to check for a particular data pattern on the price data of a stock. Close price of the stock is used along with its 20EMA. The panda dataframe is loaded from mysql db. It has the below structure.

df[['eod_dt','bull_bear','open','high','low','close','ema20']]

For the bullish case,

Condition1: the 'low' has to be above 'ema20' at least once.

Condition2: If this case is true then there needs to be at least 2 'bull' candles following the event. The first 'bull' candle should be followed by a candle whose 'high' is above the 'high' of the first 'bull' candle. The second bull candle will be my buy signal.

Currently I have done it like this.

df_bull=df[(df['bull_bear']=='bull') & (df['high']<df['EMA20']) & (df['eod_dt']>start_dt)] #start_dt is start of analysis period

for index,row in df_bull.iterrows():

    df_temp1=df[(df['index']>(row['index']-10)) & (df['index']<row['index'])] #10 day look back, the index field actually exists and acts as a proxy to actual trading days
    df_temp2=df_temp1[df_temp1['low']>df_temp1['EMA20']]

    if not df_temp2.empty: #condition1 satisfied
        df_temp1['high_1']=df_temp1['high'].shift(-1)
        df_temp2=df_temp1[(df_temp1['bull_bear']=='bull') & (df_temp1['high']<df_temp1['EMA20']) & (df_temp1['high_1']>df_temp1['high'])]

            if not df_temp2.empty and len(df_temp2)<4:
                #entry above signal bar high

The result of the 'better' code should be the same as above. But I will like to get the run time to a minimum. A shorter code will also be good.

Answer 1

To get rid of the for loop you can use pandas.DataFrame.apply

Apply allows you to perform operations on all the rows of the dataset.

It works as follow (simple example) :

def transform(row):
    # put the code that process each row here

result = df1.apply(transform, axis=1)

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.apply.html

In your case :

df_bull=df[(df['bull_bear']=='bull') & (df['high']<df['EMA20']) & (df['eod_dt']>start_dt)]

def transform(row):
    row_temp1=row[(df['index']>(row['index']-10)) & (df['index']<row['index'])]

    df_temp2=df_temp1[df_temp1['low']>df_temp1['EMA20']] 
    if not df_temp2.empty: #condition1 satisfied 
        df_temp1['high_1']=df_temp1['high'].shift(-1)
        df_temp2=df_temp1[(df_temp1['bull_bear']=='bull') & (df_temp1['high']<df_temp1['EMA20']) & (df_temp1['high_1']>df_temp1['high'])] 
        if not df_temp2.empty and len(df_temp2)<4: 
            #entry above signal bar high


df['result'] = df_bull.apply(transform, axis=1)

I am on mobile it is hard for me to write this post. Be gentle