Pandas, set on a copy of a slice from DataFrame problem

Question

I've been struggling with a Pandas warning for a while now.

I have this pretty simple piece of code:

def get_delta(df): 
    df['delta_current_day'] = df['close'] - df['open']
    df = df[pd.notnull(df['delta_current_day'])]

    df['delta_next_day'] = df['delta_current_day'].shift(-1)

    return df

Every time I get this error:

A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

I've read a lot of question about it here on stackoverflow but none works for me. I've tried somethings like this:

 df.loc[:, 'delta_next_day'] = df['delta_current_day'].shift(-1)

# OR 
list = df['delta_current_day'].tolist()
df.loc[:, 'delta_next_day'] = list[:1]

but I still get the same error.

What am I doing wrong?

Answer 1

I think you need DataFrame.copy :

df = df[pd.notnull(df['delta_current_day'])].copy()
#df = df.loc[pd.notnull(df['delta_current_day'])] #or this
df['delta_next_day'] = df['delta_current_day'].shift(-1)

---

this is a slice:

df = df[pd.notnull(df['delta_current_day'])]

A ( df['delta_current_day'].shift(-1) ) value is trying to be set on a ( df[pd.notnull(df['delta_current_day'])] ) copy of a slice from a DataFrame

Answer 2

You could also try this:

# Sample data.
df = pd.DataFrame({
    'open': [100, 102, np.nan, 102],
    'close': [101.5, 102.5, np.nan, 104]
})

def get_delta(df):
    df = df.dropna().assign(delta_current_day=df.eval('close - open'))
    return df.assign(delta_next_day=df['delta_current_day'].shift(-1))

>>> get_delta(df)
    open  close  delta_current_day  delta_next_day
0  100.0  101.5                1.5             0.5
1  102.0  102.5                0.5             2.0
3  102.0  104.0                2.0             NaN

Of course, this introduces look ahead bias. Why are you using tomorrow's delta today?