带有 np.where 的 Pandas SettingWithCopyWarning

Question

I have a pandas data frame that has a column containing a bunch of correlations (all float values). I'm trying to create another column to categorise these correlations into three distinct categories (high/medium/low). I do this using np.where:

df['Category'] = np.where(df['Correlation'] >= 0.5, 'high', 
                                   np.where(data['Correlation'] >= 0.3, 'medium','low'))

When I try doing this, I always get the SettingWithCopyWarning (it seems to work though). I have read up on the difference between copies and views, and even seen recommendations to use .where over other methods to avoid any confusion (and the SettingWithCopyWarning). I still can't quite wrap my head around why I get the warning with this method, can someone explain?

Answer 1

Most likely your df has been created as a view of another DataFrame, eg:

data = pd.DataFrame({'Correlation': np.arange(0, 1.3, 0.1)})  # Your "initial" DataFrame
df = data.iloc[0:11]

Now df holds some fragment of data , but it uses the data buffer of data .

Then if you attempt to run:

df['Category'] = np.where(df['Correlation'] >= 0.5, 'high',
    np.where(df['Correlation'] >= 0.3, 'medium', 'low'))

just the mentioned warning occurs.

To get rid of it, create df as an independent DataFrame, eg:

df = data.iloc[0:11].copy()

Now df uses its own data buffer and you may do with it whatever you wish, including adding new columns.

To check whether your df uses its own data buffer, run:

df._is_view

In your original environment (without my correction) you should get False , but after you created df using .copy() you should get True .