使用理解更新数据框列

Question

I have been updating dataframe columns with list comprehensions for a while, without issue. If i have a filter on the dataframe, this raises problems, the column is not updated, even if the comprehension returns the correct values. The below is a contrived example, purely to illustrate the issue.

I first update the Town column to be the same as Region, if region is populated. I then try to find a value for Town in the Address if it has not been populated. Issue is that the second update statement does not work.

Its clear my understanding of comprehensions is not adequate, so would appreciate advice on what i am doing wrong. Thanks!

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import math
import pyodbc

#create dataframe

data = [{'Address': '123 Fake st, someTown, Nebraska', 'Region':'nebraska', 'metric1':50,'Town':''},
    {'Address': '2345 Fake st, someTown, Nebraska', 'Region':'nebraska', 'metric1':50,'Town':''},
    {'Address': '43 Fake st, someTown, Nebraska', 'Region':'nebraska', 'metric1':50,'Town':''},
    {'Address': '1 Fake st, someTown, Nebraska', 'Region':'nebraska', 'metric1':50,'Town':''},
    {'Address': '43 Fake st, someTown, NOBraska', 'Region':'', 'metric1':50,'Town':''},
    {'Address': '6 Fake st, someTown, NOBraska', 'Region':'', 'metric1':50,'Town':''},
    {'Address': '45 Fake st, someTown, NOBraska', 'Region':'', 'metric1':50,'Town':''},]

dataset = pd.DataFrame(data)

#set Town column to the region.
dataset['Town'] = [r for r in dataset['Region']]

#if Town column is still blank, find the region in the Address, correcting for a known bad spelling
dataset[dataset['Town'] =='']['Town']  =  ['Nebraska' if sub.split(",")[2].strip() =='NOBraska' else sub.split(",")[2].strip() for sub in dataset[dataset['Town'] =='']['Address'].astype(str)]  

#RESULT: dataset['Town'] is not updated for the case when it is empty are not updated

Answer 1

The problem here is that by using df[rows][cols] access method, you are not accessing the original DataFrame values, but a copy.

You should indeed receive a warning like:

SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

This situation is described in details here .

As a rule, you should always use .iloc or .loc when assigning to a slice of the DataFrame.

Here is an example of how you can re-write your assignment to actually modify the DataFrame:

new_values = ['Nebraska' if sub.split(",")[2].strip() =='NOBraska'
              else sub.split(",")[2].strip()
              for sub in dataset[dataset['Town'] =='']['Address'].astype(str)]

# In this way I am getting the labels of the index, so that I can use .loc
empty_town_rows = dataset.index[dataset['Town'] =='']

dataset.loc[empty_town_rows, 'Town']  =  new_values

Personally, I always prefer to use .loc/.iloc when doing modifying values of the DataFrame, so I would also re-write the first assignment. But this is not necessary as there is no issue of view vs copy.

dataset.loc[:, 'Town'] = [r for r in dataset['Region']]

Answer 2

I would suggest that you should use loc to update the value in a data frame.

In your case, you should use

dataset.loc[dataset['Town'] =='', 'Town'] = ['Nebraska' if sub.split(",")[2].strip() =='NOBraska' else sub.split(",")[2].strip() for sub in dataset[dataset['Town'] =='']['Address'].astype(str)]

Personally, I would suggest that you do this way

updateTown = lambda row: row["Region"] if row["Region"] else row["Address"].split(",")[2].strip()
dataset['Town'] = dataset.apply(updateTown, axis=1)

Answer 3

@FLab describes well the problem.

But your code can be improved further to make it more performant / readable:

def replacer(sub):
    x = sub.split(',')[2].strip()
    return 'Nebraska' if x == 'NOBaska' else x

dataset.loc[dataset['Town'] == '', 'Town']  =  \
dataset.loc[dataset['Town'] == '', 'Address'].astype(str).apply(replacer)