有条件地根据另一列中的值替换值

Question

I have a very large dataframe (~1.7MM rows x 6 columns). A simplified example of the relevant data is:

City        Borough

Brooklyn    Brooklyn
Astoria     Queens
Astoria     Unspecified
Ridgewood   Unspecified
Ridgewood   Queens

So I'm trying to fill the 'Unspecified' values based on the information from the City column. So for example, the City Ridgewood is in an Unspecified Borough in one instance, but correctly has the Borough listed as Queens elsewhere in the dataset.

I've already explored Panda's fillna, but it doesn't seem to meet my needs. I've also considered the np.where method, but I'm not sure how'd it work in this situation. I'm pretty new to Pandas, but maybe the map/apply function are what I need? This can probably be accomplished a thousand different ways, but looking for something that won't crawl given the size of the data.

EDIT: I was able to create a dictionary which contains the highest occurring "pairs" between cities and boroughs with the following code:

specified = data[['Borough','City']][data['Borough']!= 'Unspecified']
paired = specified.Borough.groupby(specified.City).max()
paired = paired.to_dict()

The paired dict has the city as the key and the borough as the value. Now the last step is to apply/map it back to the borough column...how do I do that?

Answer 1

Here's one way:

>>> d
         City      Borough
0   Brooklyn     Brooklyn
1    Astoria       Queens
2    Astoria  Unspecified
3  Ridgewood  Unspecified
4  Ridgewood       Queens
>>> realData = d[d.Borough != "Unspecified"]
>>> realData = pandas.Series(data=realData.Borough.values, index=realData.City)
>>> d['Borough'] = d.City.map(realData)
>>> d
         City   Borough
0   Brooklyn  Brooklyn
1    Astoria    Queens
2    Astoria    Queens
3  Ridgewood    Queens
4  Ridgewood    Queens

This assumes that every City has exactly one non-unspecified Borough value. (If a city has no value but Unspecified, the borough will show up as NA.)

Edit: If you've already created your dict as in your edited post, just use d['Borough'] = d.City.map(paired['Borough']) to map each city to the borough from your dict. map is a useful method to know about. It can map values either with a Pandas series, with a dict, or with a function that returns the mapped value given the key.