根据cKDTree索引从熊猫数据框中选择行

Question

I was trying to do some quick-and-dirty reverse geocoding.

I have the dataframe poi (around 50,000 rows), where each point of interest has a lat/lng coordinate.

I have also the dataframe postcode_existing (around 180,000 rows), which maps lat/lng coordinates to postcodes.

I pulled out the relevant coordinate columns and used cKDTree to determine, for each point of interest in poi , the nearest lat/lng coordinate in postcode_existing .

import pandas as pd
import numpy as np
from scipy.spatial import cKDTree

# read poi and postcode csv files

# Extract subset
postcode_existing_coordinates = postcode_existing[['Latitude', 'Longitude']]

# Extract subset
poi_coordinates = poi[['Latitude', 'Longitude']]

# Construct tree
tree = cKDTree(postcode_existing_coordinates)

# Query
distances, indices = tree.query(poi_coordinates)

I end up with the relevant indices. I am now looking to select the rows from the dataframe postcode_existing using those indices.

I tried postcode_existing.ix[indices] , but this seems not to get the correct rows.

For example:

>>> postcode_existing.ix[indices].head()
       Postcode  Latitude  Longitude   Easting  Northing   GridRef  \
78579   HA3 0NS  51.57553  -0.304296  517605.0  187658.0  TQ176876   
178499      NaN       NaN        NaN       NaN       NaN       NaN   
62392       NaN       NaN        NaN       NaN       NaN       NaN   
78662   HA3 0TA  51.58409  -0.288764  518659.0  188635.0  TQ186886   
79470       NaN       NaN        NaN       NaN       NaN       NaN   

                County District    Ward DistrictCode   ...   Terminated  \
78579   Greater London    Brent  Kenton    E09000005   ...          NaN   
178499             NaN      NaN     NaN          NaN   ...          NaN   
62392              NaN      NaN     NaN          NaN   ...          NaN   
78662   Greater London    Brent  Kenton    E09000005   ...          NaN   
79470              NaN      NaN     NaN          NaN   ...          NaN   

       Parish NationalPark Population Households   Built up area  \
78579     NaN          NaN       72.0       25.0  Greater London   
178499    NaN          NaN        NaN        NaN             NaN   
62392     NaN          NaN        NaN        NaN             NaN   
78662     NaN          NaN      152.0       39.0  Greater London   
79470     NaN          NaN        NaN        NaN             NaN   

       Built up sub-division  Lower layer super output area  \
78579                  Brent                     Brent 004D   
178499                   NaN                            NaN   
62392                    NaN                            NaN   
78662                  Brent                     Brent 003E   
79470                    NaN                            NaN   

                    Rural/urban  Region  
78579   Urban major conurbation  London  
178499                      NaN     NaN  
62392                       NaN     NaN  
78662   Urban major conurbation  London  
79470                       NaN     NaN  

[5 rows x 25 columns]

But:

>>> postcode_existing.iloc[78579]
Postcode                                                  NW1 3AU
Latitude                                                  51.5237
Longitude                                               -0.143188
Easting                                                    528915
Northing                                                   182163
GridRef                                                  TQ289821
County                                             Greater London
District                                              Westminster
Ward                                       Marylebone High Street
DistrictCode                                            E09000033
WardCode                                                E05000641
Country                                                   England
CountyCode                                              E11000009
Constituency                     Cities of London and Westminster
Introduced                                             1980-01-01
Terminated                                                    NaN
Parish                                                        NaN
NationalPark                                                  NaN
Population                                                      7
Households                                                      1
Built up area                                      Greater London
Built up sub-division                         City of Westminster
Lower layer super output area                    Westminster 013A
Rural/urban                               Urban major conurbation
Region                                                     London
Name: 133733, dtype: object

Also:

>>> postcode_existing.iloc[178499]
Postcode                                        WC1E 6JL
Latitude                                         51.5236
Longitude                                      -0.135522
Easting                                           529447
Northing                                          182168
GridRef                                         TQ294821
County                                    Greater London
District                                          Camden
Ward                                          Bloomsbury
DistrictCode                                   E09000007
WardCode                                       E05000129
Country                                          England
CountyCode                                     E11000009
Constituency                      Holborn and St Pancras
Introduced                                    1980-01-01
Terminated                                           NaN
Parish                                               NaN
NationalPark                                         NaN
Population                                             1
Households                                             1
Built up area                             Greater London
Built up sub-division                             Camden
Lower layer super output area                Camden 026D
Rural/urban                      Urban major conurbation
Region                                            London
Name: 307029, dtype: object

These appear to be correct.

Why does postcode_existing.ix[indices] not select the correct rows? What should I be using instead?

Answer 1

I solved the problem. The issue was a mismatch between the position in the dataframe and the index due to the removal of certain rows.

To fix this, I simply reset the index:

postcode_existing.reset_index(inplace=True, drop=True)

I was then able to use loc to extract the relevant rows:

postcode_existing.loc[indices]

Answer 2

The problem is that you are using integers in your index. This messes things up as pandas attempts to keep track of list based locations as well as labels. ix attempts to figure it out. It is interpreting indices as list locations. In this case, use loc

Documentation

DataFrame.ix A primarily label-location based indexer, with integer position fallback.

.ix[] supports mixed integer and label based access. It is primarily label based, but will fall back to integer positional access unless the corresponding axis is of integer type.

.ix is the most general indexer and will support any of the inputs in .loc and .iloc. .ix also supports floating point label schemes. .ix is exceptionally useful when dealing with mixed positional and label based hierachical indexes.

However, when an axis is integer based, ONLY label based access and not positional access is supported. Thus, in such cases, it's usually better to be explicit and use .iloc or .loc .