按熊猫数据框中的数据分组

Question

I have a set of points in the following format. For each Id1, there is a set of Id2's with their latitude and longitude. For each group of Id1 and Id2, I need the first record in the group except for the last group, in which i need both the first and last records. Please see below the output.

Just wondering is there a way in pandas I can achieve this. Any help would be appreciated. Thanks.

  Id1       Id2       Latitude        Longitude
  179       183       33.67           -83.24
  179       183       31.33           -83.11
  179       184       33.58           -83.34
  179       184       33.61           -83.14
  179       185       33.60           -83.54
  179       185       33.60           -83.54
  180       185       33.67           -83.64
  180       185       33.79           -83.74
  180       186       33.81           -83.84
  180       186       33.81           -83.84

Output:

  Id1       Id2       Latitude        Longitude
  179       183       33.67           -83.24
  179       184       33.58           -83.34
  179       185       33.60           -83.54
  179       185       33.60           -83.54
  179       186       33.79           -83.74
  179       186       33.81           -83.84
  180       185       33.67           -83.64
  180       186       33.81           -83.84
  180       186       33.81           -83.88

Answer 1

You can first find last value of column Id2 , then filter where are no last values by boolean indexing and drop_duplicates :

last_row = df['Id2'].iloc[-1]
print (last_row)
186

df1 = df[df.Id2 != last_row].drop_duplicates(['Id1','Id2'])
print (df1)
   Id1  Id2  Latitude  Longitude
0  179  183     33.67     -83.24
2  179  184     33.58     -83.34
4  179  185     33.60     -83.54
7  180  185     33.67     -83.64

Then filter values with last value in Id2 :

df2 = df[df.Id2 == last_row]
print (df2)
    Id1  Id2  Latitude  Longitude
5   179  186     33.81     -83.84
6   179  186     33.81     -83.84
9   180  186     33.81     -83.84
10  180  186     33.81     -83.84

And last concat together:

print (pd.concat([df1,df2]).reset_index(drop=True))
   Id1  Id2  Latitude  Longitude
0  179  183     33.67     -83.24
1  179  184     33.58     -83.34
2  179  185     33.60     -83.54
3  180  185     33.67     -83.64
4  179  186     33.81     -83.84
5  179  186     33.81     -83.84
6  180  186     33.81     -83.84
7  180  186     33.81     -83.84

If need only last group change condition:

lastId1 = df['Id1'].iloc[-1]
print (lastId1)
180
lastId2 = df['Id2'].iloc[-1]
print (lastId2)
186

mask = (df.Id1 == lastId1) & (df.Id2 == lastId2)
df1 = df[~mask].drop_duplicates(['Id1','Id2']).drop_duplicates(['Id1','Id2'])
print (df1)
   Id1  Id2  Latitude  Longitude
0  179  183     33.67     -83.24
2  179  184     33.58     -83.34
4  179  185     33.60     -83.54
5  179  186     33.81     -83.84
7  180  185     33.67     -83.64

df2 = df[mask]
print (df2)
    Id1  Id2  Latitude  Longitude
9   180  186     33.81     -83.84
10  180  186     33.81     -83.84

print (pd.concat([df1,df2]).reset_index(drop=True))
   Id1  Id2  Latitude  Longitude
0  179  183     33.67     -83.24
1  179  184     33.58     -83.34
2  179  185     33.60     -83.54
3  179  186     33.81     -83.84
4  180  185     33.67     -83.64
5  180  186     33.81     -83.84
6  180  186     33.81     -83.84

Answer 2

This one is a bit less pandas-oriented but it uses group-by operations:

# Function to check if row values are equal to last_group values
def compare_to_groups(x, last_groups):
    return  any((x[['Id1','Id2']] == last_groups[i]).all(1).any() for i in range(len(last_groups)))

# Get the last groups
last_groups = df.groupby('Id1')['Id1','Id2'].last().values

# Apply to each group
df.groupby(['Id1','Id2']).apply(lambda x: x if compare_to_groups(x, last_groups) else x.head(1)).reset_index(drop=True)

It should yield:

   Id1  Id2  Latitude  Longitude
0  179  183     33.67     -83.24
1  179  184     33.58     -83.34
2  179  185     33.60     -83.54
3  179  185     33.60     -83.54
4  180  185     33.67     -83.64
5  180  186     33.81     -83.84
6  180  186     33.81     -83.84

I hope this helps.