熊猫-如何设置选择删除重复项的规则

Question

I have a dataset with values in the column and in the index datetime objects. What I want to do is drop the values that have the same index (date&time), but I want to make a rule like:

---

I have two values for the same datetime, so I want to pick the one closer to number X. That number could be for example the mean value of the whole dataset.

---

The dataset looks like this (I did reset the index when trying to make it happen because I got an error for having multiple indices):

         index                  kwh
16391   2014-10-26 03:14:59     0.0514139
16392   2014-10-26 03:29:59     0.0323344
16393   2014-10-26 03:29:59     12.3
16394   2014-10-26 03:44:59     0.0595618
16395   2014-10-26 03:59:59     0.0338677

if X (for example mean value) is 0.05 then what I want to get back is

16391   2014-10-26 03:14:59     0.0514139
16392   2014-10-26 03:29:59     0.0323344
16393   2014-10-26 03:44:59     0.0595618
16394   2014-10-26 03:59:59     0.0338677

I have tried using groupby and apply in several different ways but I cant get it to work. Any help please?

Answer 1

If you add a dist column to the DataFrame which measures the absolute distance between kwh and X :

X = df['kwh'].mean()
df['dist'] = (df['kwh'] - X).abs()

then you can groupby index and find the unique integer indices with the minimum dist for each group:

idx = df.groupby(['index'])['dist'].transform(lambda x: x == x.min()).astype(bool)

Then you can select those rows using df.loc :

df.loc[idx]

---

If data contains (note the duplicate values of kwh for the same index ):

         index                  kwh
16391   2014-10-26 03:14:59     0.0514139
16392   2014-10-26 03:29:59     0.0323344
16392   2014-10-26 03:29:59     0.0323344
16393   2014-10-26 03:29:59     12.3
16394   2014-10-26 03:44:59     0.0595618
16395   2014-10-26 03:59:59     0.0338677

then

import pandas as pd

df = pd.read_table('data', sep='\s{2,}')
print(df)
X = df['kwh'].mean()
df['dist'] = (df['kwh'] - X).abs()
idx = df.groupby(['index'])['dist'].transform(lambda x: x == x.min()).astype(bool)
print(df.loc[idx])

yields

                     index       kwh      dist
16391  2014-10-26 03:14:59  0.051414  2.033505
16392  2014-10-26 03:29:59  0.032334  2.052584
16392  2014-10-26 03:29:59  0.032334  2.052584
16394  2014-10-26 03:44:59  0.059562  2.025357
16395  2014-10-26 03:59:59  0.033868  2.051051

Note that by using transform here, we get a boolean mask which allows us to select all rows -- including those with duplicate values of kwh -- which have the minimum distance from X .

---

You could use del df['dist'] to drop the dist column when you no longer need it.