根据列值删除行中的重复项

Question

嗨，找不到任何关于此的具体内容，抱歉，如果它是重复的...

如何删除包含相同信息的单行的列值（有一些例外）

例子：

      Name     Age     Job    How_Old    Occupation   Happy   Married?
 0    John     35      Dev    35         Dev          True    True
 1    Sally    42      CA     42         CA           False   False

我想删除包含相同信息的不同名称的列，除了包含一些明显重复的列，如二进制列。

Output：

     Name     Age    Job   Happy    Married?
0    John     35     Dev   True     True
1    Sally    42     CA    False    False

谢谢，还请注意，我需要在 massvie flattend 和标准化 json 文件上执行此操作，因此循环将非常耗时。

Answer 1

First exlude boolean columns by DataFrame.select_dtypes , transpose and get duplicates by DataFrame.duplicated per all rows, then invert mask by ~ and add removed boolean columns by Series.reindex , last is filtered by DataFrame.loc for all rows by first : and按掩码的列名称：

m = (~df.select_dtypes(exclude=bool).T.duplicated()).reindex(df.columns, fill_value=True)

另一个想法是将值转换为元组并调用Series.duplicated ：

m = ((~df.select_dtypes(exclude=bool).apply(tuple).duplicated())
         .reindex(df.columns, fill_value=True))

---

df = df.loc[:, m]
print (df)
    Name  Age  Job  Happy  Married?
0   John   35  Dev   True      True
1  Sally   42   CA  False     False

详情：

#exlude boolean columns
print (df.select_dtypes(exclude=bool))
    Name  Age  Job  How_Old Occupation
0   John   35  Dev       35        Dev
1  Sally   42   CA       42         CA

#transpose
print (df.select_dtypes(exclude=bool).T)
               0      1
Name        John  Sally
Age           35     42
Job          Dev     CA
How_Old       35     42
Occupation   Dev     CA

#checked duplicates per all columns
print (df.select_dtypes(exclude=bool).T.duplicated())
Name          False
Age           False
Job           False
How_Old        True
Occupation     True

#inverse mask True->False, False->True
print ((~df.select_dtypes(exclude=bool).T.duplicated()))
Name           True
Age            True
Job            True
How_Old       False
Occupation    False
dtype: bool

---

#added removed boolean columns with Trues
print ((~df.select_dtypes(exclude=bool).T.duplicated())
           .reindex(df.columns, fill_value=True))
Name           True
Age            True
Job            True
How_Old       False
Occupation    False
Happy          True
Married?       True
dtype: bool

Answer 2

定义如下 function，返回要删除的列名列表：

def chkColToDel(df):
    # Column names excluding bool columns
    cols = df.select_dtypes(exclude=bool).columns.tolist()
    colsToDel = []
    while len(cols) > 1:
        cn1 = cols.pop(0)        # Column name, left side
        if cn1 not in colsToDel: # Not marked for deletion earlier
            c1 = df[cn1]         # The column itself
            t1 = c1.dtype.name   # Type name
            for cn2 in cols:     # Check remaining columns
                c2 = df[cn2]     # Column name, right side
                if t1 == c2.dtype.name and c1.equals(c2):
                    # Same types and equal values
                    colsToDel.append(cn2) # Mark for deletion
    return colsToDel

然后调用它：

colsToDel = chkColToDel(df)

剩下的唯一事情是删除返回的列，如果有的话：

if len(colsToDel) > 0:
    df.drop(columns=colsToDel, inplace=True)

我假设您的帖子中提到的一些例外实际上是指bool列。 如果异常列表更广泛，请相应地更改我的代码。