KeyError：使用 drop_duplicates 時的 Int64Index([1], dtype='int64')

Question

我寫了一個簡單的腳本，它應該合並（聯合）一些數據框並刪除重復項。

例如，對於輸入：

df_A:
a  1
b  2

df_B:
b  2
c  3

預期的 output 將是：

df_out:
a  1
b  2
c  3

我寫了以下代碼：

def read_dataframes(filenames, basedir):
    return [pd.read_csv(basedir + file, sep='\t', header=None, quoting=csv.QUOTE_NONE) for file in filenames]


def merge_dataframes(dfs, out):
    merged = pd.concat(dfs).drop_duplicates(subset=[0, 1]).reset_index(drop=True)
    merged = merged.iloc[:, [0, 1, 2, 7, 8, 9]]
    merged.to_csv(out, header=None, index=None, sep='\t')

我以下列方式調用這些函數：

merge_dataframes(read_dataframes(filenames, basedir), output)

我遇到了KeyError異常：

Traceback (most recent call last):
  File "analysis_and_visualization.py", line 70, in <module>
    merge_dataframes(read_dataframes(wild_emb, wild_basedir), 'wild_emb_merged')
  File "analysis_and_visualization.py", line 17, in merge_dataframes
    merged = pd.concat(dfs).drop_duplicates(subset=[0, 1]).reset_index(drop=True)
  File "/Data/user/eliran/.local/lib/python3.6/site-packages/pandas/core/frame.py", line 5112, in drop_duplicates
    duplicated = self.duplicated(subset, keep=keep)
  File "/Data/user/eliran/.local/lib/python3.6/site-packages/pandas/core/frame.py", line 5248, in duplicated
    raise KeyError(diff)
KeyError: Int64Index([1], dtype='int64')

我究竟做錯了什么？

Answer 1

查看 frame.py 中的源代碼和frame.py duplicated

您的 dataframe 中的所有列似乎都不存在。

Class DataFrame()

def duplicated(self) - 片段

  # Verify all columns in subset exist in the queried dataframe
        # Otherwise, raise a KeyError, same as if you try to __getitem__ with a
        # key that doesn't exist.
        diff = Index(subset).difference(self.columns)
        if not diff.empty:
            raise KeyError(diff)

---

df = pd.DataFrame({'col1' : [0,1,2], 'col3' : [1,2,3]})

print(df)

  col1  col3
0     0     1
1     1     2
2     2     3


df.drop_duplicates(subset=['col1','col2'])

   5246         diff = Index(subset).difference(self.columns)
   5247         if not diff.empty:
-> 5248             raise KeyError(diff)
   5249 
   5250         vals = (col.values for name, col in self.items() if name in subset)

KeyError: Index(['col2'], dtype='object')

Answer 2

我認為這里的問題不是 column 1 ，因為第一列被轉換為index ，所以一些或所有 DataFrames 只有一個名為0的列。

為了防止它在read_csv中使用index_col=False參數：

def read_dataframes(filenames, basedir):
    return [pd.read_csv(basedir + file, sep='\t', header=None, quoting=csv.QUOTE_NONE, index_col=False) for file in filenames]

另一個問題應該是由於某種原因只有一列數據，所以第二列稱為2不存在。