Modin庫在做簡單的pandas操作時拋出錯誤

Question

我遇到了應該加速一些 Pandas 操作的 modin 庫並開始測試它。

雖然使用 read_csv 加載數據明顯更快，但在純 Pandas 中完美運行的簡單條件表達式，例如：

    df.loc[df['Score'] > 8,'Score_T2B'] = 1
    df.loc[df['Score'] < 9,'Score_T2B'] = 0

拋出許多錯誤：

回溯（最近一次調用最后一次）：

  File "<ipython-input-21-0b842942ffac>", line 1, in <module>
    df.loc[df['Score'] > 8,'Score_T2B'] = 1

  File "C:\ProgramData\Anaconda3\lib\site-packages\modin\pandas\indexing.py", line 251, in __setitem__
new_col[row_loc] = item

  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\series.py", line 1244, in __setitem__
setitem(key, value)

  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\series.py", line 1221, in setitem
self.loc[key] = value

  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 204, in __setitem__
indexer = self._get_setitem_indexer(key)

  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 191, in _get_setitem_indexer
return self._convert_to_indexer(key, axis=axis, is_setter=True)

  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 1285, in _convert_to_indexer
return self._get_listlike_indexer(obj, axis, **kwargs)[1]

  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 1092, in _get_listlike_indexer
keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing

  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 1177, in _validate_read_indexer
key=key, axis=self.obj._get_axis_name(axis)

KeyError: "None of [Index([False, False, False, False, False, False, False, False, False, False,\n       ...\n       False, False, False, False, False, False, False, False, False, False],\n      dtype='object', length=169815)] are in the [index]"

這應該是一個簡單的操作。 有沒有解決辦法，或者我只是錯過了加載以外的東西：

  import modin.pandas as pm  
  df = pm.read_csv(input_file, sep='\t', encoding='utf-8', low_memory=False)

非常感謝！

Answer 1

我發現read_csv的read_csv函數與替換掉的read_csv函數不完全一樣。 特別是，它也不做異常處理，並拋出原本可以與 Pandas 一起使用的異常。

也許最好在失敗時導入兩個版本並回退到熊貓？

這是我的進口商

try:
    import modin.pandas as pd # claims to be 4X faster loading csvs, using all processor cores
    print('modin.pandas active')
except ImportError:
    import pandas as pd

這是我未能正確 read_csv 的示例：

這會導致 modin 的 TypeError，但不會導致 pandas 的錯誤。 正在加載的文件不包含列名'IlmnID' 。

try:
    sample = pd.read_csv(part, index_col='IlmnID')
except ValueError:
    sample = pd.read_csv(part)

這適用於 modin.pd.read_csv，因為沒有 try/except 包裝器

sample = pd.read_csv(part)
if 'IlmnID' in sample.columns:
    sample.set_index('IlmnID', inplace=True)
elif 'illumina_id' in sample.columns:
    sample.set_index('illumina_id', inplace=True)
    sample.rename(index={'illumin_id': 'IlmnID'}, inplace=True)
else:
    # assume first column
    guess_index = columns[0]
    sample.set_index(guess_index, inplace=True)
    sample.rename(index={guess_index: 'IlmnID'}, inplace=True)