TypeError: can't pickle _thread.RLock objects in pandas with multiprocessing

Question

我有一个 12000 行的数据框。 我想将熊猫与多处理一起使用并在数据帧上执行映射。

df =  pd.read_csv(input_file, dtype=str, names=columns)
df_split = np.array_split(df, 4)
# pool = mp.Pool(4)
for df_data in df_split:
    param = [df_data, version, logger]
    with mp.Pool(4) as pool:
        out_df_lst = pool.map(func, param)
out_df = pd.concat(out_df_lst)

所有这些程序都在 Django REST API 中，当我通过 Postman 发出 POST 请求时，它会抛出错误：'TypeError: can't pickle _thread.RLock objects'。 当我在没有任何多处理的情况下发出请求时，该程序按预期工作。

请帮助我理解这个问题，以使程序与多处理一起工作。

这是整个 Traceback：

Traceback (most recent call last):
  File "C:\Users\kotamrajua\Anaconda3\envs\ahrqcomenv\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\kotamrajua\Anaconda3\envs\ahrqcomenv\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\kotamrajua\methodologies-pyapis\ahrq_comorbidity\ahrq_comorb_app\scripts\process_ahrq.py", line 97, in run_ahrq_process
    out_df_lst = pool.map(run_comorb_mapping, param)
  File "C:\Users\kotamrajua\Anaconda3\envs\ahrqcomenv\lib\multiprocessing\pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Users\kotamrajua\Anaconda3\envs\ahrqcomenv\lib\multiprocessing\pool.py", line 644, in get
    raise self._value
  File "C:\Users\kotamrajua\Anaconda3\envs\ahrqcomenv\lib\multiprocessing\pool.py", line 424, in _handle_tasks
    put(task)
  File "C:\Users\kotamrajua\Anaconda3\envs\ahrqcomenv\lib\multiprocessing\connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:\Users\kotamrajua\Anaconda3\envs\ahrqcomenv\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle _thread.RLock objects

Answer 1

from pandarallel import pandarallel
pandarallel.initialize()

                  # Include raw=True if the func needs ndarrays.
out_df = df.parallel_apply(func, args=(version, logger), axis=1)