BigQuery 的 Python 客戶端不應該與多處理一起使用嗎？

Question

根據Python BigQuery 客戶端文檔，似乎多處理應該可以工作。 但是，在嘗試使用多處理從 pandas dataframe 對 BigQuery 表進行簡單加載時，我不斷收到錯誤消息，我想知道文檔中的以下語句是否與它有關。

在多處理場景中，最佳實踐是在multiprocessing.Pool或multiprocessing.Process調用os.fork()之后創建客戶端實例。

我根據這個GCP 文檔（google-cloud-bigquery）編寫了我的代碼，它只是試圖創建 2 個進程來加載兩個不同的 pandas dataframe 在同一張表上（我也嘗試將它們加載到兩個不同的表上並得到相同的錯誤）：

from google.cloud import bigquery
import pandas as pd
import numpy as np
import multiprocessing
import random

def trying():
    #This is just to create random values to create the dataframe.
    values = np.round(np.random.uniform(0,1, (6,14)),2)* random.uniform(0,1)
    df = pd.DataFrame(values, columns=list('abcdefghifklmn'))

    #I understand that I have to create the client for each process
    client = bigquery.Client()

    table_id = 'mydataset.new_table'
    job = client.load_table_from_dataframe(df, table_id)
    job.result()

if __name__ == '__main__':

    processes = []
    for i in range(2):
        p = multiprocessing.Process(target=trying)
        p.start()
        processes.append(p)
    for process in processes:
        process.join()

這是我得到的錯誤，我無法弄清楚發生了什么。 兩個進程都拋出相同的錯誤：

Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "test_multiproc_load.py", line 19, in trying
    job = client.load_table_from_dataframe(df, table_id)
  File "/home/.../snow/local/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 1932, in load_table_from_dataframe
    parquet_compression=parquet_compression,
  File "/home/.../snow/local/lib/python2.7/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 485, in dataframe_to_parquet
    arrow_table = dataframe_to_arrow(dataframe, bq_schema)
  File "/home/.../snow/local/lib/python2.7/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 449, in dataframe_to_arrow
    bq_to_arrow_array(get_column_or_index(dataframe, bq_field.name), bq_field)
  File "/home/.../snow/local/lib/python2.7/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 224, in bq_to_arrow_array
    return pyarrow.Array.from_pandas(series, type=arrow_type)
  File "pyarrow/array.pxi", line 755, in pyarrow.lib.Array.from_pandas
    return array(obj, mask=mask, type=type, safe=safe, from_pandas=True,
  File "pyarrow/array.pxi", line 269, in pyarrow.lib.array
    return _sequence_to_array(obj, mask, size, type, pool, c_from_pandas)
  File "pyarrow/array.pxi", line 38, in pyarrow.lib._sequence_to_array
    check_status(ConvertPySequence(sequence, mask, options, &out))
  File "/home/.../snow/local/lib/python2.7/site-packages/pandas/core/frame.py", line 2927, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/.../snow/local/lib/python2.7/site-packages/pandas/core/indexes/base.py", line 2659, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 127, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 147, in pandas._libs.index.IndexEngine._get_loc_duplicates
KeyError: 0

這是我的 dataframe 創建中的簡單索引錯誤嗎？ 你有什么建議嗎？

Answer 1

似乎問題出在columns=list('abcdefghifklmn')上。
我不確定為什么它不起作用，但如果我這樣指定columns=['a', 'b',...] ，它就會起作用。
如果有人能解釋原因，那就太好了。