BigQuery 的 Python 客户端不应该与多处理一起使用吗？

Question

As per the Python BigQuery client documentation , it seems that multiprocessing should work. But I keep getting an error when trying a simple load to a BigQuery table from a pandas dataframe using multiprocessing and I wonder if the following statement from the doc would have anything to do with it.

In multiprocessing scenarios, the best practice is to create client instances after multiprocessing.Pool or multiprocessing.Process invokes os.fork() .

I wrote my code based on this GCP doc (google-cloud-bigquery), that just tries to create 2 processes to load two different pandas dataframe on the same table (I have also tried to load them on two different tables and got the same error):

from google.cloud import bigquery
import pandas as pd
import numpy as np
import multiprocessing
import random

def trying():
    #This is just to create random values to create the dataframe.
    values = np.round(np.random.uniform(0,1, (6,14)),2)* random.uniform(0,1)
    df = pd.DataFrame(values, columns=list('abcdefghifklmn'))

    #I understand that I have to create the client for each process
    client = bigquery.Client()

    table_id = 'mydataset.new_table'
    job = client.load_table_from_dataframe(df, table_id)
    job.result()

if __name__ == '__main__':

    processes = []
    for i in range(2):
        p = multiprocessing.Process(target=trying)
        p.start()
        processes.append(p)
    for process in processes:
        process.join()

This is the error I get and I can't figure out what is going on. Both processes throw the same error:

Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "test_multiproc_load.py", line 19, in trying
    job = client.load_table_from_dataframe(df, table_id)
  File "/home/.../snow/local/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 1932, in load_table_from_dataframe
    parquet_compression=parquet_compression,
  File "/home/.../snow/local/lib/python2.7/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 485, in dataframe_to_parquet
    arrow_table = dataframe_to_arrow(dataframe, bq_schema)
  File "/home/.../snow/local/lib/python2.7/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 449, in dataframe_to_arrow
    bq_to_arrow_array(get_column_or_index(dataframe, bq_field.name), bq_field)
  File "/home/.../snow/local/lib/python2.7/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 224, in bq_to_arrow_array
    return pyarrow.Array.from_pandas(series, type=arrow_type)
  File "pyarrow/array.pxi", line 755, in pyarrow.lib.Array.from_pandas
    return array(obj, mask=mask, type=type, safe=safe, from_pandas=True,
  File "pyarrow/array.pxi", line 269, in pyarrow.lib.array
    return _sequence_to_array(obj, mask, size, type, pool, c_from_pandas)
  File "pyarrow/array.pxi", line 38, in pyarrow.lib._sequence_to_array
    check_status(ConvertPySequence(sequence, mask, options, &out))
  File "/home/.../snow/local/lib/python2.7/site-packages/pandas/core/frame.py", line 2927, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/.../snow/local/lib/python2.7/site-packages/pandas/core/indexes/base.py", line 2659, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 127, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 147, in pandas._libs.index.IndexEngine._get_loc_duplicates
KeyError: 0

Is this a simple indexing error in my dataframe creation? Would you have any suggestion?

Answer 1

It seems that the issue is with columns=list('abcdefghifklmn') .
I am not sure why it doesn't work but if I specify like so columns=['a', 'b',...] , it works.
If anybody could explain why, it would be great.