加载操作中的BigQuery错误：叶字段总数过多

Question

I am uploading a big file into BigQuery, because it is too slow to operate on my own PC. bq --location=EU load --field_delimiter='\\t' --skip_leading_rows=1 --source_format=CSV single_cells.retinal_bipolar gs://single_cells/retinal-bipolar-neuron-drop-seq/exp_matrix.txt ./schema.json However, I got an error:

BigQuery error in load operation: Too many total leaf fields: 27500

Indeed, the data has 27500 columns; doesn't BigQuery allow this?

Answer 1

The Maximum columns per table value is 10,000 which means that the error message was thrown since the load job exceeded the maximum number of fields allowed in the schema.

Based on this, an available workaround is to split the table up into smaller tables in order to reduce the number of fields contained; In this way, you will be able to avoid this issue.

Answer 2

Another workaround - as Elliott pointed - you can load your file into BigQuery table that is set with schema having ONLY one column of type STRING. Then (assuming that file and row size does not exceed size limitations ) - you will be able to use all power of BigQuery!