Json 在 Vertex AI 上使用 Apache 光束时出现加载错误

Question

I am trying to load data from datastore to bigquery using Apache beam in Vertex AI notebook. This is the code part where the loading happens-

from apache_beam.io.gcp.datastore.v1new.datastoreio import ReadFromDatastore
import apache_beam as beam
from apache_beam.io.gcp.bigquery_file_loads import BigQueryBatchFileLoads

table_row = (p 
                 | 'DatastoreGetData' >> ReadFromDatastore(query=myquery)
                 | 'EntityConversion' >> beam.Map(ent_to_json_func, table_schema)
                 | 'Final' >> BigQueryBatchFileLoads(
                      destination=lambda row: f"myproject:dataset.mytable",
                      custom_gcs_temp_location=f'gs://myproject/beam',
                      write_disposition='WRITE_TRUNCATE',
                      schema=table_schema
                 )
                )

table_schema is the json version of BigQuery table schema (attached Column mapping pic below). ent_to_json_func converts fields coming from datastore to corresponding BigQuery field in the correct format.

I am trying to load just one row from datastore, it is giving error. The data looks like this-

{ " key ": { "namespace": null, "app": null, "path": "Table/12345678", "kind": "Mykind", "name": null, "id": 12345678 }, "col1": false, "col2": { "namespace": null,
"app": null, "path": "abc/12345", "kind": "abc", "name": null, "id": 12345 }, "col3": "6835218432", "col4": {
"namespace": null, "app": null, "path": null, "kind": null, "name": null, "id": null }, "col5": false,
"col6": null, "col7": "https://www.somewebsite.com/poi/",
"col8": "0.00", "col9": "2022-03-12 03:44:17.732193+00:00",
"col10": "{"someid":"NAME","col7":"https://www.somewebsite.com/poi/", "provided":"Yes","someid2":"SDFTYI1090"}", "col11": ""0.00"", "col12": "{}", "col13": [] }

The column mapping is here

The error is as follows-

--------------------------------------------------------------------------- 
RuntimeError                              Traceback (most recent call last) 
~/apache-beam-2.41.0/packages/beam/sdks/python/apache_beam/runners/common.py in process(self, windowed_value)
   1416     try:
-> 1417       return self.do_fn_invoker.invoke_process(windowed_value)
   1418     except BaseException as exn:

~/apache-beam-2.41.0/packages/beam/sdks/python/apache_beam/runners/common.py in invoke_process(self, windowed_value, restriction, watermark_estimator_state, additional_args, additional_kwargs)
    837       self._invoke_process_per_window(
--> 838           windowed_value, additional_args, additional_kwargs)
    839     return residuals

~/apache-beam-2.41.0/packages/beam/sdks/python/apache_beam/runners/common.py in _invoke_process_per_window(self, windowed_value, additional_args, additional_kwargs)
    982         windowed_value,
--> 983         self.process_method(*args_for_process, **kwargs_for_process),
    984         self.threadsafe_watermark_estimator)

~/apache-beam-2.41.0/packages/beam/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py in process(self, element, dest_ids_list)
    753       # max_retries to 0.
--> 754       self.bq_wrapper.wait_for_bq_job(ref, sleep_duration_sec=10, max_retries=0)
    755 

~/apache-beam-2.41.0/packages/beam/sdks/python/apache_beam/io/gcp/bigquery_tools.py in wait_for_bq_job(self, job_reference, sleep_duration_sec, max_retries)
    637             'BigQuery job {} failed. Error Result: {}'.format(
--> 638                 job_reference.jobId, job.status.errorResult))
    639       elif job.status.state == 'DONE':

RuntimeError: BigQuery job beam_bq_job_LOAD_AUTOMATIC_JOB_NAME_LOAD_STEP_187_4cab298bbd73af86496c64ca35602a05_a5309204fb004ae0ba8007ac2169e079 failed. 
Error Result: <ErrorProto  
location: 'gs://myproject/beam/bq_load/63e94c1a210742aabab09f96/myproject.dataset.mytable/aed960fb-0ae6-489a-9cb8-e012eee0d9c8'  
message: 'Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details. 
File: gs://myproject/beam/bq_load/63e94c1a210742aabab09f96/myproject.dataset.mytable/aed960fb-0ae6-489a-9cb8-e012eee0d9c8'  
reason: 'invalid'>

Please let me know what to do. How to determine the exact cause of the error? I have also validated this json through a json validator, there is no issue.

UPDATE:

I found out that the issue is due to BYTES column. From datastore, bytes type is coming, which I am converting to string using decode and saving in json. When I upload that json into BigQuery, it gives error.

How to proceed in this case?

Answer 1

To solve your issue you have to set STRING type for COL10 COL11 COL12 and COL13 in the BigQuery table.

Your final Dict in the PCollection need to match exactly the schema of the BigQuery table.

In your json I saw these columns as String , your schema needs also to have STRING type for these columns.

Answer 2

The error message is accurate. The file attempted to load to bigquery is not valid json string, see "{"someid":"NAME","col7":"https://www.somewebsite.com/poi/" there are quotes without escape.

Load to bytes type should actually be fine.