将数据从 GCS 中的文件加载到 GCP firestore

Question

I have written a script which loops through each record in the file and does write to the firestore collection.

Firestore Schema {COLLECTION.DOCUMENT.SUBCOLLECTION.DOCUMENT.SUBCOLLECTION}

     '{"KEY":"1234","DATE":"2022-10-10","SUB_COLLECTION":{"KEY":1234,"SUB_DOC":{"KEY1" : :"VAL1"}}'
     '{"KEY":"1235","DATE":"2022-10-10","SUB_COLLECTION":{"KEY":1235,"SUB_DOC":{"KEY1" : :"VAL1"}}'
     '{"KEY":"1236","DATE":"2022-10-10","SUB_COLLECTION":{"KEY":1236,"SUB_DOC":{"KEY1" : :"VAL1"}}'
...

File is read in the below line

read_file = filename.download_as_string()

converted to a list of strings

    fire_client = firestore.Client(project=PROJECT)
    dict_str = read_file.decode("UTF-8");
    dict_str = dict_str.split('\n');
    dict_str = dict_str.split('\n');
        for i in range(0,len(dict_str)-1):
         i = json.loads(dict_str[i])
         doc_ref = fire_client.collection('STATIC_COLLECTION_NAME').document(i['KEY'])
         doc_ref.set({"KEY" : int(i['KEY']), "DATE" : i['DATE']})
         sub_ref = doc_ref.collection('STATIC_SUB_COLLECTION_NAME').document('STATIC_SUB_DOC_NAME')
         sub_ref.set(i['SUB_COLLECTION'])

However, this job is consuming hours to complete a file size of 100 MB. Is there a way I could do this using multiple writes at a time, example batch processing of X number of records from the file and write those to X documents and sub-collections in the firestore. Finding a way to make this more efficient instead of looping over millions of records, my current script ended up with 503 The datastore operation timed out, or the data was temporarily unavailable.

Answer 1

You'll want to use the bulk_writer to accumulate & send writes to Firestore