使用 Google Cloud 读取和写入泡菜

Question

I want to read an existing pickle (that contains a dictionary) which is stored in a folder inside a Google Cloud Bucket. Then update the pickle after performing some functions, which is equal to overwriting the pickle.

Traditionally I would do something like:

import pickle
# Read pickle:
pickle_in = open('dictionary.pickle','rb')
my_dictionary = pickle.load(pickle_in)
my_dictionary 

# MODIFY DICTIONARY BY, FOR EXAMPLE, ADDING NEW REGISTERS

# Overwrite pickle:
pickle_out = open('dictionary.pickle','wb') 
pickle.dump(my_modified_dictionary,pickle_out)
pickle_out.close()

Now I need to do something similar but on Google Cloud. So I need to change the path of the file and use cloudstorage.open():

import pickle
my_path = '/bucket_name/pickle_folder/my_dictionary.pickle'

# Read pickle:
pickle_in = cloudstorage.open(path,'r')
my_dictionary = pickle.load(pickle_in)
my_dictionary 

# MODIFY DICTIONARY BY, FOR EXAMPLE, ADDING NEW REGISTERS

# Overwrite pickle:
pickle_out = cloudstorage.open(path,'w') 
pickle.dump(my_modified_dictionary,pickle_out)
pickle_out.close()

Will this work? cloudstorage.open() seems to be the equivalent to open(). But I am not sure that if I specify the path when dumping the pickle will actually overwrite the pickle on the specified folder.

Answer 1

The basic idea of doing read-modify-write from GCS is possible. You should be aware that this will not work well with concurrency - if a second process does a read before the the first writes back, then when the second process writes back it will lose the first process's changes. The best solution to this is to use a database rather than pickling to GCS.

In addition, be aware that pickle is not secure , and you should not load pickles you didn't write.

If you do still want to use GCS for this you should use the standard GCS client library , something like:

from google.cloud import storage

storage_client = storage.Client()

bucket = storage_client.bucket('your-gcs-bucket')
blob = bucket.blob('dictionary.pickle')
pickle_in = blob.download_as_string()
my_dictionary = pickle.loads(pickle_in)

# MODIFY DICTIONARY BY, FOR EXAMPLE, ADDING NEW REGISTERS

pickle_out = pickle.dumps(my_modified_dictionary)
blob.upload_from_string(pickle_out)