如何将腌制文件从 jupyter 环境保存到 GCP 存储桶？

Question

我正在尝试使用以下 function 将 arviz 推断 object 作为泡菜文件保存到 gcp 存储桶：

def upload_to_bucket(model, blob_name, bucket_name):
    """ 
    model: trace object
    Upload data to a bucket"""
     
    # Explicitly use service account credentials by specifying the private key
    # file.
    storage_client = storage.Client.from_service_account_json(
        'key1.json')

    #print(buckets = list(storage_client.list_buckets())

    bucket = storage_client.get_bucket(bucket_name)
    blob = bucket.blob(blob_name)
    
    with open(f"gs://{config['GCS_bucket']}//{config['blob_name']}//", 'wb') as filehandler4:
        # Call load method to deserialze
        pickle.dump(model, filehandler4, protocol=4)
    
    #returns a public url
    return blob.public_url

upload_to_bucket(model=trace, blob_name='korea_hierarchy_seasonal_price.pkl', bucket_name='korea-forecasting-bucket')

但是，我不断收到以下错误：

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
/tmp/ipykernel_45766/484011007.py in <module>
----> 1 upload_to_bucket(model=trace, blob_name='korea_hierarchy_seasonal_price.pkl', bucket_name='korea-forecasting-bucket')

~/google_bucket.py in upload_to_bucket(model, blob_name, bucket_name)
     46     #     pickle.dump(model, filehandler4, protocol=4)
     47 
---> 48     with open(f"gs://{'GCS_bucket'}//{'blob_name'}//", 'wb') as filehandler4:
     49         # Call load method to deserialze
     50         pickle.dump(model, filehandler4, protocol=4)

FileNotFoundError: [Errno 2] No such file or directory: 'gs://GCS_bucket//blob_name//'

保存到 gcp 云存储时，类似问题的答案使用gs名字，所以我不确定我做错了什么。

Answer 1

您需要使用云存储 (GCS) 客户端库的方法来读取|写入对象 (blob)。

您正在创建一个常规（ext）文件系统 object 并将其提供给 GCS，这将不起作用。

一种选择是使用 GCS 等效的open 。

（我没有尝试过，但是）你应该能够：

with blob.open("wb") as f:
    pickle.dump(mode,f,protocol=4)