在按日期分区的两个 GCS 存储桶之间复制文件

Question

I have a requirement to copy the file between two bucket detailed below -

Bucket A /folder A is source inbound box for daily files which are created as f1_abc_20210304_000 > I want to scan the latest file in folder A (10 files every day) and copy the latest file and next > Copy it in to Bucket B/Folder B / FILE name (ie from 10 files) / 2021/03/04 and drop the files in 04 folder.

Any suggestion how I should proceed with the design?

Thanks RG

Answer 1

One approach is by using client libraries, for the example below I'm using the python client library for google cloud storage .

move.py

from google.cloud import storage
from google.oauth2 import service_account
import os 

# as mention on https://cloud.google.com/docs/authentication/production
key_path = "credentials.json"
credentials = service_account.Credentials.from_service_account_file(key_path)

storage_client = storage.Client(credentials=credentials)

bucket_name = "source-bucket-id" 
destination_bucket_name = "destination-bucket-id"

source_bucket = storage_client.bucket(bucket_name)
# prefix 'original_data' is the folder where i store the data
array_blobs = source_bucket.list_blobs(prefix='original_data')

filtered_dict = []

for blob in array_blobs:
    if str(blob.name).endswith('.csv'):
        #add additional logic to handle the files you want to ingest
        filtered_dict.append({'name':blob.name,'time':blob.time_created})

orderedlist = sorted(filtered_dict, key=lambda d: d['time'], reverse=True) 
latestblob = orderedlist[0]['name']

# prefix 'destination_data' is the folder where i want to move the data
destination_blob_name = "destination_data/{}".format(os.path.basename(latestblob))


source_blob = source_bucket.blob(latestblob)
destination_bucket = storage_client.bucket(destination_bucket_name)

blob_copy = source_bucket.copy_blob(source_blob, destination_bucket, destination_blob_name)

print(
        "Blob {} in bucket {} copied to blob {} in bucket {}.".format(
            source_blob.name,
            source_bucket.name,
            blob_copy.name,
            destination_bucket.name,
        )
    )

For a bit of context on the code, what I did was to use the google cloud storage python client , log in, get the list of files from my source folder original_data inside bucket source-bucket-id and add the relevant files ( you can modify the pick up logic by adding your own criteria which fits your situation ). After that I pick up the latest files based on time creation and use that name to move it into my destination-bucket-id . As a note, destination_bucket_name variable includes the folder where I want to allocate the file and also the end filename.

UPDATE : I miss the airflow tag. So on that case you should use the operator that comes with google provider which is GCSToGCSOperator . The parameters to pass can be obtained using a python operator and pass it to your operator. It will work like this:

@task(task_id="get_gcs_params")
def get_gcs_params(**kwargs):
    date = kwargs["next_ds"]
    # logic should be as displayed on move.py 
    # ...
    return {"source_objects":source,"destination_object":destination}

gcs_params = get_gcs_params()

copy_file = GCSToGCSOperator(
    task_id='copy_single_file',
    source_bucket='data',
    source_objects= gcs_params.output['source_objects'],
    destination_bucket='data_backup',
    destination_object= gcs_params.output['destination_object'],
    gcp_conn_id=google_cloud_conn_id
)  

For additional guidance you can check the cloud storage examples list . I useCopy an object between buckets for guidance.

Answer 2

Did you want to do this copy task using Airflow?

If yes, Airflow provide GCSToGCSOperator