与本地文件系统相比，Python 代码解压缩文件和写入谷歌云存储的时间太长

Question

Its quite weird, All I am trying to do is decompress the file and save it . file is having

size: 16 Mb
extension = .json.gz
Source location = Google Cloud Storage
Destination location = Google Cloud Storage / Local File System

When I use

%%time
import gzip
import shutil
import gcsfs
with gcp_file_system.open('somebucket/<file.json.gz>','rb') as fl_:
    with gzip.open(fl_, 'rb') as f_in:        
        with gcp_file_system.open('somebucket/<file.json>','wb') as f_out:
            shutil.copyfileobj(f_in, f_out)

It produces: Wall time: 5min 51s

But when I try the same and change destination to Local machine

%%time
import gzip
import shutil
import gcsfs
with gcp_file_system.open('somebucket/<file.json.gz>','rb') as fl_:
    with gzip.open(fl_, 'rb') as f_in:        
        with open('localdir/<file.json>','wb') as f_out:
            shutil.copyfileobj(f_in, f_out)

It produces: Wall time: 8.28 s

Am not sure, what is playing role like buf_size,.network speed, some gcsfs backend.

Answer 1

Instead of using a gcsfs file, use the BlobReader class from the GCS client library, for example:

Local Destination

%%time
import gzip
import shutil
from google.cloud import storage
from google.cloud.storage import fileio 

storage_client = storage.Client()
bucket = storage_client.bucket('my_bucket')
blob = bucket.blob('file.json.gz')
reader = fileio.BlobReader(blob)
f_out = open('localdir/file.json','wb')
gz = gzip.GzipFile(fileobj=reader, mode="rb")
shutil.copyfileobj(gz, f_out)
f_out.close()
gz.close()
reader.close()

GCS Destination:

%%time
import gzip
import shutil
from google.cloud import storage
from google.cloud.storage import fileio 

storage_client = storage.Client()
bucket = storage_client.bucket('my_bucket')
blob_in = bucket.blob('file.json.gz')
reader = fileio.BlobReader(blob_in)
blob_out = bucket.blob('file.json')
writer = fileio.BlobWriter(blob_out)
gz = gzip.GzipFile(fileobj=reader, mode="rb")
shutil.copyfileobj(gz, writer)
gz.close()
reader.close()
writer.close()