[英]Slow file uploading to Google Cloud Storage
I'm noticing quite low upload speeds to Google Cloud Storage, almost 2.5x slower compared with uploading a file to Google Drive.我注意到到 Google Cloud Storage 的上传速度非常低,与将文件上传到 Google Drive 相比几乎慢了 2.5 倍。 Here is a screencast comparing the two for the upload of a 1GB file:
这是一个截屏视频,比较两者上传 1GB 文件:
https://gyazo.com/c3488bd56b8118043b7df5aab813db01 https://gyazo.com/c3488bd56b8118043b7df5aab813db01
This is just an example, but I've also tried using the gsutil
command-line tool, using all the suggestions they have for uploading large files the fastest (such as using parallel_composite_upload_threshold
).这只是一个示例,但我也尝试过使用
gsutil
命令行工具,使用他们提供的所有建议以最快地上传大文件(例如使用parallel_composite_upload_threshold
)。 It is still slower than I'm accustomed to.它仍然比我习惯的要慢。 Much slower.
慢得多。
Is there any way to improve this upload speed?有什么办法可以提高这个上传速度吗? Why is upload to Drive so much faster than doing the same to GCS?
为什么上传到云端硬盘比上传到 GCS 快得多?
It took me a day to upload around 30.000 images (100kb/image) using console.google.cloud.com or using browser, the same as you did.我花了一天时间使用 console.google.cloud.com 或使用浏览器上传了大约 30.000 张图片(100kb/image),和你一样。 Then I tried to use
gsutil
to upload the files using terminal in Ubuntu.然后我尝试使用
gsutil
在 Ubuntu 中使用终端上传文件。 Following the instruction here : https://cloud.google.com/storage/docs/uploading-objects .按照此处的说明: https ://cloud.google.com/storage/docs/uploading-objects。
For single file :对于单个文件:
gsutil cp [LOCAL_OBJECT_LOCATION] gs://[DESTINATION_BUCKET_NAME]/
For directory :对于目录:
gsutil -m cp -R [DIR_NAME] gs://[DESTINATION_BUCKET_NAME]
Using gsutil
it was incredibly faster, I uploaded 100.000 images (100~400kb/image) and it only took less than 30 minutes.使用
gsutil
速度非常快,我上传了 100.000 张图片(100~400kb/image),只用了不到 30 分钟。
Honestly I haven't done lots of research why using gsutil
is totally faster than using console.老实说,我没有做很多研究,为什么使用
gsutil
比使用控制台要快。 Probably because of gsutil
provides the -m
option which performs a parallel (multi-threaded/multi-processing) copy which can significantly increase the upload performance.可能是因为
gsutil
提供了-m
选项,该选项执行并行(多线程/多处理)复制,可以显着提高上传性能。 https://cloud.google.com/storage/docs/composite-objects https://cloud.google.com/storage/docs/composite-objects
Well, first of all both these products serve different purposes.好吧,首先这两种产品都有不同的用途。 While Drive can be seen more of a small-scale file storage using cloud, Cloud Storage is focused in the integration with Google Cloud Platform products, data reliability, accessibility, availability in a small-to-high scale.
虽然可以看到 Drive 更多地是使用云的小规模文件存储,但 Cloud Storage 专注于与 Google Cloud Platform 产品的集成、数据可靠性、可访问性、小规模到大规模的可用性。
You need to take into account that when you are uploading a file to Cloud Storage, it is treated as a blob object , which means that it has to go through some extra steps, for example the object data needs to be encrypted when uploaded to Cloud Storage, and uploaded objects are checked for consistency .您需要考虑的是,当您将文件上传到 Cloud Storage 时,它会被视为blob 对象,这意味着它必须经过一些额外的步骤,例如上传到 Cloud 时需要对对象数据进行加密检查存储和上传对象的一致性。
As well, depending on the configuration of your bucket, objects uploaded might have enabled version control , and the bucket might be storing the data in various regions at the same time , which can slow the file uploads.同样,根据您的存储桶的配置,上传的对象可能启用了 版本控制,并且存储桶可能同时将数据存储在不同的区域,这可能会降低文件上传速度。
I believe some of this points, specially encryption, is what make file uploads slower in Cloud Storage compared to Drive.我相信其中一些要点,特别是加密,是导致云存储中文件上传速度比云端硬盘慢的原因。
I would recommend however, to have the bucket you are uploading to at the region closest to your area , which could make a difference.但是,我建议您将要上传到的存储桶放在离您所在区域最近的区域,这可能会有所不同。
I'm noticing quite low upload speeds to Google Cloud Storage, almost 2.5x slower compared with uploading a file to Google Drive.我注意到上传到Google云端存储的速度非常低,与将文件上传到Google云端硬盘相比,速度慢了将近2.5倍。 Here is a screencast comparing the two for the upload of a 1GB file:
以下是截屏视频,比较了两者用于上传1GB文件的情况:
https://gyazo.com/c3488bd56b8118043b7df5aab813db01 https://gyazo.com/c3488bd56b8118043b7df5aab813db01
This is just an example, but I've also tried using the gsutil
command-line tool, using all the suggestions they have for uploading large files the fastest (such as using parallel_composite_upload_threshold
).这只是一个示例,但我也尝试过使用
gsutil
命令行工具,并使用它们提供的所有建议来最快地上传大文件(例如,使用parallel_composite_upload_threshold
)。 It is still slower than I'm accustomed to.它仍然比我习惯的慢。 Much slower.
慢得多。
Is there any way to improve this upload speed?有什么办法可以提高上传速度? Why is upload to Drive so much faster than doing the same to GCS?
为什么上传到云端硬盘的速度要比上传到GCS的速度快得多?
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.