[英]File download speed from Amazon S3 drops to <2 kb/s = very slow download (on the machine it has been uploaded from)
We've been experiencing very strange behaviour in our current infrastructure setup: 在当前的基础架构设置中,我们一直在经历非常奇怪的行为:
Using wget, you see this output on the affected machines for a file we uploaded: 使用wget,您可以在受影响的计算机上看到我们上传的文件的以下输出:
--2014-07-31 16:33:38-- http://s3-eu-west-1.amazonaws.com/not_the_real_file_url
Resolving s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 178.236.6.160
Connecting to s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|178.236.6.160|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2801149 (2.7M) [text/plain]
Saving to: `/dev/null'
0% [ ] 10,111 1.05K/s eta 68m 26s
and it stays like this for 68m! 像这样保持68m! (it does finish the download after that time though) (虽然在那之后确实完成了下载)
And this output for a random file hosted on amazon s3 by somebody else: 此输出是其他人托管在Amazon s3上的随机文件的输出:
--2014-07-31 16:39:21-- https://s3.amazonaws.com/Minecraft.Download/versions/14w31a/minecraft_server.14w31a.jar
Resolving s3.amazonaws.com (s3.amazonaws.com)... 72.21.211.199
Connecting to s3.amazonaws.com (s3.amazonaws.com)|72.21.211.199|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10342238 (9.9M) [application/octet-stream]
Saving to: `/dev/null'
32% [====================================> ] 3,370,945 747K/s eta 12s
Our current solution, is to use our HAProxy as a transparent HTTP proxy. 我们当前的解决方案是将HAProxy用作透明的HTTP代理。
Meaning we have a frontend "cloud.example.com" defined, and a backend that first replaces the requests HOST with "s3-eu-west-1.amazonaws.com" and then uses s3-eu-west-1.amazonaws.com:80 as a server. 这意味着我们定义了一个前端“ cloud.example.com”,一个后端首先用“ s3-eu-west-1.amazonaws.com”替换了请求主机,然后使用了s3-eu-west-1.amazonaws。 com:80作为服务器。 For amazon it then looks like the request is coming from our proxy and we can download the files we stored on S3 thousands of times again. 对于亚马逊,请求似乎来自我们的代理,我们可以再次下载存储在S3上的文件数千次。 :) :)
[2014-07-31 16:56:57 +0200] RUN[28] AVG: '0.9612743812142854' s, LAST_RUN: '0.711118431' s
--2014-07-31 16:56:57-- https://cloud.example.com/not_the_real_file_url
Resolving cloud.example.com (cloud.example.com)... 1.2.3.4
Connecting to cloud.example.com (cloud.example.com)|1.2.3.4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2801149 (2.7M) [text/plain]
Saving to: `/dev/null'
100%[====================>] 2,801,149 2.47M/s in 1.1s
Ok, solved it. 好,解决了
I'm still researching why this solved the issue, but here is what fixed it now: 我仍在研究为什么可以解决此问题,但是现在可以解决此问题:
As I described above, the behaviour occurs on an Ubuntu 12.04.5 KVM-Guest running on an Ubuntu 12.04.4 KVM-Host system. 如上所述,该行为发生在运行于Ubuntu 12.04.4 KVM-Host系统上的Ubuntu 12.04.5 KVM-Guest上。 I took a look today, if we use different kernels (linux-image-*) on the guests (which can still happen since we're not provisioning them with puppet yet). 我今天看了一下,如果我们在来宾系统上使用了不同的内核(linux-image-*)(由于我们还没有为它们提供p,所以这种情况仍然可能发生)。
On KVM-guests where we have the strange <5 KB/s S3 download behaviour, we're using: 在KVM访客上,我们有奇怪的<5 KB / s S3下载行为,我们正在使用:
On KVM-guests with >5 MB/s S3 download speed, we're using: 在S3下载速度> 5 MB / s的KVM客户机上,我们正在使用:
Hope this helps you if you ran into the same issue. 如果您遇到相同的问题,希望对您有所帮助。 I'll post more, if I truly understand why this happens. 如果我能真正理解为什么会发生,我会发布更多信息。
Of course: You should use a *-virtual kernel on a VM-guest, I know. 当然:我知道,您应该在VM guest虚拟机上使用*虚拟内核。 Why only S3 download is slow though kind of confuses me. 为什么只有S3下载速度慢,但让我感到困惑。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.