Amazon S3. Strange bucket behavior, slow response with curl

Question

I have got Amazon S3 bucket with some strange behavior. When a to a request similar to GET /?delimiter=%2F&marker=&max-keys=1000&prefix= to get contents or a "root" folder of a bucket, i am getting success headers back, but a response body is delayed. Depending on a timeout i am settings, it can receive different parts of a body. It looks like i am getting < symbol, after a second? and later x, m, l, and so on. Data is received with some delays almost after every byte. If i set a request timeout to 100 seconds i almost always have full response. If i set timeout to 30 seconds i can get empty body or something like "

This strange problem happens only in one S3 account and with one bucket in that account. All other bucket are fine.

Same behavior i see when use the tool s3cli . I used the https traffic sniffer ii really see how bytes appears with a delay from a network.

BUT. When i use the too Cyberduck for Amazon S3, it works fast with same bucket!

I tried to experiment with http headers, make them same as cyberduck, but no success. For my curl based tool problem exists

My request is as this one

GET /?delimiter=%2F&marker=&max-keys=1000&prefix= HTTP/1.1
User-Agent: My S3 Explorer
Host: s4-******.s3.amazonaws.com
x-amz-request-payer: requester
X-Amz-Date: 20190920T151135Z
Date: Fri, 20 Sep 2019 15:11:35 GMT
x-amz-content-sha256: e3******55
Authorization: AWS4-HMAC-SHA256 *****
Connection: Keep-Alive

And request from Cyberduck is as this

CONNECT s4-*********.s3.amazonaws.com:443 HTTP/1.1
Host: *********.s3.amazonaws.com:443
User-Agent: Cyberduck/7.0.1.30930 (Mac OS X/10.14) (x86_64)
HTTP/1.0 200 Connection established
GET /?max-keys=1000&versions&prefix&delimiter=%2F HTTP/1.1
Date: Fri, 20 Sep 2019 09:26:20 GMT
x-amz-request-payer: requester
x-amz-content-sha256: e3*********55
Host: s4-********.s3.amazonaws.com
x-amz-date: 20190920T092620Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.0.1.30930 (Mac OS X/10.14) (x86_64)

It returns full response fast. But why?

There is one difference, before doing a request it executed some CONNECT command (i see this from sniffer). Can that command affect? If yes, how do repeat same with curl (php)?

What can be the secret here?

Answer 1

I have found the solution. If i add the argument "&versions=" to my request, all works fine, I don't understand this, but it is the fact

GET /?delimiter=%2F&marker=&max-keys=1000&prefix=&versions= HTTP/1.1

This works fine without any other changes.

UPDATE . This solution is not really a solution for me. I don't want to modify Amazon S3 API calls i do with that extra argument. I still hope to find some better solution, because this case looks like Amazon S3 has some broken server or equipment or configuration on some server